Kafka Introduction

Introduction

This is an introduction to Kafka which describes itself as a messaging system

LinkedIn Worked Example

Here is some stats from LinkedIn

And here is there architeture pre-2010.

Post 2010 architecture.

This is the current scale.

Terms

• Log shipping is the process of automating the backup of transaction log files on a primary (production) database server, and then restoring them onto a standby server.
• Message Broker is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver
• ZooKeeper is essentially a service for distributed systems offering a hierarchical key-value store, which is used to provide a distributed configuration service, synchronization service, and naming registry for large distributed systems (see Use cases). ZooKeeper was a sub-project of Hadoop but is now a top-level Apache project in its own right.