kafka

  • Producer produce data consisting of topic, key, and values.
  • Consumer subscribe to a specific topic
  • The broker/Kafka server receives data from the producer, and for the given topic, append the data to the corresponding partition.
    • the partition is selected by computing the key’s hash, or as specified by the producer
    • this provides the guarantee that a key goes to a specific partition. kafka also guarantees that data from a producer is committed to the partition in the same order that it is received, although ordering between multiple producers is not guaranteed
    • kafka server has many acknowledgement policy to the producers. either the leader acknowledge, or the leader confirms that all replica has acknowledge, or that a quorum of replicas has acknowledge
  • The consumer subscribe to a topic. It then repeatedly poll the broker for new messages.
    • The broker might give the consumer messages for all that topic, or if the consumer is in a group, assign a partition of a top to different consumers of the same group.
    • The consumer can restart polling given an offset of a partition.

Kafka is:

  • horizontally scalable (add more partitions)
  • durable (writes to a file)
  • available (replications + leader election)

Hot shard/partition can be circumvented by having the producer repartitioning the data further. Eg NVDA trades can now be NVDA:{trade_id % 10};

Ordering guarantees:

  • none across partitions.
  • Multiple producers writing to the same partition: Kafka gives them one definitive append order, but does not guarantee any meaningful ordering between the producers. Whichever record gets appended first gets the lower offset.
  • One producer writing multiple records to the same partition: Kafka can preserve the producer’s send order, assuming the producer is configured to avoid retry-induced reordering; modern idempotent producer behavior is designed to preserve this.
  • Same key: records with the same key normally go to the same partition, so they can benefit from per-partition ordering. But this depends on the partitioning scheme and can change if the topic’s partition count changes.