42 Kafka interview questions with detailed answers, sorted from basic to advanced. Free to practice, with progress saved automatically.
Kafka is a distributed event streaming platform that stores messages in an ordered, replicated log so many systems can publish and consume them independently. Topics include producers and consumers, partitions and ordering, retention and log compaction, delivery semantics, idempotence and exactly-once processing, dead letter queues, event sourcing, stream processing and operations.
and 6 more topics.
Apache Kafka is an event streaming platform: services write events to Kafka and other services read them to process, without either side knowing about the other. Kafka was developed at LinkedIn and open-sourced in 2011. How it differs from a typical message queue: reading does…
A Kafka cluster is made of several brokers. Data is split by topic, each topic is split into partitions to run in parallel, and a consumer group shares the work of reading those partitions. Component What it is ------ Broker One Kafka server. A cluster…
A producer is an application that writes messages to a topic. A topic is split into partitions, and the producer picks a partition for each message (same key, same partition). A consumer reads messages from a partition by offset (the message's position in that partition)…
An offset is a message's position within a partition, starting at 0. Each consumer group stores the processed offset of every partition in the internal topic consumeroffsets. The committed offset is the offset of the next message to read. When a consumer restarts or a…
Kafka only guarantees order within a partition: a message written earlier has a smaller offset and is read first. There is no ordering across partitions. So to keep related messages in order, give them the same key: the partitioner hashes the key, so the same…
Replication means Kafka keeps several copies of each partition on different brokers. Each partition has one leader that takes all writes, and followers continuously fetch from the leader to copy its data. The ISR (In-Sync Replicas) is the set of replicas that are caught up…
Kafka Connect is a framework that ships with Kafka for moving data between Kafka and other systems (databases, Elasticsearch, S3…). Instead of writing a producer/consumer for each system, you declare a connector in a config file and submit it to Connect through its REST API.…
Kafka Streams is a Java library for writing stream-processing applications that read from Kafka topics and write results to other topics. It runs inside your own application as a dependency, with no separate cluster like Spark or Flink. Two core concepts: - KStream: a sequence…
Kafka fits problems where several systems need to read the same stream of events. Messages are kept according to retention rather than deleted once read, can be read again (replay), and each consumer group tracks its own offset, so readers are independent of one another.…
An idempotent producer is one where, when it retries the same batch, the broker writes that batch to the partition only once. It blocks duplicates caused by retries, not your own code calling send() twice. The case it handles: the broker writes a batch but…
The producer does not send messages one by one. It groups records for the same partition into an in-memory batch and sends the whole batch in one request. Two settings decide when a batch goes out: - batch.size (default 16384 bytes = 16 KB): the…
At-most-once (fire-and-forget): the producer sets acks=0 and the consumer commits the offset before processing. Messages can be lost but never duplicated. Fine for metrics and logs, where absolute accuracy is not the point. At-least-once: the producer sets acks=all with retries and the consumer commits after…
In a consumer group, each partition goes to exactly one consumer, so consumers split the reading without overlap. The split is decided by the assignor and is recomputed whenever the group changes (a rebalance). In the classic protocol (the default), the group coordinator is a…
The partition key decides which partition a message goes to. Kafka only guarantees order within one partition, so events that must stay in order relative to each other need the same key. Choosing a key balances ordering against an even load. The same key always…
A message queue is an asynchronous communication channel between services: a producer pushes messages in, and a consumer pulls them out and processes them at its own pace. The producer does not wait for the consumer to finish. Benefits: - Decoupling: producer and consumer do…
Kafka keeps messages according to each topic's retention config, independent of whether consumers have read them. Two limits apply: - By time — retention.ms: keep messages for a set duration (default 7 days = 604800000ms). Past that, old segments are deleted. - By size —…
The producer sets compression.type, choosing one of: none, gzip, snappy, lz4, zstd. The producer compresses an entire batch of messages before sending, the broker stores it still compressed, and consumers decompress — saving both network bandwidth and disk. Interaction with batching: compression works on a…
Three delivery guarantees: - At-most-once: each message is processed at most once and may be lost. The consumer commits the offset before processing; if processing fails the message is considered consumed. Only acceptable for loss-tolerant data (metrics, trace logs). - At-least-once: messages are never lost…
A DLQ (dead letter queue) holds messages that could not be processed after retries are exhausted. Its purpose is to remove broken messages from the main flow so the consumer keeps running, without losing the data. A message entering the DLQ must carry enough context…
Kafka only guarantees ordering within a partition, never across partitions. To keep related events in sequence you must route them to the same partition by giving them the same key. Pick the key as the unit that needs ordering, usually an entity id: orderid for…
Consumer lag = the partition's latest offset (log end offset) minus the offset the consumer group has committed. It tells you how many messages the consumer is behind the producer. Measuring: The LAG column is per partition. In a real system you scrape the metric…
A rebalance redistributes partitions among consumers in a group. It triggers when a consumer joins or leaves, when a consumer is considered dead (no heartbeat within session.timeout.ms, or no poll() within max.poll.interval.ms), or when the topic's partition count changes. Impact: with the older eager protocol,…
