Committed offset less than last record read

Committed offset greater than last record read

We have read so far that Kafka consumers poll topic partitions for records using the poll() method. The method returns some number of records for the consumer to process. However, if a rebalance occurs and the partition gets assigned to a different consumer within the consumer group, then the newly assigned consumer must know where the previous consumer stopped in order to resume reading records from that point on in the partition.

The offset identifies the position in a partition up to which a consumer has read. The act of durably storing or updating that position is called the commit. Unlike some other messaging systems, Kafka doesn’t track acknowledgments from the consumers of read records. Instead, the onus of tracking a consumer’s position within a partition is on the consumer itself. Each consumer commits its offset for every partition it is reading by writing a message to a special Kafka topic called __consumer_offsets. 

Current offset is the offset from which next new record will be fetched.

Committed offsets is the last committed offset for the given partition. Consumer produces a message to Kafka, to a special __consumer_offsets topic, with the committed offset for each partition.

If a consumer crashes or a new consumer joins the consumer group, this will trigger a rebalance. In order to know where to pick up the work, the consumer will read the latest committed offset of each partition and continue from there. We call this action of updating the current position in the Kafka partition a commit.