Breaking News: Grepper is joining You.com. Read the official announcement!

Committing a Specific Offset#

Sumit Rawal answered on June 1, 2023 Popularity 1/10 Helpfulness 1/10

answer Committing a Specific Offset#

related Committing a Specific Offset#

Committing a Specific Offset#

Comment

So far we have seen how offsets can be committed synchronously or asynchronously. In both cases, we commit the offset after all the returned records have been processed. If we were to return a large number of records, though, we may want to commit the offset midway or at a time of our choosing. Fortunately, both the APIs commitSync() and commitAsync() allow us to do just that by passing-in the offset for each partition of a topic that we want to commit. A sample snippet appears below:

  Map<TopicPartition, OffsetAndMetadata> currentOffsets = new HashMap<>();
  int count = 0;

  while (true) {
      Duration oneSecond = Duration.ofMillis(1000);
      System.out.println("Consumer polling for data...");
      ConsumerRecords<String, String> records = consumer.poll(oneSecond);

      for (ConsumerRecord<String, String> record : records) {
          /**
            * Process records here
            */

          // commit affset after processing every hundred records
          if (count % 100 == 0) {
              // Set-up the topic partition for the record
              // we are processing currently
              TopicPartition topicPartition = new TopicPartition(record.topic(), record.partition());

              // We add 1 to the current record's offset so that
              // we start reading from the record after the current
              // one in case of a rebalance.
              OffsetAndMetadata metadata = new OffsetAndMetadata(record.offset() + 1, "no metadata");
              currentOffsets.put(topicPartition, metadata);

              consumer.commitAsync(currentOffsets, null);
          }
          count++;
      }
  }

Popularity 1/10 Helpfulness 1/10 Language java

Source: Grepper

Tags: java

Link to this answer
Share Copy Link

Contributed on Jun 01 2023

Sumit Rawal

0 Answers Avg Quality 2/10

Closely Related Answers

Committing a Specific Offset#

Comment

  Map<TopicPartition, OffsetAndMetadata> currentOffsets = new HashMap<>();
  int count = 0;

  while (true) {
      Duration oneSecond = Duration.ofMillis(1000);
      System.out.println("Consumer polling for data...");
      ConsumerRecords<String, String> records = consumer.poll(oneSecond);

      for (ConsumerRecord<String, String> record : records) {
          /**
            * Process records here
            */

          // commit affset after processing every hundred records
          if (count % 100 == 0) {
              // Set-up the topic partition for the record
              // we are processing currently
              TopicPartition topicPartition = new TopicPartition(record.topic(), record.partition());

              // We add 1 to the current record's offset so that
              // we start reading from the record after the current
              // one in case of a rebalance.
              OffsetAndMetadata metadata = new OffsetAndMetadata(record.offset() + 1, "no metadata");
              currentOffsets.put(topicPartition, metadata);

              consumer.commitAsync(currentOffsets, null);
          }
          count++;
      }
  }

Popularity 1/10 Helpfulness 1/10 Language java

Source: Grepper

Tags: java

Link to this answer
Share Copy Link

Contributed on Jun 01 2023

Sumit Rawal

0 Answers Avg Quality 2/10

Committing a Specific Offset#

Contents

More Related Answers

Committing a Specific Offset#

Closely Related Answers

Committing a Specific Offset#

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.