Breaking News: Grepper is joining You.com. Read the official announcement!

Example use of onPartitionsRevoked#

Sumit Rawal answered on June 1, 2023 Popularity 1/10 Helpfulness 1/10

answer Example use of onPartitionsRevoked#

Example use of onPartitionsRevoked#

Comment

We can implement the interface and pass an instance of the implementing class at the time of invoking the subscribe() call of the consumer object. The code widget below demonstrates an example of the onPartitionsRevoked method in use to commit offsets just before a consumer loses ownership of a partition.

public class RebalanceListenerExample {

    private Map<TopicPartition, OffsetAndMetadata> currentOffsets =
            new HashMap<>();

    class RebalanceHandler implements ConsumerRebalanceListener {

        KafkaConsumer<String, String> consumer;

        RebalanceHandler(KafkaConsumer<String, String> consumer) {
            this.consumer = consumer;
        }

        @Override
        public void onPartitionsRevoked(Collection<TopicPartition> collection) {
            // We are committing the offsets for the most recently processed records
            // as we update the currentOffsets hashmap in the poll loop in the run()
            // method after processing each record. This way we don't save the offset
            // from the last poll() call which would have resulted in re-processing
            // some of the records.
            consumer.commitSync(currentOffsets);
        }

        @Override
        public void onPartitionsAssigned(Collection<TopicPartition> collection) {

        }
    }

    public void run() {

        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        props.put("group.id", "DataJekConsumers");
        props.put("auto.offset.reset", "earliest");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);

        // Pass-in the instance of RebalanceHandler when subscribing to topics
        consumer.subscribe(Pattern.compile("datajek-.*"), new RebalanceHandler(consumer));

        try {
            while (true) {
                Duration oneSecond = Duration.ofMillis(1000);
                System.out.println("Consumer polling for data...");
                ConsumerRecords<String, String> records = consumer.poll(oneSecond);
                for (ConsumerRecord<String, String> record : records) {

                    TopicPartition topicPartition = new TopicPartition(record.topic(), record.partition());

                    // We add 1 to the current record's offset so that
                    // we start reading from the record after the current
                    // one in case of a rebalance.
                    OffsetAndMetadata metadata = new OffsetAndMetadata(record.offset() + 1, "no metadata");
                    currentOffsets.put(topicPartition, metadata);
                }

                // Async commit without callback
                consumer.commitAsync();
            }
        } catch (Exception e) {
            // log exception
        } finally {

            try {
                // Initiate a synchronous commit since this is the last commit
                // before the consumer exits. The synchronous commit will retry
                // untill success or fail in case of an unrecoverable error.
                consumer.commitSync();
            } finally {
                consumer.close();
            }
        }
    }
}

Popularity 1/10 Helpfulness 1/10 Language java

Source: Grepper

Tags: java

Link to this answer
Share Copy Link

Contributed on Jun 01 2023

Sumit Rawal

0 Answers Avg Quality 2/10

Example use of onPartitionsRevoked#

Contents

More Related Answers

Example use of onPartitionsRevoked#

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.