xxxxxxxxxx
Kafka topics are separated into partitions, each of which contains records in a fixed order.
A unique offset is assigned and attributed to each record in a partition.
Multiple partition logs can be found in a single topic.
This allows several users to read from the same topic at the same time.
Topics can be parallelized via partitions, which split data into a single topic among numerous brokers.
Replication in Kafka is done at the partition level.
A replica is the redundant element of a topic partition.
Each partition often contains one or more replicas, which means that partitions contain messages that are duplicated across many Kafka brokers in the cluster.
One server serves as the leader of each partition (replica), while the others function as followers.
The leader replica is in charge of all read-write requests for the partition, while the followers replicate the leader.
If the lead server goes down, one of the followers takes over as the leader.
To disperse the burden, we should aim for a good balance of leaders, with each broker leading an equal number of partitions
xxxxxxxxxx
Is this a good answer :- Partitioning help in Serealising. commpress and load balance data across Kafka cluster.
Helps to decongest data load on a topic, enables concurrency, faster reads etc
Partitioning takes the single topic log and breaks it into multiple logs, each of which can live on a separate node in the Kafka cluster. This way, the work of storing messages, writing new messages, and processing existing messages can be split among many nodes in the cluster.
So eventually better performance. Messages are referred to as record. Every message has a particular key and value We partition records in the form of key. Partitioning is Append only sequence of record. Once a record is written to partition it is given an offset sequential ID
Partitioning is done on the basis of key
Partitioning takes the single topic log and breaks it into multiple logs, each of which can live on a separate node in the Kafka cluster. This way, the work of storing messages, writing new messages, and processing existing messages can be split among many nodes in the cluster.
xxxxxxxxxx
@EnableKafka
@Configuration
public class KafkaConsumerConfig {
@Bean
public ConsumerFactory<String, String> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
bootstrapAddress);
props.put(
ConsumerConfig.GROUP_ID_CONFIG,
groupId);
props.put(
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class);
props.put(
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class);
return new DefaultKafkaConsumerFactory<>(props);
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
}
https://medium.com/event-driven-utopia/understanding-kafka-topic-partitions-ae40f80552e8