This lesson explains the algorithm used to assign topic partitions to brokers, including the case when Kafka is rack-aware.

We'll cover the following

Directory allocation

Kafka follows an algorithm when assigning partitions of a topic to brokers. Kafka attempts to evenly divide the partitions among the brokers as much as possible. As an example, consider a topic with 5 partitions and a replication factor of 3, making for a total of 15 partition replicas to be allocated amongst 5 brokers. The general algorithm will proceed as follows:

There should be 3 partition replicas per broker.

Initially, we start with determining the locations for the partition replica leaders. In our example, the replica leader for partition 0 is placed on broker 1, the replica leader for partition 1 is placed on broker 2, the replica leader for partition 2 onto broker 3, and so on and so forth. A pictorial representation is shown below:

Broker 1 Broker 2 Broker 3 Broker 4 Broker 5

Leader Partition#0 Leader Partition#1 Leader Partition#2 Leader Partition#3 Leader Partition#4

Once the replica leaders have been assigned to the brokers, we start assigning the replica followers for each partition.

We start with partition 0. Its leader is already assigned to broker 1 and we need to assign two more replicas for this partition. We’ll assign the remaining replicas of partition 0 on broker 2 and broker 3. For each partition, we place the replicas at an increasing offset from the leader. The goal is not to co-locate the replicas on the same broker as the replica leader or on the same broker together.

Broker 1 Broker 2 Broker 3 Broker 4 Broker 5

Leader Partition#0 Leader Partition#1 Leader Partition#2 Leader Partition#3 Leader Partition#4

Replica Parition#0 Replica Parition#0

Similarly, the replicas for partition 1 get placed on broker 3 and broker 4 since the leader resides on broker 2.

Broker 1 Broker 2 Broker 3 Broker 4 Broker 5

Leader Partition#0 Leader Partition#1 Leader Partition#2 Leader Partition#3 Leader Partition#4

Replica Partition#0 Replica Partition#0 Replica Partition#1

Replica Partition#1

Possible partition allocation on 5 brokers for a topic with 5 partitions and a replication factor of 3

Kafka has become rack aware starting at version 0.10.0. When assigning partition replicas in a rack-aware environment, the algorithm attempts to assign partition replicas in different racks so that, if an entire rack experiences failure, availability isn’t affected. Consider the following rack setup of our example consisting of 5 brokers:

Rack 1 Rack 2 Rack 3

Broker 1 Broker 3 Broker 5

Broker 2 Broker 4

When assigning partition replicas for partition 0 the algorithm first places the leader onto broker 1. It can’t place the next replica onto broker 2 because then there will be two replicas on the same rack. The algorithm can place the second replica of partition 0 onto broker 3 or broker 4. Finally, the third replica will be placed on broker 5 in the third rack. Now all three replicas live in separate racks and the partition as a whole can tolerate rack failures. Given rack awareness, the algorithm prepares a rack-alternating broker list instead of picking brokers in numerical order:

Broker#1 Broker#3 Broker#5 Broker#2 Broker#4

The brokers for partition assignments are then picked in the order above.

Rack 1 Rack 2 Rack 3

Broker 1 [Leader Partition#0] Broker 3 Broker 5 [Replica Partition#0]

Broker 2 Broker 4 [Replica Partition#0]