Kafka Partition Reassignment

When a new Kafka node is added, Kafka automatically does not do the partition reassignment. The following steps are used to perform manual reassignment of Kafka partitions for specified topic(s).

1. Enter shell of Kafka Docker container:

   docker exec -it kafka-1 bash
   

2. Create /tmp/topics.json with topics whose partitions should be reassigned in the following format:

   cat << EOF | tee /tmp/topics.json
   {
   "topics": [
   {"topic": "events.tenant.stream"}
   ],
   "version": 1
   } 
   EOF
   

3. Generate reassignment JSON output from list of topics to be migrated. Specify the broker IDs in the broker list:

   /usr/bin/kafka-reassign-partitions \
   --bootstrap-server localhost:9092 \
   --topics-to-move-json-file /tmp/topics.json \
   --broker-list "1,2" \
   --generate
   

   Kafka prints out current layout and suggested layout. Save the new suggested layout to /tmp/reassignment.json.

   In example:

   cat << EOF | tee /tmp/reassignment.json
   {"version":1,"partitions":[{"topic":"events.tenant.stream","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"events.tenants.stream","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"events.tenants.stream","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"events.tenants.stream","partition":3,"replicas":[1],"log_dirs":["any"]},{"topic":"events.tenants.stream","partition":4,"replicas":[2],"log_dirs":["any"]},{"topic":"events.tenants.stream","partition":5,"replicas":[1],"log_dirs":["any"]}]}
   EOF
   

4. Use the output from the previous command as input to the execution of the reassignment/rebalance:

   kafka-reassign-partitions \
   --bootstrap-server localhost:9092 \
   --reassignment-json-file /tmp/reassignment.json \
   --execute
   

   Tip

   If there are previous reassignment tasks running, Kafka responds with an error: There is an existing partition reassignment running.

   You can use --additional flag in this case.

   kafka-reassign-partitions \
   --bootstrap-server localhost:9092 \
   --reassignment-json-file /tmp/reassignment.json \
   --additional \
   --execute
   

That's it! Now Kafka should perform the partitions reassignment within the following hours.

For more information, see Reassigning partitions in Apache Kafka Cluster.