Something strange happened while I worked with Kafka.

While adding a new consumer from Kafka to one of our services, the service stopped consuming from ALL other existing consumers. As part of my job at Taboola as a team leader on a production team in the Infrastructure group, we’re supposed to remove bottlenecks, not create them.

This talk will describe how I investigated the issue, explain what I discovered, and share my insights into the whole situation.

Taboola’s recommendations appear on tens of thousands of web pages and mobile apps every second. As users engage with the content, multiple events are fired to signal that recommendations are rendered, opened, clicked, and so on. Each event triggers one or more Kafka messages, which translates into a lot of Kafka messages for every recommendation.

We add new types of events to our infrastructure all the time. We usually just add the new topic and relevant consumers, test everything locally with different CI procedures, and then on production servers running existing consumers.

This time, something strange happened when we added a consumer for a new event type.

When we added the new consumers on one server in the server pool, other consumers on that server stopped consuming from all other topics. The new consumer had a different group ID and consumed from a new topic, so this shouldn’t have happened. We were surprised that it affected other groups and other topics.

I will explain what we did and how we solved the issue.

At the end of the day we learned that:

1. There’s a relationship between consumers on the same service and the assignment of topics on a partition, regardless of the group or topic ID.
2. Each server increments consumer IDs unless the order is explicitly overridden.
3. All consumers on a service affect the lexicographical order of consumers on the same service .