In high-volume event pipelines, it’s tempting to think scaling equals more hardware. But sometimes the real scaling solutions lie in the architecture.

Our ingestion service pulled batches from Kafka, regrouped events by shard, and processed them in parallel. The catch? The batch couldn’t finish until every shard completed its work. If one shard ingestion lagged, the entire system slowed.

Instead of rushing to scale databases, we turned to observability. Metrics exported to Prometheus and visualized in Grafana revealed the truth: throughput was bottlenecked by the slowest shard more than 90% of the time.

Armed with this insight, we reimagined the pipeline. Inspired by stream processing systems, we shifted from a single consumer group to shard-specific consumer groups. Each service instance filtered out irrelevant events and owned its shard fully. This independence meant shard 3 could take its time without holding back the others. Scaling became precise: we could add resources for one shard without touching the rest.

The results were immediate: higher throughput, fewer wasted resources, and a leaner ingestion backbone. Beyond performance, the bigger lesson was cultural — observability didn’t just alert us to problems, it guided an architectural shift.

This session will walk through the full journey: spotting the bottleneck with metrics, challenging the “one service = one consumer group” assumption, and designing for shard-level independence at scale.

Key Takeaways:
- Observability can drive architecture decisions, not just alerts.
- Why one-consumer-group-per-service isn’t always the right fit.
- How shard-aware design localizes scaling pressure.
- Practical patterns for faster, more resilient ingestion pipelines.