Jungwook Song
SPITHA Inc., Research engineer, CTO, Application architect, Application Developer
Seoul, South Korea
Actions
2008 PhD in Computer Science and Engineering @ KKU
2008-2009 Research Professor @ KKU
2010 Post-Doc @ KIST
2011-2014 Freelancer Application Developer
2015 CTO @ Brick&
2016-2022 Research Engineer @ SPARK&ASSOCIATES
2020- CCAAK, CCDAK (had renewed in 2022, 2025)
2022- CTO @ SPITHA Inc.
Confluent Certified [Data Streaming Engineer | Administrator for Apache Kafka | Developer for Apache Kafka]
Badges
Area of Expertise
Topics
Don't Judge the Consumer by Its Lag: Uncovering the Metrics That Truly Matter
In today’s fast-paced world of real-time data processing, Apache Kafka has become essential for managing massive streams of information. A key performance metric is consumer lag—the number of messages waiting unprocessed in a consumer group. At first glance, rising lag appears to signal that consumers are falling behind. Yet, this metric alone can be misleading.
Imagine a busy restaurant where orders pile up on the counter. It might be tempting to blame the chefs, but delays could also stem from late ingredient deliveries or a malfunctioning oven. Similarly, spikes in consumer lag might not indicate a failing consumer at all; they can result from external factors like sluggish downstream systems, temporary bottlenecks in external services, or sudden surges in data volume.
This presentation challenges the conventional reliance on consumer lag as the sole indicator of performance. We will explore how integrating additional metrics—such as message ingestion rates, processing throughput, and the health of interconnected services—provides a more holistic view of your Kafka ecosystem. Through real-world case studies and practical insights, you’ll learn to diagnose issues more accurately and uncover hidden bottlenecks that might otherwise go unnoticed.
Join us as we peel back the layers of Kafka’s consumer dynamics and move beyond a single metric. Discover strategies to optimize your data pipelines, ensuring they remain robust and agile amid evolving challenges.
Is your Kafka 'truly' load balancing? Pro tips for leader partition distribution with network usage.
One of the primary goals of distributing leader partitions across all brokers in a Kafka cluster is load balancing.
This practice helps prevent overloading any single broker and optimizes the throughput of the entire cluster.
The distribution of these leader partitions is mainly determined by the preferred leaders set during the creation of topic partitions.
However, the built-in CLI or API-provided leader election functionality does not effectively achieve the core objective of distributing leader partitions while considering network usage.
Anticipating future usage when creating topic partitions is challenging, and usage patterns can change over time.
Therefore, while maintaining the balance of the number of leader partitions in a Kafka cluster, significant fluctuations in actual network usage among leader brokers can still occur.
To address these issues, I aim to introduce methods for evenly distributing leaders while taking network usage into account.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top