Avinash Upadhyaya
Platform Engineer at Platformatory
Bengaluru, India
Actions
Avinash is a platform engineer and an experienced consultant helping business deploy, scale and architect data platforms using open source technologies such as Apache Kafka, Apache Flink, Kubernetes, etc. He is an avid technical evangelist advocating for open source technologies in the local communities through meetups and conferences. He helps run multiple local communities around data streaming, Apache Kafka, Apache Flink, Apache Pinot among others. Apart from consulting, Avinash is an open source contributor with contributions to various projects in the CNCF ecosystem.
Links
Area of Expertise
Topics
Transforming APIs into an event source with Kafka Streams
REST-based Request-response APIs are the lifeblood of most architectures. API transactions,
specifically involving the CUDs (Creates, Updates, Deletes) are great sources of events and/or
change data. This is also true of various RPC style APIs.
However, because of the client-server nature, such events need to be published by the service,
using an "outboxing" mechanism and subsequently disseminated to pub-sub event broker,
most often Apache Kafka. This involves code changes to the application and in addition, a more
expansive solution like CDC (change-data-capture) to collect those changes from the outbox
table.
In this session, we will present an alternative pattern that leverages standard API proxies /
gateway solutions to process and push events to Kafka, as they occur in the API data plane at
real-time.
We will address the following important questions during this Show Me How session:
1. How to source event data from APIs (and API gateways) such as Kong and Istio
2. How to synthesize event data into the desired form using Kafka Streams
3. How to source change feeds against aggregates using Kafka Streams state stores
4. Considerations and tradeoffs for this architecture
5. Key use-cases (hint:Event Sourcing, Data exchange Audit-trails and more!)
This is a potential no-code alternative to CDC, but with added benefits such as broad access to
domain and context-specific request-response data, as opposed to having to process multiple
row-level changes. Finally, this also enables treating eventing as a cross-cutting concern
associated with API management and operations.
Attendees can expect to take away a new and interesting approach towards a progressive,
low-friction transition to an event-driven architecture
Distributed Benchmarking for Apache Kafka
In most distributed systems, the clients are distributed along with the servers. Traditionally, benchmarking an Apache Kafka cluster has involved running the perf-test and e2e-latency CLI tools from a single machine. This approach is only valid when there is a single client instance using the Kafka cluster.
- A typical benchmarking approach does not represent the real world traffic and is synthetic.
- When the clients are distributed, concurrency is the devil, which is not accounted for by most benchmarking tools.
- Traditional perf test tools also lack the ability to interpret, record and view the metrics for a period of time.
The Kafka Benchmark Operator is an open source Kubernetes operator to run the performance tests in a truly distributed fashion using the Kafka perf tools. The operator includes a Prometheus based scaffold to record the performance metrics, thereby supporting Prometheus Remote Write to systems such as New Relic, Elastic, Influx, etc.
In this session, we will discuss the essentials for an accurate and developer friendly benchmarking tool for Kafka and how the Kafka Benchmark Operator solves for this. We will take a look at other benchmarking tools for Kafka and discuss the merits and demerits of these tools. The session includes a live demo where we benchmark the cluster using the operator and demonstrate using the operator for various conditions. The attendees will be able to understand how to approach benchmarking an Apache Kafka cluster in a distributed manner and utilize the operator for fine tuning Apache Kafka clusters.
Interceptor masterclass for librdkafka clients
Interceptors play a critical role in the Apache Kafka ecosystem, with use cases around observability, logging, and simple transformations with Kafka producers and consumers. This lightning talk explores the concrete need for interceptors and implementing interceptors with librdkafka clients.
Most non-JVM Kafka clients are based on librdkafka, making it one of the most popular libraries for Kafka clients. Traditionally, the development of custom interceptors has primarily revolved around JVM-based languages, leaving non-JVM clients, particularly those relying on librdkafka, in a void. This session addresses this gap by focusing on the imperative role of interceptors and demonstrating how librdkafka can serve as a robust solution for crafting custom interceptors in non-JVM environments.
The interceptor API is currently only available for the C langauge clients. The session aims to provide factual insights into the step-by-step process of developing custom interceptors for both producers and consumers using librdkafka for other languages, ensuring attendees acquire practical knowledge to address specific use cases in their Kafka deployments.
Current Bengaluru 2025 Sessionize Event
Kafka Summit Bangalore 2024 Sessionize Event
Avinash Upadhyaya
Platform Engineer at Platformatory
Bengaluru, India
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top