UHG is on a transformational journey modernizing its technology assets to improve overall experience for their customers. One of the challenges was to provide real time experience to the consumers by processing big data at scale. By leveraging the modernized streaming platform powered by Apache Kafka, Provider Data platform at UHG was able to setup data pipelines to process of billions of events with throughput of greater than 100k eps.

The data processing platform that consumes the data from various sources, compute metrics at the real time is implemented as set of micro services using the Kafka Feature set such as Kafka Stream joins & aggregation, Kafka Connects, RocksDB State Store.

In this session, we will to dive deep into the architecture and the tools that we used to implement our pipeline, the operational processes established, performance improvements we achieved and day-to-day operational challenges we faced.