Session

Big Bang Monolithic Data Store to a Distributed Data Streaming platform with Apache Kafka

Problem Statement:
An eCommerce giant had monolithic data store and analytical solutions which is addressing key programs like customer loyalties membership programs, identifying digital solutions for personalised offering to expend the business in newer markets.

With rapid customer increase the current solution is not able to scale to meet business requirements, real time product offering, struggling to launch new initiatives that takes time due to dependencies and tight coupling among multiple systems. ~ 50% of the new initiatives required integration with multiple systems increases time and cost. Making it difficult to derive insights for fraud detection in loyalties & vouchers.

Solutions :
Designed an event driven distributed streaming based microservices platform using Apache Kafka and Kubernetes on Hybrid cloud environment to support rapid customer growth and unlimited scalability. Identified distributed database solutions at domain level (consumer acquisitions, membership, digital comms, analytics, voucher) and enabled communication with event streaming.

Kafka as the backbone of the entire platform, heterogenous applications/systems publishes, events, streams asynchronously and consumes at real time on Kafka. Data available from various source system to enable customers 360-degree views. Data cleaning, transformation, filtering, aggregations achieved with KStream.

Designed Centralised logging platform with Kafka, applications microservices, commercial tooling like Force.Com, kubernetes infra, AWS logs streams to Kafka using various connectors and log agents. Enabling entire logs availability at one platform for multiple purpose like monitoring & alerting, auditing, analytics etc.

Business Values:
• Achieved the business target to double the revenue to 2x
times, supporting rapid customer growth, targeting
abandoned cart behaviour on real time streams that allows
business to offer rewards on conversion of purchase on
abandon cart.
• 40% cost reduction with no middleware/messaging licensing
& support cost
• Reduction of time to market for new business initiatives with
no integration or single integration.
• Real time customer behaviour availability allowing greater
customer purchase conversion & product offering, 360-
degree customer view.
• No Scalability issue during pick like black Friday

Key Take Away for Attendees:
• Hybrid Kubernetes enabled Cluster setup on OnPrem & AWS
• Data Mirroring among cluster using Mirror Maker
• Created common reusable Pattern for
Producing/Consuming Transaction vs non-Transaction Data,
Streaming & Filtering common library for org level.
• Created pattern for Rest proxy deployment using helm
• Centralised logging platform, streaming from various system
like Forec.com, k8s, AWS, Kafka itself etc.

Dipak Kumar

Architecting & Modernisation on Hybrid Cloud using Domain Driven, Event Driven, Microservices pattern

View Speaker Profile