What is Kafka?: Ticketmaster's Data Streaming Journey

Imagine you have thousands of fans trying to buy Taylor Swift concert tickets on the on-sale day. As soon as you like some seats and click to complete the transaction, the application fails stating the seats which were being shown as open are no longer available. Due to stale data presented on the app, customers are unable to see the current available seats on run time. How frustrated would those customers be? How much loss would the company face for not being able to provide a better customer experience? Distributed application architecture requires continuous interactions within applications to pass the data over the network. Communication is the key to enable these systems to scale and improve upon data availability. A decade ago, REST APIs was the best possible answer for communication between distributed applications. However this architecture is unable to solve the issues of network latency, unattainability of large volumes of data transfer and designing resilient systems. At Ticketmaster, we have large volumes of data being processed in a ticketing host which is required by the front-end applications to be able to take quicker decisions based off of the received data. Over the past two years, we have invested our time in designing stream based messaging solutions for making our data highly available to our clients with low latency. Kafka provides a stream based mechanism to deliver large volumes of data with high throughput. With the help of Kafka streams, the ticketing host commit logs are made available to the front-end applications. By making this data highly available at low latency, our clients have achieved the ability to react and take quicker decisions in real time. This talk would help the participants understand the underlying architecture of Kafka message model and use cases where this technology can be applied.