Every business (most, if not all), maintains application data up to some extent.

These data sources typically feed analytical systems to gather insights, or integrate with other systems for actionable outcomes. Common data engineering practices like scheduled jobs, pipelines, ETL are often a natural choice to operate such data feeds. However, these are more batch-oriented, and introduce delay before they are able to yield results.

This talk explores the alternative approach of building data systems in real-time. Real time systems facilitate faster decision making, enhance richer customer experiences and can reduce the total cost of ownership. From data originating within an operational store and moving all the way down to its analytical state while having the need to orchestrate complex workflows along the way, I will uncover methods to go real time. By leveraging change data capture, stateful stream processing, and durable execution frameworks this talk will demonstrate that building with a batch mindset is becoming obsolete.

The investment required to build real-time systems is now comparable to, or even less than, building batch systems. Join Gaurav in this session as he makes a compelling case for adopting real-time data engineering, and reserve batch execution only… and only if the need arises.

Technical requirements: general software system understanding, distributed systems, event driven architectures

Target audience: CTO's, developers, product people, software philosophy

preferred session duration - 45 mins