Data Ingestion and processing are the core of any tech stack. So, in this session, we will discuss various patterns for designing and implementing stream and batch processing pipelines. We will also discuss the real-world use cases at Nutanix, where we developed multiple solutions to provide Multi-Cloud governance to our customers. So, we will cover real-time stream processing of events from clouds like AWS/AZURE, etc.,

We will also present an architecture that has been battle-tested in production for over five years. The components include Apache Pulsar, Apache Flink, PostgreSQL, Redis, Neo4J DB, rule/ML engine, etc., to name a few technologies.

After this presentation, you will learn more about
1. Designing systems to handle and process both Ordered and Unordered data streams
2. Features of distributed Messaging systems that we can take advantage of while designing systems
3. Understand Stream-processing frameworks and design solutions that can process events at the scale of Billions
4. Metrics-driven development of distributed systems to keep production issues in check.
5. Good practices like Implementing heart-beat monitoring and auto-restart systems in your production, so you can take a break without worrying about outages.
6. Achieving fault tolerance while developing event-driven systems by understanding message delivery and processing guarantees.