Building a real-time analytics pipeline for market data can be time-consuming, but leveraging open-source tools can help speed up the process.

In this session, I’ll present a project template that serves as a foundation for building a high-performance system using Open Source tools.

We’ll start with data ingestion. While we could write directly to a fast database, I’ll use Apache Kafka to ingest data and demonstrate how to send messages using Python, JavaScript, and Go.

Now, we need an analytics database, and for real-time data, a time-series database seems like a good match. I will demonstrate how to use QuestDB, an Apache 2.0-licensed project, to ingest data at high speed and run queries in milliseconds or faster.

Dashboards are essential when working with market data. I’ll show how to use Grafana OSS to create real-time charts that update multiple times per second and demonstrate how Perspective, a FINOS project, can be used to embed real-time financial visualizations.

To wrap up, I’ll integrate Jupyter Notebook for interactive data exploration and time-series forecasting.

This session will be demo-driven, and I’ll share all the code so you can use it as a starting point.