Modern security detection is no longer a tooling problem — it is a data engineering and analytics problem. Enterprise environments generate vast volumes of telemetry across endpoints, identities, networks, and applications, yet most organisations struggle to transform this data into actionable intelligence.

In this session, I present a highly technical, data-centric walkthrough of building a heuristic-based machine learning threat detection pipeline using the Microsoft Data Platform, based on my postgraduate research into ML-driven Trojan detection on Windows.

We will explore how raw security telemetry can be ingested, modelled, enriched, and analysed at scale using Microsoft Fabric, Azure analytics services, and SQL-based processing, before applying machine learning for behavioural detection.

Key technical areas covered include:

Designing a lakehouse-based architecture in Microsoft Fabric for security telemetry

Ingesting high-volume event data (process execution, API calls, entropy metrics, behavioural signals) into OneLake

Feature engineering using T-SQL, Spark SQL, and Dataflows to extract heuristic indicators such as execution patterns, import anomalies, entropy shifts, and behavioural sequences

Training and scoring ML models (XGBoost, Random Forest, ensemble classifiers) using Fabric notebooks and Azure ML integration

Persisting detection results back into structured tables for analytics and reporting

Visualising detection outcomes and trends using Power BI for analysts and decision-makers

The session focuses on how data professionals can design scalable, production-ready analytics pipelines that support advanced security use cases, demonstrating that modern threat detection is fundamentally an analytics and data modelling challenge.

Attendees will leave with practical architectural patterns and technical insight applicable to any organisation using Microsoft Fabric, Azure SQL, or Power BI to analyse large-scale operational or security data.