Migrating on-premises insurance data platform to the cloud: A story from the insurance industry
The insurance tech industry has evolved from old, expensive systems but complex rules and highly sensitive PII data often mean that the systems remain long after their expiry date. Huge amounts of technical debt accumulate as analysts try to band aid a deteriorating code base and in turn make the business line dependent on shadow IT departments as time goes on. Often there doesn’t exist a clear dividing line between the OLTP and OLAP workloads causing the reporting layer becoming intractable. Users end up relying on data extracts to get what they need, reducing data engineers to data babysitters.
As with other traditional industries trying to modernise workloads, the insurance industry struggles with its analytics solutions. You have analysts that are stuck keeping production running, never getting beyond descriptive analysis. Also, shifting the analysis space right and the data governance to the left is an intrinsically hard problem for most businesses. Margins are often thin and earnings requirements high, so that the paths to moving to more data-driven ops are narrow and difficult to navigate. For many, the solution is the cloud. However, moving to the cloud has many challenges for a highly regulated business such as the Insurance/pension industry.
In this session, we will discuss how Gabler is transforming our on-premises analysis (excel) platform to cloud architecture using synapse pipelines, Azure Databricks change data feed, Databricks lakehouse and synapse serverless architecture. We’ll also discuss how we are implementing data governance using unity catalog, and how we went about breaking down a stateful data model to a more event-based star model (Kimball for the win!).
Lakeflow Declarative Pipelines Demystified: A Practical Guide to Delta Architecture
This session introduces Lakeflow Declarative Pipelines (formerly Delta Live Tables) and provides a framework for shifting from static to dynamic data engineering, grounded in medallion architecture principles.
We will cover ingestion with Autoloader, configuring Lakehouse Federation and Lakeflow Connectors for external data sources, and evaluating compute options by comparing classic Spark clusters with serverless execution. The session will also address differences between streaming tables and materialized views, data quality frameworks including Expectations, DQX, and Lakehouse Monitoring, and approaches to CI/CD and DataOps for operationalizing Lakeflow workflows.
Participants will gain a clear understanding of how to use Lakeflow Declarative Pipelines to build production-ready workflows, along with practical insights into trade-offs encountered when deploying Databricks in enterprise environments.
Azure Databricks vs. Microsoft Fabric - what really is the difference?
If you've ever wondered, 'Am I a brickhead? Am I a fabricator? How do I choose?' then we are here to help.
This session will explore the nuances of Azure Databricks and Microsoft Fabric, sprinkled with a touch of Azure magic! We will guide you through similarities, differences, and standout features from both solution.
But do you have to choose sides, or can you build your platform using datafabricks?
Get ready for a session packed with key insights drawn from firsthand experiences by three data enthusiasts and let's unravel this conundrum together!
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top