Paul Andrew

Information & Communications Technology

Azure Data Platform

Derby, England, United Kingdom

Paul Andrew

Principal Consultant & Solution Architect, Data Platform MVP

Principal consultant and architect specialising in big data solutions on the Microsoft Azure cloud platform.
Data engineering competencies include Azure Data Factory, Data Lake, Databricks, Stream Analytics, Event Hub, IoT Hub, Functions, Automation, Logic Apps and of course the complete SQL Server business intelligence stack.
Many years’ experience working within healthcare, retail and gaming verticals delivering analytics using industry leading methods and technical design patterns.
STEM ambassador and very active member of the data platform community delivering training and technical sessions at conferences both nationally and internationally.
Father, husband, swimmer, cyclist, runner, blood donor, geek, Lego and Star Wars fan!

Current sessions

Azure Data Factory - What? Why? How? Let's Build It!

If we want to achieve any data processing in Azure we need an umbrella service to manage, monitor and schedule our solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS packages. It’s now time to upgrade our skills and start using cloud native services to achieve the same thing on the Microsoft Cloud Platform. Within a PaaS only Modern Data Warehouse, the primary component for delivering that orchestration is Azure Data Factory, combined with Azure Databricks.

In this full day of training we’ll start with the basics and then get hands on with the tools. We’ll build our own Azure ETL/ELT pipelines using all Data Factory has to offer. Plus, consider hybrid architectures, think about lifting and shifting legacy packages, and explore complex bootstrapping to orchestrate everything in Azure with Data Factory.

Specifically, we’ll be covering the following topics:

- An introduction to Azure Data Factory. What is it and why use it?
- How to extend our orchestration processes with Custom Activities, Azure Functions and Web Hooks.
- Using SSIS packages in Azure.
- Data Factory Data Flows (Mapping & Wrangling) with support from Azure Databricks
- Dynamic metadata driven pipelines.
- Data Factory alerting and monitoring.
- Data Factory DevOps.
- Lessons learnt from using Azure Data Factory in production.

Azure Data Factory has been around since 2015 and has matured massively to form a complete feature rich cloud offering. Add this tech to your development toolbox today! Join me for this complete lesson in everything you need to deliver Azure Data Factory within your data platform solution.

Have your laptops to hand and come armed with your Azure subscription, including credit and rights to deploy resources.

Please check with bill payer.

Infrastructure not included.


An Introduction to Azure Synapse - What is it? Why use it? And how?

The Microsoft abstraction machine is at it again with this latest veneer over what we had come to understand as the 'modern data warehouse'. Or is it?! When creating an Azure PaaS data platform/analytics solution we would typically use a set of core Azure services; Data Factory, Data Lake, Databricks and SQL Data Warehouse. Now with the latest round of enhancements from the MPP team it seems in the third generation of the Azure SQLDW offering we can access all our core services as a bundle. Ok, so what? Well, this is a reasonable starting point to in our understanding of Azure Synapse, but is also far from the whole story. In this session we'll go deeper into the evolution of our SQLDW to complete our knowledge on why Synapse Analytics is a game changer for various data warehouse architectures. We'll discover what Synapse has to offer with its Data Virtualisation layer, flexible storage and multi model compute engines. A simple veneer of things, this new resource is not. In this introduction to Synapse we'll cover the what, that why and importantly the how for this emerging bundle of exciting services.


An Architects Guide to Delivering Data Insights on the Microsoft Azure Data Platform

The resources on offer in Azure are constantly changing, which means as data professionals we need to constantly change too. Updating knowledge and learning new skills. No longer can we rely on products matured over a decade to deliver all our solution requirements. Today, data platform architectures designed in Azure with best intentions and known good practices can go out of date within months. That said, is there now a set of core components we can utilise in the Microsoft cloud to ingest and deliver insights from our data? When does ETL become ELT? When is IaaS better than PaaS? Do we need to consider scaling up or scaling out? And should we start making cost the primary factor for choosing certain technologies? In this session we'll explore the answers to all these questions and more from an architects viewpoint. Based on real world experience lets think about just how far the breadth of our knowledge now needs to reach when starting from nothing and building a complete Microsoft Azure Data Platform solution.


Beyond IoT - How to Handle Streaming Data in Azure

The desire and expectation to use real-time data is constantly growing, businesses need to react to market trends instantly. In the new data driven age a daily ETL load/processing window is not enough. We need a constant stream of information and analytics achieved in real-time. In this session will look at how that can be achieved using Azure Stream Analytics. Building streaming jobs that can blend and aggregate data as it arrives to drive live Power BI dashboards. Plus, we will consider options for using Stream Analytics in production and what other tools could be used in the ever-growing list of Azure services. Lastly, we will explore the theory behind a complete Lambda architecture where we combine streaming and batch data together and how this different from the Databricks Delta architecture.


Azure Orchestration - Applying Data Factory in Production

If you have already mastered the basics of Azure Data Factory (ADF) and you are now looking to advance your knowledge of the tool, this is the session for you. Yes, Data Factory can handle the orchestration of our ETL pipelines. But what about our wider Azure environment? In this session we will take a deeper dive into the service, considering how to build custom activities, create metadata driven dynamic pipelines and think about hierarchical design patterns. Plus, explore ways for optimising our Azure compute costs by controlling other resource scaling as part of our normal data processing pipelines. How? Well, once we can hit a REST API from an ADF web activity anything is possible, extending our Data Factory and orchestrating everything in any data platform solution. All this and more in a series of short lessons (based on real world experience) I will take you through how to use Azure Data Factory in production. Finally, we will look at how Data Factory can be deployed using Azure DevOps.


ETL in Azure Made Easy with Data Factory Data Flows (Mapping & Wrangling)

What happens when you combine a cloud orchestration service with a Spark cluster?! The answer is a feature rich, graphical, scalable data flow environment to rival any ETL tech we’ve previously had available in Azure. In this session we will look at Azure Data Factory and how it integrates with Azure Databricks to produce a powerful abstraction over the Apache Spark analytics ecosystem in the form of Mapping and Wrangling Data Flows. If you have ever transformed datasets using SQL Server Integration Services (SSIS) packages or via Power BI’s Power Query tool this is the session for you. Now we can transform data in Azure using our favourite interfaces but with the support of Azure Databricks doing the heavy lifting. In this session you will get a quick introduction to Azure Data Factory before we go deeper into the services new Mapping and Wrangling Data Flows features. Start using cloud native technology and scale out compute within a convenient, easy to use Data Factory rich graphical interface.


A Complete Introduction to Azure Data Factory

SQL Server Integration Services has been a good friend since its first appearance in SQL Server 2005. But now, after a slightly bumpy start, Azure Data Factory is here and ready to replace all our DTSX package capabilities. This cloud native orchestration tool is a powerful equivalent for SSIS and the SQL Agent as a primary component within the Modern Data Warehouse. In this complete introduction session, we will cover the basics of Azure Data Factory. What do we need to build cloud ETL/ELT pipelines? What’s the integration runtime? Do we have an SSIS equivalent cloud data flow engine? Can we easily lift and shift existing SSIS packages into the cloud? The answers to all these questions and more in this session. Come to this session knowing nothing about Azure Data Factory and leave with enough knowledge to start building pipelines tomorrow.


Using Azure DevOps for Azure Data Factory

DevOps as a process is here to stay and is typically a must have requirement for any data platform solution. But how does the concept translate to the technology when implemented? Sadly, the answer isn’t always straight forward. In this short session I’ll introduce how we can continuously integrate and deliver our cloud orchestration resource Azure Data Factory (ADF). We’ll discuss three options for getting our service JSON deployed to production using the popular Azure DevOps environment, previously known as VSTS, and think about the suitability of the Microsoft provided ARM templates for our highly dynamic orchestrator. Does the ADF portal UI really support the DevOps methodology? Can we confidently publish our pipelines? How should we handle the branching of our source code for ADF developers? The answers to all these questions and more in this lightning session. All based on real world experience with the products.


Past and future events

Data Relay 2019

7 Oct 2019 - 11 Oct 2019

DATA:Scotland 2019

12 Sep 2019
Glasgow, Scotland, United Kingdom

DataGrillen 2019

19 Jun 2019 - 20 Jun 2019
Lingen, Lower Saxony, Germany

Global Azure Bootcamp 2019

27 Apr 2019
Birmingham, England, United Kingdom

Intelligent Cloud Conference 2019

7 Apr 2019 - 9 Apr 2019
Copenhagen, Capital Region, Denmark

Global Azure Boot camp - Birmingham UK

21 Apr 2018
Birmingham, England, United Kingdom