Sunilkumar Reddy
Barclays - Data Engineer
London, United Kingdom
Actions
Sunil Kumar Reddy is a Data Engineer specialising in cloud-based data platforms, enterprise data engineering, and financial analytics. His professional experience includes developing data transformation pipelines, implementing data quality controls, and building scalable data solutions for financial services environments.
His technical expertise spans AWS, Microsoft Azure, SQL, Python, dbt, PySpark, and modern data engineering practices. His work focuses on building maintainable data architectures, improving data reliability, and strengthening traceability across complex reporting workflows.
Sunil has also explored the application of explainable artificial intelligence in financial risk management, with an interest in making AI-driven systems more transparent, accountable, and suitable for regulated environments. His work bridges practical data engineering with responsible AI, data governance, and trustworthy analytical systems.
In addition to his industry experience, Sunil has delivered a technical session on explainable AI for financial risk management to an audience of approximately 80 professionals, covering concepts such as SHAP, LIME, and responsible AI practices.
Through community speaking, Sunil aims to share practical engineering lessons, promote knowledge exchange, and help data professionals build reliable, scalable, and trustworthy cloud data platforms.
Links
Area of Expertise
Topics
Governance by Design: Building Azure Data Platforms for Regulated Industries
Organisations operating in regulated industries must ensure that their data platforms deliver more than scalability and performance. They must also provide strong security, consistent governance, data lineage, auditability, and demonstrable control over how data is accessed, transformed, stored, and consumed. In sectors such as financial services, healthcare, and insurance, treating governance as an afterthought can introduce operational risks, compliance challenges, and a lack of trust in critical business data.
In this session, I will explore the principles of **governance by design** and demonstrate how governance capabilities can be embedded throughout the architecture and lifecycle of enterprise data platforms built on Microsoft Azure. Rather than adding compliance controls after implementation, we will examine how security, access management, data quality, lineage, and auditability can become foundational architectural components.
The session will cover the role of Azure Data Lake Storage Gen2, Azure Data Factory, Azure Databricks, Delta Lake, Microsoft Purview, and Azure security services in building governed data ecosystems. I will discuss practical approaches to identity and access management, role-based access control, data classification, metadata management, lineage tracking, and controlled data access across ingestion, transformation, and consumption layers.
We will also explore how governance can be integrated into modern lakehouse architectures using the medallion pattern, with data quality validation, schema enforcement, controlled transformations, and traceable processing across Bronze, Silver, and Gold layers. Additional considerations will include secrets management, environment separation, automated deployment controls, monitoring, and maintaining audit evidence for critical data workflows.
Particular attention will be given to the challenges of regulated environments, where organisations must demonstrate that data is accurate, access is appropriately restricted, transformations are traceable, and reporting outputs can be reconciled and reproduced. Through practical architecture examples and implementation patterns, attendees will learn how to balance governance requirements with engineering efficiency, scalability, and self-service analytics.
**Key takeaways:**
- Design Azure data platforms with governance, security, and compliance built into the architecture.
- Implement identity management, least-privilege access, and role-based security controls.
- Apply data classification, metadata management, and lineage tracking using Microsoft Purview and relevant Azure capabilities.
- Establish data quality, schema enforcement, and reconciliation controls across lakehouse layers.
- Improve auditability through execution logs, historical tracking, and traceable data transformations.
- Integrate governance controls into CI/CD workflows and automated data engineering practices.
- Understand architectural trade-offs when balancing regulatory requirements, platform scalability, and data accessibility.
This session is designed for data engineers, cloud architects, data governance professionals, platform engineers, and technology leaders building enterprise data platforms in regulated industries. Attendees will gain practical architectural insights into creating secure, governed, and audit-ready Azure data ecosystems that support trusted analytics, regulatory reporting, and future AI initiatives.
From Legacy ETL to Scalable Cloud Data Pipelines: Lessons from Enterprise Migration
Enterprise organisations are increasingly modernising legacy ETL systems to meet the demands of cloud-scale analytics, real-time data processing, and AI-driven applications. However, migrating from traditional data warehouses and on-premises ETL platforms to modern cloud data architectures involves more than moving data. It requires careful planning, architectural decisions, data quality management, performance optimisation, and a strategy for maintaining reliability throughout the migration.
In this session, we will explore practical approaches to modernising legacy ETL workflows and building scalable cloud data pipelines using technologies such as Azure Data Factory, Azure Databricks, PySpark, Delta Lake, and cloud storage services. We will examine how to assess existing ETL workloads, identify migration priorities, redesign transformation logic, and adopt modern ELT patterns that improve scalability, maintainability, and operational efficiency.
The session will cover common enterprise migration challenges, including handling complex dependencies, managing incremental data loads, implementing schema evolution, maintaining data consistency, and validating results between legacy and cloud environments. We will also discuss strategies for orchestration, automated data quality checks, monitoring, error handling, CI/CD, and infrastructure management to ensure pipelines remain reliable and auditable in production.
Through practical architecture examples and lessons from enterprise data engineering, attendees will learn how to approach phased migrations, minimise disruption to downstream reporting, optimise cloud processing costs, and establish repeatable engineering practices. We will also explore how modern data modelling and governance practices help organisations build a stronger foundation for analytics, regulatory reporting, and future AI initiatives.
**Key takeaways:**
- Assess legacy ETL workloads and develop a structured cloud migration strategy.
- Choose between lift-and-shift, refactoring, and cloud-native redesign approaches.
- Build scalable ETL/ELT pipelines using Azure Data Factory, Databricks, PySpark, and Delta Lake.
- Implement incremental processing, schema evolution, and reliable data reconciliation.
- Improve pipeline observability through monitoring, logging, automated testing, and error handling.
- Apply CI/CD and infrastructure-as-code practices to make deployments repeatable and maintainable.
- Identify common migration risks and practical approaches to performance optimisation and cost control.
This session is designed for data engineers, cloud architects, analytics engineers, technical leads, and data platform professionals who are responsible for modernising enterprise data infrastructure and transitioning from traditional ETL systems to scalable, cloud-native data platforms.
Designing Reliable Real-Time Analytics Pipelines for Financial Services
Financial services organisations rely on timely, accurate, and trustworthy data to support fraud detection, transaction monitoring, risk management, regulatory reporting, and business decision-making. Designing real-time analytics pipelines for these environments requires more than low-latency data processing. It demands resilient architectures that can handle high-volume transaction streams while maintaining data consistency, security, auditability, and operational reliability.
In this session, we will explore the architecture and engineering principles behind reliable real-time analytics pipelines for financial services. We will examine how event-driven data ingestion, stream processing, cloud-native storage, and scalable analytical platforms can be combined to transform continuously arriving financial data into actionable insights.
The session will cover key architectural components, including event streaming, data ingestion, stream and micro-batch processing, schema management, data validation, and fault-tolerant pipeline design. We will discuss how technologies such as Apache Kafka, Apache Spark Structured Streaming, Azure Event Hubs, Azure Databricks, Delta Lake, and cloud data storage can support different real-time analytics requirements.
Particular attention will be given to financial services challenges, including duplicate transaction events, out-of-order data, late-arriving records, idempotent processing, replay and recovery, data reconciliation, and exactly-once processing semantics. We will also explore strategies for monitoring pipeline health, managing processing latency, implementing automated data quality checks, and maintaining end-to-end observability.
Beyond performance and scalability, the session will examine how security controls, data lineage, access governance, and audit trails can be incorporated into real-time architectures to support regulated workloads. Practical architecture examples will demonstrate how to balance latency, throughput, reliability, and cost while designing pipelines for demanding financial use cases.
**Key takeaways:**
- Design scalable, event-driven architectures for real-time financial analytics.
- Understand streaming and micro-batch processing patterns and their trade-offs.
- Handle duplicate events, late-arriving data, ordering issues, and pipeline recovery.
- Apply fault tolerance, idempotency, data reconciliation, and reliable processing strategies.
- Implement monitoring, alerting, data quality validation, and end-to-end observability.
- Integrate security, governance, lineage, and auditability into financial data pipelines.
- Evaluate latency, throughput, scalability, and cost when selecting an architecture for production workloads.
This session is designed for data engineers, cloud architects, analytics engineers, financial technology professionals, and data platform teams building reliable, secure, and scalable real-time data solutions for banking, payments, fraud detection, risk analytics, and regulatory reporting.
Responsible AI for Financial Risk Management: From Explainability to Production
Artificial intelligence is transforming financial risk management, but deploying AI in regulated environments requires more than model accuracy. Financial institutions must ensure that AI-driven decisions are explainable, transparent, auditable, and governed throughout the model lifecycle.
This article explores practical approaches to building responsible AI systems for financial risk management, covering explainability techniques such as SHAP and LIME, data quality, model governance, monitoring, and MLOps. It examines how organisations can move from experimental AI models to production-ready systems while maintaining transparency, regulatory compliance, and stakeholder trust.
The article provides a practical framework for engineering trustworthy AI solutions that balance predictive performance with explainability, governance, and operational reliability in high-stakes financial applications.
Building Production-Ready Data Pipelines with Microsoft Fabric
uilding reliable enterprise data platforms requires more than moving data between systems. Data engineers must design scalable architectures, implement robust transformation workflows, maintain data quality, and establish effective governance.
In this session, I will explore practical engineering patterns for building production-ready data pipelines using Microsoft Fabric. The discussion will cover the medallion architecture, Lakehouse, data ingestion, transformation, data quality validation, and governance.
Drawing on my experience in cloud data engineering and financial services, I will discuss how established engineering principles can be applied to modern data platforms. Through architecture diagrams and a practical demonstration, attendees will learn how to structure data pipelines, implement validation controls, manage transformations, and improve the reliability of downstream analytics.
The session will also examine common implementation challenges, architectural trade-offs, and practical considerations when designing maintainable data platforms for enterprise analytics and AI.
Key Learning Outcomes
Attendees will learn how to:
Structure data pipelines using Bronze, Silver, and Gold layers.
Understand the role of Microsoft Fabric Lakehouse in modern data architecture.
Apply data quality checks and validation practices.
Design maintainable transformation workflows for analytical workloads.
Identify governance and traceability considerations for enterprise data platforms.
Building Governed, AI-Ready Lakehouses on Azure and Databricks
Modern enterprises need more than scalable data pipelines. They need trusted, secure, and well-governed data platforms that support advanced analytics, regulatory reporting, and enterprise AI. Achieving this requires an architecture that integrates reliable data engineering, robust governance, data quality, and AI readiness from the ground up.
In this session, we will explore how to design and build a governed, AI-ready lakehouse using Microsoft Azure and Databricks. We will examine how Azure Data Lake Storage Gen2, Azure Data Factory, Azure Databricks, Delta Lake, and Unity Catalog work together to create scalable, secure, and production-ready data platforms.
The session will cover the implementation of a medallion architecture (Bronze, Silver, and Gold), data ingestion and transformation patterns, incremental processing, schema evolution, automated data quality validation, and metadata-driven governance. We will also explore how Unity Catalog enables centralized access control, data discovery, lineage, and consistent governance across enterprise data assets.
Beyond traditional analytics, we will discuss how to prepare governed lakehouse data for machine learning and generative AI applications. Key considerations include trusted datasets, secure data access, traceability, responsible data usage, and maintaining data quality throughout the lifecycle. Practical architecture examples will illustrate how these principles apply to enterprise environments where security, reliability, auditability, and regulatory compliance are essential.
**Key takeaways:**
- Design an end-to-end lakehouse architecture using Azure and Databricks.
- Implement scalable data pipelines using Azure Data Factory, Databricks, and Delta Lake.
- Apply medallion architecture principles to improve data quality and reliability.
- Implement data governance, access controls, and lineage using Unity Catalog.
- Establish trusted data foundations for machine learning and generative AI workloads.
- Understand real-world architectural challenges, implementation trade-offs, and best practices for production-grade data platforms.
This session is designed for data engineers, analytics engineers, cloud architects, data platform engineers, and technology professionals looking to build secure, scalable, and governed enterprise data platforms that are ready for the next generation of AI-driven applications.
Building Audit-Ready Cloud Data Pipelines for Financial Risk
Financial institutions depend on cloud data platforms to produce accurate, traceable, and auditable regulatory reports. However, building reliable pipelines in regulated environments involves more than moving data between systems. Engineers must address data quality, historical traceability, transformation logic, reconciliation, and governance while maintaining scalable and maintainable architectures.
In this session, I will explore practical engineering patterns for building audit-ready cloud data pipelines for financial risk and regulatory reporting. Drawing on experience developing data transformation workflows in financial services, I will discuss how modular dbt models, AWS cloud services, snapshot-based historical tracking, automated data quality tests, and structured reconciliation can help improve reporting reliability and reduce manual intervention.
The session will walk through a representative financial reporting architecture, from raw data ingestion and transformation to validation, historical tracking, and consumption by downstream reporting systems. I will also discuss common engineering challenges, implementation trade-offs, and practical approaches to making data pipelines more transparent, maintainable, and trustworthy.
Attendees will leave with an actionable framework for designing cloud-native data pipelines that balance scalability, data quality, auditability, and operational reliability. The principles discussed will be applicable to banking, insurance, financial analytics, and other environments where data accuracy and traceability are essential.
Building Enterprise Data Platforms with Azure Databricks and Microsoft Fabric
Modern enterprises need scalable, reliable, and cost-effective data platforms to transform raw data into trusted business insights. Building these platforms requires more than selecting cloud services; it involves designing effective architectures, implementing robust data pipelines, maintaining data quality, and establishing appropriate governance.
In this session, I will discuss practical approaches to building enterprise data platforms using Microsoft Azure, Azure Databricks, and Microsoft Fabric. We will explore data ingestion, transformation, lakehouse architecture, the medallion architecture pattern, data quality validation, and governance considerations.
Drawing on my experience in cloud data engineering and financial services, I will share practical engineering lessons, architectural trade-offs, and approaches to building maintainable data pipelines for enterprise analytics.
The conversation will also examine how organisations can improve pipeline reliability, manage data transformations effectively, and establish a foundation for trustworthy analytics and AI.
Key Discussion Points
Designing scalable cloud-native data architectures.
Implementing Bronze, Silver, and Gold data layers.
Using Azure Databricks, PySpark, and Delta Lake for data transformation.
Applying data quality checks and validation practices.
Managing data governance, security, and traceability.
Preparing enterprise data platforms for analytics and AI.
The Azure and AI Show User group Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top