James Bowkett
Technical Delivery Director, Open Credo
London, United Kingdom
Actions
James is the Technical Delivery Director at OpenCredo. He has over 20 years of experience as a lead developer delivering data-centric applications in the finance industry at various banks, hedge funds and startups.
He has presented at a number of industry conferences around data technologies. He cares deeply about design and building quality into the heart of all software and is a firm advocate of XP practices such as TDD and BDD.
Kafka, Devops ...And Resilience for all
So you’ve made the enlightened choice to use Kafka in your architecture, and you know you need to provision your cluster using infrastructure as code (IAC), but why stop at just provisioning brokers?
In this talk we explore the available options to make deploying your Kafka-based applications more repeatable, resilient and observable. We’ll look at specific examples and techniques for applying IAC to the following areas :
Topic and partition creation
Secrets management
Configuration options
I’ll also share some lessons I’ve learned related to tooling, monitoring and backup management.
And finally we’ll look at some resilience do’s and don’ts with some specific examples of where these tools have helped, and where their absence has hindered.
Tracing Your data's DNA
As data becomes ubiquitous and deeply interconnected, tracing where, who or which system that data comes from - its lineage - will create bigger problems and opportunities for us on the horizon:
• How can we trust this document/row of data? what is its lineage? where did it come from?
• If there is a problem with a piece of data, how do we recalculate and publish just the affected data and not the entire dataset?
• How can we apply modern engineering practices - such as blue-green deployments - to our data estate and data pipelines?
Furthermore, as our data estate becomes ever-more business-critical, it will be important to be able to secure that data from its source system all the way through the estate, using techniques such as field+row level security (aka cell-based security).
In this talk I will use live demos and coding examples to explore some techniques of how to create the data lineage graph of individual rows or documents using Change Data Capture (CDC) in source systems. We will store the lineage graph within a graph Database to start with, then explore how other types of database could be used instead. This will create a lineage catalogue that can be queried for all manner of use cases, such as incremental data batch operations, blue-green deployments and "cell-based security" of data fields.
Neo4j Online Developer Education Summit (NODES) 2022 Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top