Hartmut Armbruster
Software Architect, Developer
Berlin, Germany
Actions
Hartmut Armbruster is a senior software engineer with experience designing and building solutions for high-load projects using distributed technologies. Over the past six years, he has been working in real-time data processing at HSBC, NEX Group plc, and Deutsche Bahn on mission-critical platforms. He strives to see the bigger picture and is passionate about architecture, combining, integrating, and bringing all things together.
Links
Area of Expertise
Topics
Maximising Cassandra's Potential: Tips on Schema, Queries, Parallel Access, and Reactive Programming
In this talk, we will design the backend and data layer for a typical data-rich example of a social platform feed for an authenticated user.
We will start with UI wireframes and move on to logical and physical Cassandra data models and query patterns. Using reactive programming paradigms, we will then optimise the process flow to perform queries efficiently in parallel.
Finally, we'll implement a POC using Kotlin, Quarkus and Mutiny!
Reactive programming can look intimidating, but it doesn't have to. It's productive, elegant and fun once you get used to it.
This talk intends to give new ideas and inspiration for what’s possible with a modern, tailored, efficiently utilised stack.
Prior knowledge of CQL, data partitioning/sharding concepts, and reactive programming is beneficial but optional.
Zero Downtime and Risk with Kafka Streams Blue/Green Deployments
Learn how Deutsche Bahn confidently deploys mission-critical Kafka Streams apps using blue/green deployments.
The customer information department processes real-time data, keeping travellers informed across various channels. Any downtime or delay in our data feeds immediately affects our customers, e.g. frozen displays on train platforms and outdated loudspeaker announcements throughout Germany.
Blue/green deployment is a well-known, established continuous delivery practice. It is widely used to deploy in frontend or "common API backend" applications, commonly supported out of the box by tools (such as Kubernetes) and SaaS platforms.
Applying this strategy to the world of stream processing poses numerous challenges.
Deploying the 'new' inactive environment may require data reprocessing to bootstrap an aggregated state. How can switching between the two environments be realised while preventing loss of events and keeping event duplication to a minimum?
This practical talk shares how our team has solved the architectural, functional, and procedural challenges. Expect a walk-through of the deployment workflow and how it has evolved, from implementations using Jenkins and Gitlab pipelines to our ongoing migration to a Kubernetes-native, GitOps-compatible solution based on Argo Workflows.
Keeping your State at Bay: Patterns to Limit Kafka Streams Store Sizes
The delivery service startup OPS (Otter Parcel Service) is efficiently managing its fulfillment process.
A team of skilled otters had written a small but mighty fleet of Kafka Streams topologies, and soon OPS was serving its first customers. All data streams ran smoothly, and operators and clients tracked parcel movements in real-time. Everyone was happy, and OPS was on the road to success.
But with the business growing, the otters started to notice the applications getting more and more unwieldy to operate by the day, and infrastructure costs kept increasing. The experienced otter team also knew the reason for that. Their streams state is growing larger and larger over time. The very first fulfilled delivery, which was completed six months back and idling away, is still in the state stores.
Join the otters in tackling this tech debt story to
- understand the implications and causes for the deterioration of operations
- find a solution to keep only the data that is still needed (unfulfilled deliveries)
- learn about different patterns to purge or evict entries from state stores and KTables, such as TTL-based cleanup to expire data
- review, evaluate, and compare implementations
Are you curious if the team can solve their problem and which strategy will prevail? See you at my session, fellow otter 🦦!
(Although this story is a work of fiction, the use cases and solutions presented are based on real-world examples.)
Elasticity vs. State? A New Kafka Streams State Store
'kafka-streams-cassandra-state-store' is a drop-in Kafka Streams State Store implementation that persists data to Apache Cassandra.
By moving the state to an external datastore the stateful streams app (from a deployment point of view) effectively becomes stateless. This greatly improves elasticity and allows for fluent CI/CD (rolling upgrades, security patching, pod eviction, ...).
It also can also help to reduce failure recovery and rebalancing downtimes, with demos showing sporty 100ms rebalancing downtimes for your stateful Kafka Streams application, no matter the size of the application’s state.
As a bonus accessing Cassandra State Stores via 'Interactive Queries' (e.g. exposing via REST API) is simple and efficient since there's no need for an RPC layer proxying and fanning out requests to all instances of your streams application.
Hartmut Armbruster
Software Architect, Developer
Berlin, Germany
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top