Session

Tracking and Triggering Pattern with Spark Stateful Streaming

Inside Adobe Experience Platform we noticed that multiple times we need to track actions happening at the control plane level and act upon them at lower levels like Data Lake, Ingestion processes, etc. Using Apache Spark Stateful Streaming we've been able to create services that, based on rules and conditons, act by starting processes like compacting data, consolidating data, cleaning data but not limited here, at the proper time minimising the process time while keeping everything under the defined SLAs. The cost of operation is minimal as the applications/services did require no attention, they are reliable, offer exactly once execution through Spark Stateful Streaming, auto-scaling by the way the pattern is architected, and high resiliency in case of downstream dependencies failures. This talk presents a pattern that we've been using in production for the last 2-3 years inside Adobe Experience Platform in multiple services and with no high-severity on-call interventions and minimal-to-none operational costs in these years while the services where used on high throughput ingestions flows.

Andrei Ionescu

Senior Software Engineer, Adobe

Bucharest, Romania

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top