The arduous path to CDC with Oracle at large scale, low cost, high performance, and seamless tunning

Data is like tiny drops of water that alone cannot influence decision-making. When millions of water drops are deposited in the same place, the force they emanate can produce energy to power entire cities.

Oracle is one of the most complex and challenging environments for collecting and capturing data. The world is currently going through difficult times with reduced investments and demanding software engineers and integration specialists to create mechanisms that allow the decoupling of legacies.

Change Data Capture (CDC) can be considered a widely disseminated strategy, studied, and presented to the market as a solution for data collection. However, it is more complex to implement than described in blogs, books, or even online tutorials.

We will show all the real challenges we have experienced in the last two years, collecting real-time data from Oracle databases at a rate of up to 280 thousand transactions per second(TPS).

We can collect more than 1TB events in this environment over an atypical sales day (Brazil Black Friday) without impacting day-to-day operations. We will present hacks and tuning lessons for optimizing Logminer to collect this data volume.

This use case explains our challenges in collecting data from textile manufacturing, retail, and SAP ERP systems.