Platform Overview: The Amadeus Hexadata Platform (AHP) is an advanced suite of capabilities for constructing and managing multiple End-to-End (E2E) big data pipelines.
- Processes approximately 2 billion transactions daily.
- Fully migrated from on-premise to Azure Cloud.
- Data size 10PB.
Data Integration:
- Integrates data from around 6,000 servers across various Amadeus divisions and external clients.
- 90% of data handled via Kafka, hosted on Azure or within a private data center.
- Utilizes Mirror Maker to replicate data across four Kafka clusters.
- Utilizes KafkaConnectors to stream files to Kafka
Technologies Used:
KStreams, Databricks, Kafka Connectors, Airflow, Internal Framework
Additional Capabilities:
Operational on OpenShift. (4000 Pods)
ElasticSearch
MLOPS pipeline : CICDCD for ML models on the platform in Azure. - on-going -
Key Benefits:
Provides scalable, reliable, and efficient big data processing and analytics solutions.