Using Gremlin API and Apache Airflow to facilitate Complex System Integrations

LongView, like many other businesses, has a complex system environment with many individual work management systems. A comprehensive integration layer needed to be implemented to facilitate data between business units and analytics.

After intensive planning we decided to utilize a database layer that could be independently updated by an integration with each system. The database layer would then become a map for any data that needed to be facilitated from one system to another. In pursuing that goal we ran into a massive problem. The data architecture needs for each system were complex in their own right, and we needed to build out the appropriate relationships between specific entities to properly model reality. Using SQL it would be so complex in its implementation that it would be unusable in any scalable manner.

Then we found Gremlin and Graph. Using Azure Cosmos DB's Gremlin API and Apache Airflow we were not only able to build the integration layer, but additional business applications have presented themselves. The integration layer has now become both a map across our system environment, allowing data to flow from one system to another without breaking continuity, and an operating system in its own right.

This talk will outline how we accomplished that with 1. extensive data modelling using a graph , 2. Ongoing development using Azure Cosmos DB's Gremlin API, and 3. The implementation of Apache Airflow for data orchestration.