Arpit Garg
Sr. Principal Architect, Fourkites
Actions
Arpit Garg is a senior principal architect at FourKites where he is responsible for architecture of distributed and scalable global supply chain platform that tracks 3M+ shipments daily. Arpit has deep expertise in building data platforms, with a focus on cloud-native solutions, from his work at FourKites, Uber, Amazon, CDK Global and more.
How FourKites runs scalable and cost-effective Log Analytics Solution to handle Petabyte of logs
FourKites manages 900TB of logs. This presents significant cost challenges, as they rely on external monitoring solutions. Referencing these logs in context with other datasets from other data stores at FourKites is another key challenge.
In this talk, they will walk through how they built and manage cost-effective log analytics solutions, while covering:
1) Scaling log ingestion from VM's and Kubernetes nodes.
2) Strategic utilization of S3 along with Parquet compression to control storage cost and add scalability.
3) Using lambda functions for log transformation and extraction at scale.
4) Leveraging Trino as the distributed query engine to support search across the PB scale log datasets and join log data with other key datasets in FourKites.
5) How they used Superset as the frontend for Trino.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top