FourKites manages 900TB of logs. This presents significant cost challenges, as they rely on external monitoring solutions. Referencing these logs in context with other datasets from other data stores at FourKites is another key challenge.

In this talk, they will walk through how they built and manage cost-effective log analytics solutions, while covering:
1) Scaling log ingestion from VM's and Kubernetes nodes.
2) Strategic utilization of S3 along with Parquet compression to control storage cost and add scalability.
3) Using lambda functions for log transformation and extraction at scale.
4) Leveraging Trino as the distributed query engine to support search across the PB scale log datasets and join log data with other key datasets in FourKites.
5) How they used Superset as the frontend for Trino.