Problem:
Security data analytics serves a variety of downstream situational awareness goals including network monitoring, anomaly detection, attack detection, and machine learning. For instance, operators want to routinely check for “heavy hitters”, “new application patterns” or “anomalous trends in packet/flow distributions” that may be indicative of attacks. With rising volumes of data and an increased need for real-time analytics, cloud costs for security observability and analytics are spiraling out of control. We argue that to tame this cost, the world of security data analytics needs a fundamental shift in how these use cases are served by big data stacks, to provide low cost and accurate analytics.

Opportunity:
Sketches aka sketching algorithms provide an opportunity to reduce the cost of analytics, while providing accurate analytics. These algorithms are designed to accurately estimate statistical aggregates over data, such as percentiles, heavy hitters (i.e. most frequent items) and, cardinality (number of distinct items), at a fraction of the cost. Sketches also provide theoretical error guarantees backed by extensive scientific literature. Unfortunately, sketches are (a) are difficult to use, (b) require tuning low-level knobs to get optimal performance, and (c) are not well integrated with analytics frameworks.

Contribution:
Our research re-imagines big data analytics from an approximation-first lens and takes a fundamentally new approach to data analytics. We design SketchDB, a drop-in sketch-based optimizer that integrates with an existing big data deployment. SketchDB provides high-accuracy analytics with a fraction of the cost and latency of existing systems. Our initial experiments show that SketchDB can estimate queries with < 1% error and 10x lower latency, while consuming 10-30x lower memory while ingesting and querying data. Our grand vision is to reduce analytics costs by multiple orders of magnitude and democratize the use of approximation primitives like sketches.

Example Usecae and Deployment:
Consider an operator who wants to report the top 10 source IPs every minute based on the volume of traffic sent to a datacenter in the last hour. The state-of-the-art deployments will use Netflow to collect this data, which will then be ingested into a monitoring or observability tool such as ThousandEyes or Prometheus. Every minute, this tool will compute the top 10 source IPs based on the last hour of data and serve the results. While accurate, this is extremely costly and inefficient. Instead, SketchDB deploys a streaming precompute layer (such as using Apache Flink) that performs lightweight computation on the data as it is being streamed into the monitoring tool. When the query hits, SketchDB uses its query engine to quickly aggregate the precomputed results and answer the query, instead of having to compute on the raw data each time. This reduces CPU time, memory usage, query latency and energy consumption, all while providing approximate yet high accuracy analytics!