Achieving a strong & sustainable financial governance across AWS cloud ecosystem.

Document Link - https://docs.google.com/document/d/1zYILy77Ck3PXuEcfeKTUTocNqQ8G96mT/edit#

Background
Being a data-driven business, MiQ maintains an extensive data infrastructure.
At MiQ, on average, we onboard 15+ TB of compressed Adtech data daily into our infrastructure, with real time and batched workloads comprising approximately 300 pipelines. We are running AWS EMR, Apache Spark, Presto, Apache Hive, AWS Redshift, AWS Kinesis & Firehose, AWS Athena, Apache Airflow, Streamsets, Apache Nifi. MIQ, being a medium sized company, it is imperative to achieve sustainable growth at scale so managing cost and using resources efficiently is of great importance to us.

OKR for Financial Governance - “0% deviation from P&T budget for Tech infrastructure“
Annual Cost Forecasts
At the beginning of every year we have our product and tech budgets estimated and precise understanding of cost are key to ensure the estimations are correct and helps to forecast infrastructure cost growths in comparison to topline and bottomline numbers. We have cost related OKRs to make sure that everyone in the Org is aligned and to bring in a culture where teams are aware of the costs associated with infra. The AWS Pricing Calculator helps forecast precise estimates for new services / products being built on AWS as well as and our historical data here is equally helpful in forecasting.

Cost Tracking Tools & Techniques
Once budgets are approved we try our best to track and keep an eye on the cost trends over a weekly and monthly basis. For this purpose we have built an in-house Cost Reporting Service which integrates with AWS Cost Explorer, Qubole Cost Reporting Reports and Databricks log level cost data and many such third party providers that we work with. This sends us weekly and monthly automated reports and deviation alerts if any. Based on the alerts, if someone wants to debug anything in detail, we leverage a combination of AWS CloudTrail Insights, CloudTrail, S3, Athena. We also use AWS Budgets & CUR reports extensively for alerting and reporting aspects.

As a result of all the tools and techniques discussed above, teams have visibility and awareness of cost implications of any new service / product / pipeline development and productionizing. Leadership team also has visibility to high level dashboards and reports at all levels of granularity as they like to view.

Cost Dashboards (1,2,3) & Weekly Team Wise Cost Report in Mail (4)

Talking of some of the cost optimizations practices we have followed in the last few years. We generate periodic reports giving us average utilization of all the different Machines, Kube Pods, Databases, Caches, etc and do time to time calibration in terms of right sizing of resources, using the right instance types. We have utilized AWS Cost Savings Plan and Reserved Instance Types to have more stability and efficiency in terms of cost as well as performance. As indicated earlier, MIQ being a data-driven company we use different SPOT machine offerings from AWS & AWS Glacier very heavily because of the reporting nature of our workloads.
Impact
After following the above practices and discipline in the last 2-3 years we have been able to grow as a business in revenue but still have minimal to negative deviation in Product and Tech approved budgets in year start. Few key results we have been able to achieve are :
0 or -ve deviation from our estimated costs
>25% reduction in AWS cost in the first year of tracking & right sizing implementation
100% awareness / visibility among different teams regarding cost applicability of their products and service offerings.

Many of the above mentioned techniques can be followed by any company with a small amount of effort and the cost savings can be big and might surprise you. “Because at the end of the day, the dollars saved are dollars earned”.