Netflix's Secret to Cloud Efficiency
How Netflix uses data to reduce its cloud costs
TL;DR
Situation
Netflix relies on AWS for its cloud infrastructure, supporting a vast array of applications and services. This extensive ecosystem generates large volumes of data, necessitating efficient resource management to optimize costs and performance.
Task
The Platform Data Science and Engineering (DSE) team was asked to build a tool that could show engineers how much cloud resources they were using, how efficient they were being, and how much it was costing. The goal was to help teams make smarter and more cost-effective decisions.
Action
The team developed a two-component solution:
Foundational Platform Data (FPD): This centralized data layer offers a consistent data model and standardized processing methods by establishing data contracts with various platform data providers. This ensures data quality and reliability across the board.
Cloud Efficiency Analytics (CEA): Built atop FPD, CEA delivers time-series efficiency metrics tailored to diverse business needs. It provides detailed cost and ownership attribution, enabling teams to understand their resource consumption and associated expenses.
Result
Netflix’s teams now have access to clear and trustworthy data about their cloud usage and costs. This helps them make better decisions about how to use resources, which improves efficiency and saves money.
Use Cases
Cloud Resource Usage, Cost Attribution, Efficiency Analysis
Tech Stack/Framework
Apache Spark, Data Contracts