Airbnb’s Platform: Real-Time Data Meets Personalisation
How Airbnb handles 1 Million events per second for scalable personalisation
TL;DR
Situation
Airbnb aimed to improve personalization by processing user engagement data during activities like browsing and booking, requiring a scalable, real-time processing platform.
Task
Build a platform to process and store real-time and historical user data, support low-latency serving, and allow non-expert teams to define data pipelines.
Action
Developed the User Signals Platform (USP) with a Lambda architecture, combining real-time processing and batch corrections. Simplified workflows enable teams to define data transformations through configurations.
Result
The USP now processes over 1 million events per second across 100+ Flink jobs, supporting personalization at scale. Its service handles 70k queries per second, empowering teams to deliver real-time insights and personalized experiences.
Use Cases
Personalised Recommendation, User Segmentation, Latency Monitoring
Tech Stack/Framework
Apache Kafka, Apache Flink, Apache Hive
Explained Further
Architecture Overview
USP employs a Lambda architecture comprising two main layers: