Today we will look at how Notion scaled its AI Q&A to millions of users.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Notion’s AI Q&A level up!

TL;DR

Situation

Notion launched AI Q&A on top of vector search and quickly faced massive demand across millions of workspaces. The initial system hit limits in capacity, onboarding speed and cost.

Task

Scale onboarding, keep indexes fresh and reduce rising infrastructure costs. At the same time, simplify a growingly complex architecture without hurting latency.

Action

They introduced dual ingestion paths, generation-based indexing, serverless architecture and migrated to turbopuffer. Then reduced recomputation with page state tracking and moved embeddings to Ray for unified compute.

Result

600x onboarding growth, 15x workspace growth and major cost reductions across layers. Latency improved and the system became simpler and more efficient.

Use Cases

Real-time search indexing, semantic search, document retrieval

Tech Stack/Framework

Apache Spark, AWS EMR, Apache Airflow, Apache Kafka, AWS S3, DynamoDB, Ray, turbopuffer

Explained further

Context

When Notion launched AI Q&A in November 2023, the core idea sounded simple enough: let people ask natural-language questions and retrieve relevant knowledge from across their workspace and connected tools. In practice, that meant building a vector search system that could ingest huge amounts of content, stay fresh as pages changed and do all of it at a cost that made sense at Notion scale.

That is the real story here. Not just “vector search powers AI” but what happens after launch, when adoption jumps faster than expected and the infrastructure underneath has to keep up. Over two years, the Notion team pushed that system through several big transitions: scaling onboarding, dealing with storage pressure, changing database architecture, reworking indexing logic and moving embeddings workloads onto Ray. The headline numbers are hard to ignore: 10x scale and roughly one-tenth the cost.

This is a good example of how modern AI infrastructure usually evolves. The first version gets the product live. The next few versions are about survival, then simplification, then cost, then latency, then getting rid of all the awkward bits that built up during the rush.

Vector search, explained through Notion’s lens

Traditional keyword search is literal. It works when users type the exact words that exist in the content. It starts falling apart when the wording changes but the meaning stays the same. Someone searching for “team meeting notes” may still want a page called “group standup summary,” but keyword search does not naturally understand that those are closely related.

Vector search solves that by representing text as embeddings. Instead of storing only words, it maps text into a high-dimensional space where semantically similar ideas sit closer together. That means retrieval is based on meaning, not exact phrasing.

For Notion AI, this matters a lot. The system needs to answer questions in natural language by finding useful content across a workspace and even across connected sources like Slack and Google Drive. That is exactly the sort of setup where semantic retrieval becomes more useful than plain lexical matching. A user is not thinking about the title of the page or the exact phrasing inside a paragraph. They are asking a question in their own words and expecting the system to bridge the gap.

That expectation becomes expensive very quickly.

Part 1: Scaling beyond what the original system expected

At launch, Notion’s ingestion and indexing pipeline had two paths.

The first was an offline path. Batch jobs running on Apache Spark would chunk existing documents, generate embeddings through an API and bulk-load those vectors into the vector database. This handled the heavy lifting for backfilling large amounts of existing content.

The second was an online path. Kafka consumers processed page edits in near real time so live workspaces stayed up to date with sub-minute latency.

It is a practical split. The offline side handles the backlog and large initial loads. The online side keeps things fresh once a workspace is active. Together, the two-path setup gave Notion a way to onboard workspaces at scale without sacrificing freshness for day-to-day edits.

The vector database itself ran on dedicated ‘pod’ clusters, where storage and compute were coupled. The Notion team designed sharding in a way that echoed their Postgres setup: workspace ID was the partitioning key, routing used range-based partitioning and a single config referenced all shards.

That all made sense on paper. Then the product launched and demand was overwhelming.

Notion quickly built up a waitlist of millions of workspaces that wanted access to Q&A. The problem was no longer whether the system worked. It was how fast it could onboard people without cracking under the pressure.

When the indexes started to fill up

Only a month after launch, the original indexes were already nearing capacity.

That is the kind of problem that sounds good in product meetings and bad in infrastructure meetings. If the indexes filled up, Notion would have to pause onboarding. That would slow down rollout and delay access for everyone waiting.

The team had two obvious options.

One was to re-shard incrementally. Clone data into another index, delete half, repeat and keep doing that every couple of weeks as new customers came in.

The other was to re-shard for the final expected volume. But their vector database provider charged for uptime, so over-provisioning would have been painfully expensive.

Instead, the Notion team went with a third approach. When a set of indexes got close to full, they provisioned a new set and directed all newly onboarded workspaces there. Each set was assigned a generation ID, which determined where reads and writes should go.

It is not the prettiest long-term design, but it was a smart short-term move. It avoided repeated re-shard operations and kept onboarding moving. Sometimes the right scaling decision is not the most elegant one. It is the one that buys breathing room without stopping the business.

Turning onboarding into a throughput problem

Even with the architecture in place, the initial onboarding rate was nowhere near enough. At launch, Notion could onboard only a few hundred workspaces per day. At that pace, clearing a multi-million waitlist would have taken decades which is obviously not a real option.

So the team pushed hard on throughput. Using Airflow scheduling, pipelining and Spark job tuning, they dramatically increased capacity.

The results were big:

• Daily onboarding capacity increased by 600x

• Active workspaces grew 15x

• Vector database capacity expanded 8x

By April 2024, the Q&A waitlist was cleared.

That is the kind of milestone that looks clean in hindsight but it came with a cost. Managing multiple generations of databases helped during the hypergrowth phase but it also added operational complexity and financial overhead. The team had solved the immediate scaling problem, but the architecture was starting to feel heavy.

That set up the next phase of the story.

Part 2: Cost becomes the next constraint

In May 2024, Notion migrated its embeddings workload from the original dedicated ‘pod’ architecture to a serverless setup that decoupled storage from compute and charged based on usage instead of uptime.

The effect was immediate. Costs dropped by 50 percent from peak usage, translating into several millions of dollars in annual savings.

That alone would have made the migration worthwhile, but the serverless design also fixed two practical problems. First, it removed the storage capacity constraints that had become a serious scaling bottleneck. Second, it simplified operations because the team no longer had to provision capacity ahead of demand.

Still, even after cutting costs in half, the annual run rate for vector database spend was still in the millions. From an engineering point of view, this is where things get interesting. The easy win had already happened. Now the team had to go after deeper structural gains.

A new search foundation (turbopuffer)

While working on the first round of savings, Notion also evaluated alternative search engines. turbopuffer stood out because it offered significantly lower projected costs.

At the time, turbopuffer was a newer player in search. Its architecture was built on object storage with a focus on cost-efficiency and performance. It also supported both managed and bring-your-own-cloud deployment models and it made bulk modification of stored vector objects easier.

That combination lined up well with what Notion needed.

After a successful evaluation, the team decided to migrate its entire multi-billion-object workload to turbopuffer in late 2024. Since they were already making a provider switch, they used the migration as a chance to clean up the broader architecture too.

Several changes happened together.

First, they fully re-indexed the corpus, increasing write throughput in the offline indexing pipeline to rebuild everything in turbopuffer.

Second, they upgraded the embeddings model during the migration to be more performant.

Third, they simplified the architecture. turbopuffer treats each namespace as an independent index which removed the need to think about sharding and generation-based routing in the same way as before.

Finally, they handled the cutover gradually, migrating one generation at a time and validating correctness before moving on.

This is a strong pattern: if a migration is painful anyway, use it to pay off other infrastructure debt at the same time.

The outcome was solid on several fronts:

• 60 percent cost reduction on search engine spend

• 35 percent reduction in AWS EMR compute costs

• p50 production query latency improved from 70–100ms to 50–70ms

That is a meaningful improvement across cost and performance, which is not always easy to pull off together.

Avoiding full reprocessing with page state tracking

The next optimization went after a very expensive inefficiency in the indexing pipeline.

Notion pages can be long, so the team chunks each page into spans, embeds each span and stores those vectors with metadata such as authors and permissions. In the original implementation, any edit to a page or its properties triggered a full re-chunk, full re-embed and full re-upload of all spans on that page.

That meant even a tiny change could trigger a lot of unnecessary work.

The team narrowed the problem down to two things that actually mattered:

1. The page text changes which means embeddings need updating

2. The metadata changes which means metadata needs updating

To detect those cases, they tracked two hashes per span: one hash for the span text and another for the metadata fields. They chose 64-bit xxHash because it offered a good balance of speed, simplicity, low collision risk and storage footprint.

For caching, they used DynamoDB. Each page had one record containing the state of all spans on that page, including text and metadata hashes.

The win came from using that state to avoid unnecessary work.

Case 1: The page text changes

Imagine Herman Melville editing Moby Dick halfway through a page. Before this improvement, the whole page would have been re-embedded and reloaded. After the change, the system chunks the page, fetches the previous state from DynamoDB and compares text hashes span by span. It can then detect which spans actually changed and only re-embed and reload those.

That is the kind of fix that getting the balance right matters. Miss a changed span and search quality suffers. Reprocess too much and cost stays high.

Case 2: The metadata changes

Now imagine Melville updates permissions so the page becomes visible to everyone. The permissions metadata changes but the text does not.

Previously, that still meant re-embedding and reloading the entire page. With the new approach, Notion compares both text and metadata hashes. If the text hashes are unchanged but metadata hashes differ, the system skips embedding entirely and issues a PATCH command to the vector database to update only the metadata. That is much cheaper than recomputing embeddings.

Across these changes, the Page State Project reduced data volume by 70 percent. That saved money on both embeddings API costs and vector database write costs.

Moving embeddings to Ray (indexing)

In July 2025, Notion started migrating its near real-time embeddings pipeline to Ray on Anyscale.

The motivation came from several pain points in the earlier setup.

One was the ‘double compute’ problem. Spark on EMR handled preprocessing like chunking, transformations and API orchestration, but embeddings themselves were still generated through an external provider that charged per token. So the team was paying for both preprocessing infrastructure and embedding API usage.

Another issue was endpoint reliability. Fresh search indexes depended on the stability of an external embeddings API.

The third problem was clunky pipelining. To smooth traffic and avoid API rate limits, the team had built a multi-step handoff process where Spark jobs passed batches through S3. It worked but it was clunky.

Ray and Anyscale gave Notion a cleaner path.

Ray let the team run open-source embedding models directly, which meant more model flexibility and less dependence on external providers. By consolidating preprocessing and inference onto a single compute layer, they could cut out the double-compute setup. Ray also supports pipelining CPU-bound work such as chunking and page-state detection with GPU-bound embedding generation on the same nodes, which helps keep utilization high.

There was also a developer productivity angle. Anyscale workspaces let engineers write and test pipelines from their preferred tools without having to provision infrastructure manually.

And on the product side, self-hosting embeddings removed a third-party API hop from the user-facing path, which helped reduce end-to-end latency.

The rollout is still ongoing, but early results suggest a 90+ percent reduction in embeddings infrastructure costs. That is a major shift in how the economics of the system work.

Real-time query embeddings on Ray (serving)

Indexing is only half the picture. When users or agents search in Notion, queries must also be embedded on the fly before the vector database can be searched.

That makes serving latency-sensitive. The embedding has to happen fast enough that the search still feels responsive.

Hosting large embedding models is not trivial. GPU allocation, ingress routing, replication and autoscaling all matter, especially when traffic is uneven and expectations for responsiveness are high.

Ray Serve helped Notion here by handling much of that operational layer out of the box. The team could wrap open-source embedding models in persistent deployments that stay loaded on GPU, configure request batching and replication and manage the serving setup with normal Python code plus YAML-based infrastructure configuration.

That is a pretty practical endpoint for the broader journey.

What started as a vector search stack built quickly enough to launch AI Q&A turned into a much more refined system: simpler in some places, more selective in others, cheaper across multiple layers and faster where users feel it. The interesting part is not any single tool choice. It is how the Notion team kept removing bottlenecks one by one: storage limits, awkward shard routing, redundant recomputation, external API dependence and fragmented compute layers.

That is usually what mature AI infrastructure looks like in the real world. Not one giant redesign. A sequence of sharp decisions, each fixing the thing that has become too expensive, too slow or too annoying to keep around.

The full scoop

To learn more about this, check Notion's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how LinkedIn ingests data at scale.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Venice: LinkedIn’s data storage platform

TL;DR

Situation

Venice powers LinkedIn’s AI-driven products and has scaled to 2,600+ stores with workloads spanning bulk loads, streaming updates and active/active replication. The ingestion pipeline had to handle throughput-heavy, CPU-heavy and latency-sensitive traffic under eventual consistency.

Task

Redesign ingestion to scale to 230M writes/sec while preserving ordering and protecting read and write SLAs. Support hybrid stores, partial updates and multi–data center replication without destabilizing clusters.

Action

Scaled bulk ingestion with partition tuning, shared consumer/writer pools and direct SST writes; tuned RocksDB via compaction triggers and BlobDB to manage amplification. Optimized CPU-heavy paths using Fast-Avro and parallel processing, then enforced priority pools and adaptive throttling to protect current-version latency.

Result

Venice now handles 175M+ key lookups/sec and 230M+ writes/sec in production. It maintains a write latency SLA under 10 minutes while safeguarding read latency as the top priority.

Use Cases

Large-scale feature stores, real-time recommendation systems, hybrid data serving, low-latency notification

Tech Stack/Framework

Apache Spark, Apache Samza, Apache Kafka, RocksDB, Fast-Avro, Adaptive Throttling

Explained further

Background

Venice is an open-source derived data storage platform and LinkedIn’s default storage layer for online AI use cases. It sits behind products like People You May Know, feed, videos, ads, notifications, the A/B testing platform, LinkedIn Learning and more.

Since Venice launched internally in 2016 it has scaled from a handful of stores to over 2,600 production stores. The workloads also evolved a lot. It started with “just bulk load a dataset” and grew into a mix of:

• Bulk loading huge offline datasets

• Nearline streaming updates

• Active/active replication across data centers

• Partial updates that merge fields and collections

• Deterministic write latency expectations under eventual consistency

This post walks through how the ingestion pipeline was revamped to hit 230 million records per second in production, what changed across the architecture, which optimizations moved the needle and how different workload types get tuned. A lot of these ideas are portable if you run any distributed ingestion system where ordering, throughput and predictable latency all matter at once.

Venice overall ingestion pipeline

At a high level, store owners write to Venice through three paths:

1. Bulk loads from an offline processing platform (example: Spark)

2. Nearline writes from a streaming processing platform (example: Samza)

3. Direct writes from online applications

No matter which path you take, the writes all pass through an intermediate PubSub broker layer. From there, the Venice Storage Node (VSN) consumes messages and persists data locally using RocksDB (an embedded key-value store).

The pipeline sounds straightforward until you operate it at scale. The same ingestion path has to support very different workloads. Some are throughput-driven (bootstrapping a massive store). Some are latency-driven (current-version updates). Some are CPU-heavy (partial updates and conflict resolution). Some are I/O-heavy (compaction, SST churn).

The following sections will look at the challenges and how the LinkedIn team resolved them.

Use case 1: bootstrapping from offline dataset

Venice users can run bulk load jobs using offline processing platforms such as Spark to push new data versions to Venice stores. The hard part is performance for large or massive stores. If you want to find bottlenecks you need to understand the ingestion path end to end.

What happens during a bulk load

• A Venice Push Job (VPJ) creates a new version topic for the new store version, split into multiple partitions

• The Spark job uses a map-reduce framework to produce messages to that version topic

• It keeps one reducer per topic partition so message ordering is preserved

• On the other side, the VSN spins up consumers, reads messages and persists them into RocksDB

• There is one RocksDB instance per topic partition

So you can hit bottlenecks in three obvious places:

1. producing

2. consuming

3. persisting

Production experience says you will hit all three, just not on the same day.

Improving producing and consuming throughput

The usual first lever is increasing the number of partitions for large stores so you can use more of the PubSub cluster capacity. More partitions tends to mean more parallelism and more throughput.

But it comes with trade-offs:

• more partitions means more management overhead across Venice and PubSub

• there is a throughput ceiling per PubSub broker

So partition count is not a free lunch. It’s a knob that buys you throughput and charges you complexity.

Enhancing consumption scalability

To keep up with production, VSN uses shared consumer pools across all hosted stores.

Instead of “one store version, one set of consumers,” each store version can use multiple consumers by distributing hosted partitions among them. The point is to keep multiple connections per PubSub broker to speed up consumption (similar to a Download Manager).

The pool approach also does something boring but important: it sets an upper limit on total consumers which puts a ceiling on cost.

Optimizing I/O performance

VSN uses a shared writer pool to persist changes concurrently across multiple RocksDB instances and use local SSD capacity effectively.

Ordering is critical in Venice so for any given RocksDB instance there is only one writer actively writing to it. You still get concurrency across instances, not inside one instance which is the compromise that keeps ordering intact.

Minimizing memory overhead

Because messages for a partition are strictly ordered (thanks to the map-reduce framework), Venice uses RocksDB’s SSTFileWriter to generate SST files directly. That significantly reduces memory overhead during ingestion.

Ingestion workflow in Venice Server

Put together, the optimized workflow is basically: use the PubSub layer for distribution, use consumer pools for scalable reads, use writer pools for SSD throughput, preserve ordering by design and avoid memory blowups by writing SST files directly.

Use case 2: hybrid store

Venice supports Lambda architecture style use cases by merging updates from both bulk loads and nearline writes. Users query a single store and get a unified view.

Venice hybrid store workflow

How it works:

• each bulk load creates a new store version

• that version has a new Kafka topic and a new database instance

• real-time updates produced by a Samza job via a real-time topic are appended to both version topics to keep them current

• once the new version catches up fully, it is swapped in as the active version to serve reads

The hybrid store is important because it gives you a clean “new version build” story without losing real-time freshness. But it creates a new challenge: the database transitions from read-only to read-write.

That’s where RocksDB tuning matters, because duplicates start showing up more often. Keys get updated or deleted after they were inserted. RocksDB uses log compaction to remove stale entries, but that compaction has overhead: scan, merge, rewrite SST files, consume CPU, I/O and disk.

So the core problem becomes: tune RocksDB so you can balance three competing types of pain.

• Write amplification: bytes written to storage vs bytes written to the DB

• Read amplification: number of disk reads per query

• Space amplification: size of DB files on disk vs the actual data size

Venice uses leveled compaction by default and relies primarily on two methods to balance those trade-offs.

1. Tuning the compaction trigger

The key setting here is:

• level0_file_num_compaction_trigger

This controls the max number of files allowed in Level-0. Once you exceed it, compaction kicks in to push SST files from Level-0 to Level-1 and onward as upper levels fill.

Why it matters:

• higher threshold → fewer compactions → lower write amplification

• but also more Level-0 files → higher read amplification since reads may need to scan multiple files

• plus higher space amplification because duplicates hang around longer

Venice tunes this per cluster because clusters have different bottlenecks:

• memory-serving clusters want data in RAM to speed up lookups. Memory is the limiting resource, so they set a lower threshold to reduce space amplification

• disk-serving clusters are often limited by disk I/O, so they set a higher threshold to reduce compaction frequency and lower disk write rate

This is a practical tuning philosophy: tune to your real bottleneck, not a generic best practice.

2. RocksDB BlobDB integration

BlobDB is aimed at large-value workloads through key-value separation:

• Large values go into blob files

• LSM tree stores small pointers

This avoids copying large values repeatedly during compaction, reducing write amplification. The cost is additional space amplification because blobs can become unreferenced and require garbage collection.

For Venice, BlobDB integration reduced write amplification significantly in multi-tenant clusters, especially for large-value use cases. The reported impact here is big: more than a 50% reduction of disk write throughput. That matters because it avoided scaling out clusters when CPU and storage space were still available.

The win here is: you stop paying the compaction tax over and over on the same large payloads.

Use case 3: Active/active replication with partial update

Venice guarantees eventual consistency, not strong consistency. That matters because it means you cannot just do read-modify-write operations directly due to write delays.

To handle this, Venice introduces partial update, a specialized operation that supports field-level updates and collection merges.

Venice partial update workflow

Inside the Venice server, the leader replica:

• decodes the incoming payload

• applies the update

• re-encodes the result

• writes to the local database

• writes to the Version Topic

• follower replicas consume the merged results

Most of that is CPU-heavy.

Then the platform evolved further with active/active replication across multiple data centers. The key mechanism is deterministic conflict resolution (DCR), similar to CRDTs. Venice tracks update timestamps at row and field levels, compares incoming timestamps with existing ones and decides to apply or skip.

Venice Active/Active workflow

Now the leader replica has even more to do for DCR:

• timestamp metadata lookup

• decoding

• encoding

Again: CPU heavy. So the optimisation below focus on CPU efficiency.

1. Fast-Avro adoption

Fast-Avro was originally developed by RTBHouse but LinkedIn took over maintenance under the LinkedIn namespace and introduced many optimizations.

The key idea: Fast-Avro is an alternative to Apache Avro serialization and deserialization using runtime code generation which performs significantly better than the native implementation. It supports multiple Avro versions at runtime and is widely adopted inside LinkedIn.

Venice fully integrated Fast-Avro and saw, in one major use case, up to a 90% improvement in deserialization latency at p99 on the application side.

2. Parallel processing

In the traditional pipeline, DCR and partial update operations were executed sequentially, record by record within the same partition. That leads to CPU underutilization.

Venice introduced parallel processing so multiple records can be handled concurrently within the same partition before producing them to the version topic, while still preserving strict ordering in the final step.

Result: significantly improved write throughput for these complex record types.

Use Case 4: Active/active replication with deterministic write latency

Eventually consistent systems still get judged by human expectations. People want their writes to show up and they want it to happen predictably.

Venice is versioned and can ingest backup, current and future versions concurrently in a single server instance. In practice though, only the current version serves reads so deterministic write latency guarantees focus mostly there.

To improve determinism, Venice introduced a pooling strategy in ingestion with different priorities for different workload types. The Venice consumer phase is the first phase in the server ingestion pipeline and controlling the polling rate via pools is how prioritization happens.

Broad priority tiers:

• top priority: active/active and partial update workloads for the current version on the leader replica (CPU-intensive and latency-sensitive)

• next: other workload types targeting the current version

• then: active/active or partial update workloads for backup or future versions on the leader replica

• finally: everything else in a lower-priority bucket

This design is trying to do a few practical things:

• isolate CPU-heavy workloads so they don’t slow down lighter ones

• prioritize the current version so the most up-to-date data flows smoothly

• keep the number of pools limited to avoid resource management turning into a second job

The catch is tuning. Clusters see different workloads, store behavior varies widely even within one cluster, throughput swings over time and read traffic changes throughout the day. Static configs force you to tune for worst-case, which wastes resources most of the time.

So Venice introduced adaptive throttling: dynamically adjust ingestion based on recent performance.

• if the system is within agreed SLAs, ingestion rates are adjusted according to priorities

• if an SLA is violated, ingestion is throttled back immediately

Defining the SLAs matters. Venice focuses on two key criteria:

1. Read latency SLA: highest priority. Never violate read latency SLAs, even if it costs ingestion throughput

2. Write latency SLA for the current version: while read latency SLAs are met, write latency for the current version becomes top priority, pools are tuned proportionally to maximize utilization and throughput

Wrapping up

With these optimizations, Venice at LinkedIn handles:

• Over 175 million key lookups per second

• Over 230 million writes per second

• While maintaining a write latency SLA under 10 minutes

The full scoop

To learn more about this, check LinkedIn's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Grab detects data issues in real-time.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Grab’s real-time work!

TL;DR

Situation

Grab runs critical systems on Kafka streams, where bad data can spread and break downstream consumers. Existing checks were slow and mostly limited to schemas, making issues hard to catch and debug.

Task

Detect bad streaming data early, cover both schema and value-level issues and give stream owners fast, actionable visibility without centralising ownership.

Action

Grab built contract-driven stream checks on Coban, turning schemas, field rules and ownership into real-time FlinkSQL tests with Slack alerts and UI-based inspection of bad records.

Result

The system now monitors 100+ Kafka topics in real time, surfaces poison data quickly and helps teams stop issues before they cascade downstream.

Use Cases

Root cause analysis, real-time monitoring, real-time alerting

Tech Stack/Framework

Apache Kafka, Apache Flink, Amazon S3, Slack, LLM

Explained further

About Grab

Grab is often called the Uber of Southeast Asia but that might be selling it short. What started as a ride-hailing app now powers food delivery, groceries, payments and even insurance all bundled into one super app. They run across over 800 cities in 8 Southeast Asian countries. Behind the rides, meals, and payments lies an enormous stream of events flowing through Grab’s systems.

Background

Grab runs a lot of business on streaming data. Kafka topics feed online systems, offline analytics and machine learning pipelines. When those streams are clean, life is good: teams can move faster, models behave, dashboards run smoothly. But when they’re not clean, it’s a major headache.

The tricky part is that ‘bad data’ in Kafka isn’t always obvious. Sometimes it’s quiet: the stream still parses but key fields are wrong, missing or shaped differently than what downstream teams assume.

That’s why Grab decided to introduce a platform-level solution: Kafka stream contracts that let stream stakeholders define what ‘good’ looks like, then automatically test streams in real time, catch issues as they happen and alert the owners quickly.

The core idea is simple:

• Let users define a data contract for a Kafka topic

• Convert that contract into executable tests

• Run those tests continuously

• Capture the poison data plus context

• Notify the right people with enough detail to act

This supports a more decentralized, data-mesh style world where teams own their data products while still keeping the overall system reliable for everyone else.

What wasn’t working before

Historically, monitoring Kafka stream data processing didn’t have a strong, end-to-end solution for data quality validation. That created three big issues: detecting bad data, speed of detection and lack of visibility.

1- Detecting bad data

This can be broken down into two further categories:

1.1 Schema issues

These are schema mismatches between producers and consumers that can trigger deserialization errors. Even if schema backward compatibility is validated during schema evolution, the data inside the Kafka topic can still drift from the defined schema.

One concrete example: a rogue producer writes to a topic without using the expected schema. Now you’ve got a topic that ‘has a schema’ but real events don’t match it. The painful bit is not just knowing something broke, it’s identifying which fields are causing the mismatch.

1.2 Rule and value issues
These are disagreements about what a field means or what shape it should take. Kafka stream schemas define structure but they don’t enforce rules like:

• expected length for an identifier

• expected string pattern

• valid numeric ranges

• constant values that should never change

There wasn’t an existing framework where stakeholders could define and enforce field-level semantic rules for streams.

2- Speed of detection

The second issue was speed of detection. There was no real-time mechanism to automatically validate data against predefined rules, identify issues quickly and alert stakeholders promptly.

Without real-time validation, issues could stick around for a while, quietly impacting multiple online and offline downstream systems before being discovered.

3- Lack of visibility

Even when teams did detect a problem, it was hard to pinpoint the exact ‘poison data’ and understand what violated the schema or the semantic expectations.

Root cause analysis becomes painful when you cannot easily answer:

• Which records were bad?

• Which fields failed?

• What did the bad values look like?

• When did it start and how frequent is it?

The fix

Grab’s Coban platform provides a standardized, platform-level data quality testing and observability setup for Kafka streams. It’s built around four core ideas:

1. Data Contract Definition: Stream stakeholders define a contract that includes schema agreements, semantic rules the topic data must follow, and ownership metadata for alerts and notifications.

2. Automated Test Execution: A long-running test runner automatically executes real-time tests based on that contract.

3. Real-time Data Quality Issue Identification: The system detects data issues in real time at both schema and rules/values levels.

4. Alerts and Result Observability: It alerts the right people and makes it easier to observe issues through the platform UI and downstream tooling.

Put simply: define the rules once, then let the platform watch the stream continuously.

The architecture has three main components:

1. Data contract definition

2. Test execution and data quality issue identification

3. Result observability

All Flow mentions after this refer to those diagrammed steps above

Data contract definition

Coban’s contract acts as a formal agreement among Kafka stream stakeholders. It includes a few building blocks.

Kafka Stream Schema (Flow 1.1)

The contract includes the schema used by the Kafka topic under test. This helps the Test Runner validate schema compatibility across data streams.

Importantly, this is not only about “did the schema change.” It’s also about “does the data actually match what everyone believes the schema is.”

Kafka Stream Configuration (Flow 1.2)

This includes essential config like endpoint and topic name. Coban automatically populates this so users don’t have to wire everything manually.

Observability Metadata (Flow 1.3)

This is where ownership becomes real. The contract includes contact details for stream stakeholders and alert configurations so the right people get notified when issues show up.

Kafka Stream Semantic Test Rules (Flow 1.5)

This is the heart of the semantic side. Users can define intuitive field-level rules such as:

• string pattern checks

• number range checks

• constant value checks

The point is to make the “meaning” of fields enforceable, not just their data types.

LLM-Based Semantic Test Rules Recommendation (Flow 1.4)

Defining dozens or hundreds of field rules can overwhelm people. To reduce that setup burden, Coban uses an LLM-based feature that recommends semantic test rules based on:

• the provided Kafka stream schema

• anonymized sample data

This feature helps users set up semantic rules efficiently, as demonstrated below

The practical benefit: users get a starting point quickly, instead of staring at a schema and trying to invent rules from scratch.

Data contract transformation

Once a contract is defined, Coban’s transformation engine converts it into configurations the Test Runner can interpret (Flow 2.1).

This transformation covers four things:

Kafka Stream Schema: The contract schema is translated into a schema reference format the Test Runner can parse.

Kafka Stream Configuration: The Kafka stream is set up as a source for the Test Runner.

Observability metadata: Contact information is turned into runtime configs for alerting and routing.

Kafka Stream Semantic Test Rules: Human-readable semantic rules are transformed into an inverse SQL query that captures data violating the rules.

‘Inverse SQL’ here means the query is designed to return the bad rows, not the good ones. That’s a smart design choice because it keeps the output focused on what needs investigation.

Test execution & data quality issue identification

Once the transformation engine generates the configuration, the platform automatically deploys the Test Runner.

Test runner

The Test Runner uses FlinkSQL as its compute engine. FlinkSQL was chosen because it makes defining rules straightforward using SQL statements, which also makes it easier for the platform to convert contracts into enforceable checks.

Test execution workflow and problematic data identification

Below are the 4 steps undertaken to execute the test and identify problematic data:

1. Consume Kafka data (Flow 2.2)
   FlinkSQL consumes data from the Kafka topic under test using its own consumer group. This is important because it avoids impacting other consumers.

2. Run inverse SQL (Flow 2.3)
   The Test Runner runs the inverse SQL query to identify:

   • data that violates semantic rules

   • data that is syntactically incorrect “in the first place”

3. Publish data quality issue events (Flow 3.2)
   When bad data is found, the Test Runner packages it into a data quality issue event enriched with:

   • a test summary

   • total count of bad records

   • sample bad data

   Then it publishes the event to a dedicated Kafka topic.

4. Sink events to S3 (Flow 3.1)
   The platform also sinks all data quality events to an AWS S3 bucket for deeper observability and analysis.

This combo (Kafka for realtime events, S3 for deeper inspection) gives both fast alerting and a more durable store for later analysis.

Result observability

Grab’s in-house data quality observability platform, Genchi, consumes the problematic data captured by the Test Runner (Flow 3.3).

Alerting

Genchi sends Slack notifications to stream owners listed in the contract’s observability metadata (Flow 3.5).

Those notifications include useful debugging context such as:

• links to sample data in the Coban UI

• observed time windows

• counts of bad records

• other relevant details

The key point is that alerts are not just ‘something broke’, they include the information you need to start investigating.

Observability

Users can access the Coban UI (Flow 3.4) to see:

• Kafka stream test rules

• sample bad records

• highlighted fields and values that violate rules

That UI piece matters because it shortens the path from ‘alert received’ to ‘I know what field is failing and what the bad values look like.’

Results so far

Since deploying earlier in the year, this solution enabled Kafka stream users to:

• define contracts with both schema and semantic rules

• automate real-time test execution

• alert stakeholders when problematic data is detected so they can act quickly

It has been actively monitoring data quality across 100+ critical Kafka topics.

The solution also offers the capability to immediately identify and halt the propagation of invalid data across multiple streams.

Wrapping up

Grab implemented and rolled out a real-time data quality monitoring solution for Kafka streams through the Coban platform.

The key outcomes include:

• engineers can define syntactic and semantic tests through a data contract

• tests run automatically in real time via a long-running Test Runner based on FlinkSQL

• issues trigger fast Slack alerts through Genchi using ownership metadata in the contract

• teams get better visibility into exactly which data fields violate rules via the Coban UI

In short: Coban turned data quality from a vague hope into something stream owners can specify, enforce and observe in real time.

The full scoop

To learn more about this, check Grab's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Uber moved from batch to streaming in their data lake.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Uber’s streaming solution

TL;DR

Situation

Batch-based ingestion meant data freshness was hours to days, slowing experimentation, analytics and ML across Uber’s core business domains.

Task

Move ingestion to minutes-level freshness at petabyte scale while lowering compute cost and keeping operations reliable across thousands of datasets.

Action

Built IngestionNext using Flink streaming from Kafka to Hudi, plus a control plane for operating ingestion at scale. Solved streaming bottlenecks (small files, partition skew, checkpoint vs commit alignment) to keep performance and correctness intact.

Result

• Freshness improved from hours to minutes-level.

• Compute usage reduced by ~25% vs batch ingestion.

• Compaction performance improved by ~10x with row-group merging.

Use Cases

Near-real time analytics, personalisation, operational analytics

Tech Stack/Framework

Apache Spark, Apache Kafka, Apache Flink, Apache Hudi, Apache Parquet

Explained further

Why data freshness became a platform priority at Uber?

Uber’s data lake sits underneath a lot of the company’s analytics and machine learning. If a team wants to measure an experiment, monitor performance, train a model or sanity-check a business change, it usually starts with is the data in the lake yet?

Historically, ingestion into the lake was batch-based. Freshness was measured in hours. That was fine when decisions moved at daily report speed. It starts to hurt when the business wants near-real-time loops: faster experiments, faster model iteration, faster detection of issues.

Over the past year, the team built and validated IngestionNext, a new ingestion system that switches the default mindset from batch to streaming. It’s centered on Apache Flink, reads events from Kafka, writes to the data lake in Apache Hudi format and operates at petabyte scale. Along the way, they had to solve the stuff that makes streaming annoying in practice: small files, partition skew, checkpoint vs commit alignment and the operational problem of running thousands of jobs reliably.

Why batch ingestion became a bottleneck?

Two main reasons: freshness and efficiency.

Freshness

As the business sped up, teams across Delivery, Rider, Mobility, Finance and Marketing Analytics kept asking the same thing: “Can we get the data sooner?”

Batch ingestion creates delays measured in hours and sometimes days. That lag slows down iteration and decision-making. In a world of continuous experimentation and fast model cycles, hours of latency is basically a tax on everything.

By moving ingestion to Flink-based streaming, the team reduced freshness from hours to minutes. That directly supports faster model launches, quicker experiments and more accurate analytics because the lake stays closer to what’s happening now.

Efficiency

Batch ingestion with Apache Spark is heavy by nature. Jobs run on a schedule, kick off distributed work at fixed intervals and keep doing that even when the workload is uneven. At Uber’s scale, with thousands of datasets and hundreds of petabytes, that adds up to hundreds of thousands of CPU cores running daily.

Streaming smooths this out. Instead of repeatedly spinning up large batch work, resources can scale with traffic in a more continuous way. Less overhead from scheduling, less big bang compute and more efficient usage overall.

IngestionNext: A streaming ingestion platform built for scale

At the data plane, events land in Apache Kafka. Flink jobs consume those events and write them into the data lake using Apache Hudi. Hudi provides transactional behavior like commits, rollbacks and time travel. Freshness and completeness are measured end-to-end from source to sink, not just “did the job run.”

Operating ingestion at this scale is not a set it and forget it situation. So the team built a control plane focused on automation and safety. It manages the ingestion job lifecycle (create, deploy, restart, stop, delete), handles config changes and runs health verification. The goal is simple: run thousands of ingestion jobs consistently without turning the platform into a giant manual babysitting exercise.

The system also supports regional failover and fallback strategies. If there’s an outage, ingestion can shift across regions. If needed, jobs can temporarily fall back to batch mode so ingestion stays available and data is not lost.

Solving the hard parts of streaming ingestion

Streaming buys freshness but it also introduces new failure modes. The team highlighted three major ones: small files, partition skew and checkpoint/commit synchronization.

Small files

Streaming writes data continuously. That tends to create lots of small Parquet files. Small files are a classic way to make query performance worse while also increasing metadata and storage overhead. You get fresher data, then you pay for it every time someone queries.

The common compaction approach merges Parquet files record by record. That means each file gets decompressed, decoded from columnar format into rows, merged, then encoded and compressed again. It works but it’s expensive and slow because you keep doing encode/decode work over and over.

To fix this, the team introduced row-group-level merging. Instead of dropping down into row format, the merge operates directly on Parquet’s native columnar structure. That avoids the expensive recompression path and improves compaction performance by more than an order of magnitude, around 10x.

There are open-source efforts exploring schema-evolution-aware merging using padding and masking to align schemas but that comes with added implementation complexity and maintenance risk.

So the team took a simpler path: enforce schema consistency during merging. Only files with identical schema are merged together. No masking, no low-level code modifications, less engineering overhead and still faster, more efficient and more reliable compaction.

Partition skew

Streaming ingestion depends on steady consumption from Kafka across Flink subtasks. The messy reality is that short-lived downstream slowdowns, like garbage collection pauses can unbalance consumption. Some partitions get read more than others. You end up with skew.

Skew doesn’t just look ugly on a dashboard. It can reduce compression efficiency and lead to slower queries downstream.

The fixes came from three angles:

• Operational tuning: aligning Flink parallelism with Kafka partitions and adjusting fetch parameters.

• Connector-level fairness: adding mechanisms like round-robin polling, pause/resume for heavy partitions and per-partition quotas.

• Observability: exposing per-partition lag metrics, adding skew-aware autoscaling and setting targeted alerts.

This is a good reminder that streaming issues often show up first as weird lag and then become why are queries slower now” If you can’t see skew clearly, you’ll chase symptoms forever.

Checkpoint and commit synchronization

Flink and Hudi each track progress but they track different things.

• Flink checkpoints track consumed offsets.

• Hudi commits track writes.

If failures happen and these drift out of sync, the system can skip data or duplicate it. In ingestion, either outcome is a serious problem.

The team solved this by extending Hudi commit metadata to embed Flink checkpoint IDs. With that linkage, recovery becomes deterministic during rollbacks or failovers. The system can reason about which checkpoint corresponds to which commit and recover without guessing.

Production results: faster data with lower cost

The team onboarded datasets to the Flink-based ingestion platform and validated performance on some of Uber’s largest datasets.

The early results:

• Freshness: improved from hours to minutes-level freshness.

• Efficiency: 25% reduction in compute usage compared to batch ingestion.

Extending real-time beyond ingestion

IngestionNext improves ingestion latency from online Kafka into the offline raw data lake. That’s a big step but it’s not the full story.

Freshness still stalls downstream in transformation and analytics layers. If ingestion is minutes but transformation is still slow, the point of decision is still stale.

The next frontier for Uber is extending real-time capability end-to-end: ingestion → transformation → real-time insights and analytics. This matters because Uber’s lake powers a long list of domains: Delivery, Mobility, Machine Learning, Rider, Marketplace, Maps, Finance and Marketing Analytics. Freshness is a cross-cutting requirement.

Conclusion

Uber’s shift from batch to streaming ingestion is a meaningful platform milestone. By re-architecting ingestion around Apache Flink, IngestionNext delivers fresher data, stronger reliability and scalable efficiency across a petabyte-scale lake.

The design is not just run Flink jobs. It includes operational foundations like an automated control plane, resiliency strategies and streaming-specific engineering work: faster compaction via row-group merging, skew controls and deterministic recovery by linking Flink checkpoints to Hudi commits.

The bigger idea is the mindset shift: treating freshness as a first-class dimension of data quality. With IngestionNext proven in production, the next push is clear: bring streaming into downstream transformation and analytics so the company can close the real-time loop, not just ingest data faster.

The full scoop

To learn more about this, check Uber's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Snap unified Spark, ML workflows and 10k+ daily jobs under one platform.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Snap’s ML platform transformation.

TL;DR

Situation

Snap’s ML teams relied on Apache Spark for analytics but raw Spark was painful for ML workflows: spiky, iterative training data jobs, multiple data formats, scattered tooling and heavy cluster babysitting for every experiment.

Task

They needed an ML-focused data platform on top of Spark that hid infra, handled diverse formats, supported fast experimentation through to stable production and gave a single, coherent experience for Spark users.

Action

They built Prism, a unified Spark platform with a UI and SDK, config-driven Prism Templates to define jobs in YAML, a control plane with Temporal workflows for cluster lifecycle, centralised metrics, autotuning and deep integration with Snap’s billing, orchestration and lakehouse tools.

Result

Prism grew from a handful of daily jobs to several thousand per day, with peaks over 10k, cut onboarding friction, standardised patterns, improved reliability and let ML engineers focus on experiments and models instead of Spark internals and cluster management.

Use Cases

Feature engineering, batch model pipelines, lakehouse ingestion, experiment workflows

Tech Stack/Framework

Apache Spark, Apache Iceberg, Dataproc, Apache Airflow, Kubeflow, Apache Parquet, Trino

Explained further

Context

Apache Spark has been a core part of Snap’s analytical stack for a long time. It runs the pipelines that feed reports, dashboards and batch data products. For that world, Spark is a good fit.

Machine learning puts new pressure on that foundation.

ML development is inherently iterative. An engineer can spend a week trying variations of the same broad idea: a different label definition, a refined feature set, a new way to slice users, a new pre-processing recipe. Each iteration often means regenerating training data from very large raw sources. Doing that repeatedly is not a nice, predictable nightly job. It is a series of intense, sometimes spiky workloads that hit the platform whenever someone has another idea.

The development lifecycle is also more fluid. Early in a project, ML engineers want freedom. They want to pull ad hoc samples, tweak schemas on the fly, and see results quickly. Once the same model is ready for production, the expectations flip. Pipelines need to be stable, observable and efficient on real traffic. The platform has to support both modes without asking people to throw away all their early work and start again from scratch.

Then there is the question of formats. ML workloads do not live in a single file format. Snap’s teams use:

• TFRecord when they are feeding TensorFlow

• Protobuf when they are working with gRPC-based serving systems

• JSON for lightweight exploration and simple tests

• Parquet and Iceberg for analytical and lakehouse-style storage

Forcing everything through a single “blessed” format would only slow teams down. A realistic platform needs to work comfortably across all of these.

Where raw Spark started to hurt ML teams

Spark itself is not the weak link. It is powerful, battle tested and extremely good at scaling SQL and batch workloads. The problem is not capability; it is usability for ML engineers.

Without the right abstractions:

• Engineers need to understand Spark internals and distributed systems just to write reasonable jobs.

• They rebuild the same boilerplate, like data validation or common preprocessing across teams.

• They manage clusters, dependencies, and upgrades themselves.

• They spend time in Spark UI and logs chasing down failures that add no value to the model itself.

All of this is on top of their actual core stack: TensorFlow or PyTorch, notebooks, experiment tracking tools, workflow systems like Kubeflow and internal ML platforms. Spark is an ingredient they need for scale, not the centre of their role. When the support around it is thin, that ingredient becomes a constant source of overhead.

Snap’s ML engineers wanted to spend their time on experiments and models, not on reverse engineering cluster failures. That is the capability gap the team set out to close.

What Snap wanted from an ML data platform

The Snap team set a clear goal: build an ML-focused data platform on top of Spark that feels consistent, friendly, and scalable, instead of “bare-metal Spark with some helpers”.

The platform should let ML engineers:

• Describe what data they need instead of hand-coding how to compute it

• Iterate quickly without piles of glue code

• Reuse proven, secure patterns for data processing

• Spend their energy on model and product logic instead of infrastructure

That means heavy lifting in one place: infrastructure abstraction, patterns, observability, integration with Snap’s internal ecosystem for metrics, billing, cost tracking, scheduling.

With that foundation in place, ML development becomes faster and more consistent, and the platform team can invest in shared improvements instead of firefighting one-off jobs.

Boiling the problem down

So the problem is not “Spark is bad for ML”. The problem is that raw Spark is too low level for the way ML teams actually work.

What Snap’s team built with Prism is a layer on top of Spark that:

• Hides cluster-level pain

• Standardizes job patterns

• Bakes in observability and cost awareness

• Fits naturally into ML workflows rather than generic analytics

Prism is the answer built around those constraints. It keeps Spark, but wraps it in a set of tools that match how ML teams at Snap actually work.

Meet Prism: Snap’s ML data platform on Spark

Prism is Snap’s unified Spark platform. From the outside, it looks like one coherent system that handles job authoring, productionisation and post-production operations.

Instead of each team creating its own way of submitting Spark jobs and wiring up clusters, Prism offers a consistent experience. Engineers can define jobs in a configuration-driven way, work in a UI when they want a visual surface, or go through an SDK when they need more control. Underneath, the platform handles cluster lifecycle, resource management, metrics collection and cost accounting.

Prism also aims for a serverless feel. Users submit work and adjust configurations, while the system decides how to spin up clusters, scale them and shut them down. That does not remove Spark, but it changes how people interact with it.

From experiment to production: the Prism user journey

If you follow a typical workflow through Prism, you see three distinct phases.

In pre-production, engineers are experimenting. They want to get a job off the ground quickly, test logic and refine their approach. Prism supports this by offering configuration-based templates and a UI where most of the setup is already done. Predefined profiles cover common use cases which means teams are not spending their first week just tuning cluster settings.

Once a job is ready to run regularly, it enters productisation. At this point, Prism controls cluster setup, scaling and teardown through a unified API. Jobs can be tied into orchestration tools such as Airflow or Kubeflow without every team reinventing the wheel. Dashboards, metrics and metadata tracked by Prism give users a cleaner window into how their jobs behave.

Post-production, attention moves to reliability and efficiency. Prism takes on this work by centralising monitoring and alerts, storing rich metrics and offering autotuning features that can recommend or apply improvements. Job costs are tracked, and infrastructure upgrades happen at the platform layer instead of per-team.

Prism architecture

Prism’s architecture is organised around two main user-facing interfaces and a set of internal systems.

For users, the first touchpoint is the Prism UI, a console where they can author jobs, inspect runs, debug failures and tune performance. The second is a client SDK that exposes Prism’s capabilities programmatically. Together, these give ML and data engineers both an interactive and an automated way to work with Spark.

Behind those sits Prism Template, a framework for composing Spark jobs out of structured, reusable blocks. Instead of asking every engineer to shape their own Spark application, Prism Template gives them a vocabulary of modules they can chain together with YAML.

All external requests hit a central API and then flow into the Prism Control Plane. This control plane is responsible for managing job metadata and configuration. It delegates orchestration work to a workflow system powered by Temporal. Temporal workflows handle cluster provisioning, job submission, retries, cancellation and similar runtime tasks.

The whole stack ties into Snap’s internal services for metrics, cost tracking and orchestration. The idea is to have one platform that understands both the Spark world and the rest of Snap’s infrastructure landscape.

A single home for Spark users: Prism UI console

Before the Prism UI, Spark users at Snap lived inside a messy toolkit. They had an infra-owned library and CLI to standardise job submission, then Spark UI and History Server to debug jobs, HDFS tooling for storage, Dataproc Console and Stackdriver for cloud-level views, and separate internal systems for metrics and cost.

Each of these tools did something useful, but taken together, they were a scattered experience. New users had to learn not only Spark, but also the map of where to click when something went wrong. Even experienced teams were wasting time stitching context together from five tabs.

The Prism UI Console was built to compress this spread. It gives Spark users a single place to search jobs, view runs, understand configuration, inspect metrics and author new work. Platform teams now have a clear surface to invest in, and engineers have one source of truth for their Spark workloads.

The result is lower friction, faster onboarding and a much clearer path for incremental usability improvements in the future.

What you can do in the Prism UI

The Prism UI is the primary surface area for Spark users at Snap. Key capabilities include:

• Unified job search: A central search page lets users filter by job name, namespace, cluster ID, and other attributes. When something breaks, they no longer have to bookmark multiple systems.

• Metadata storage: Job and cluster metadata, such as configurations, metrics, and lineage, are stored in a scalable backend. This supports analytics, audits, and better platform decisions.

• Logical job grouping for trend analysis: Jobs are grouped by orchestration task IDs, for example from Airflow or Kubeflow. This makes it easy to look at long-term trends in runtime, cost, and resource use for a specific workflow.

• Integrated real-time cost estimation: Through integration with internal billing systems, users can see cost estimates while jobs run. This is especially helpful during heavy experimentation when budgets matter.

• One-click utilities and deep links: Utilities like job cloning, as well as deep links into Spark UI, logs, and output tables, make iteration and debugging faster.

• Integrated job authoring: Users can

  • Configure sources and sinks with built-in support for Iceberg, TFRecords, Parquet, and BigQuery

  • Use an SQL editor with autocomplete powered by a metastore-aware schema integration

  • Pick preconfigured job profiles created by Spark experts

  • Jump directly to the Lakehouse UI for Iceberg-backed outputs and query them via Trino

  • Create low-code jobs using Prism Templates

This is where most ML users feel the impact. Instead of wrestling with scattered tools, they have a single console designed for how they work.

Prism templates: opinionated Spark jobs without the boilerplate

Spark’s flexibility is both a strength and a risk. The same workload can be written in very different ways which leads to wide differences in structure, resource usage and maintainability. In a large organisation, that turns into a support headache.

Prism Template is Snap’s way of putting structure on top of that flexibility. Instead of everyone writing full Spark applications, users define jobs in YAML using reusable modules and standardised patterns. The platform takes ownership of job bootstrap, configuration and core logic wiring.

This approach makes experimentation easier for ML engineers. They can assemble pipelines quickly without having to understand every detail of Spark’s internals. Later on, as jobs move toward production, teams can adjust centralised configurations and modules to scale more gracefully, without rewriting their application code.

Why templates matter in practice

The core benefits of this approach:

• Simplified job authoring: Users describe jobs via a YAML file instead of writing low-level Spark boilerplate. The same definition can move from local development to staging and then production.

• High-quality, reusable components: The platform ships with modules that cover tasks like:

  • Iceberg ingestion

  • Feature extraction

  • Sequence building and manipulation

  These modules encode best practices, which reduces user errors and keeps jobs consistent.

• Integrated observability and tooling: Because Prism owns the bootstrap and core modules, it can inject metrics, logging, and other operational hooks in a uniform way.

• Managed versioning: Job definitions, driver JARs, and plugin JARs are versioned and managed centrally. This supports safe upgrades and stable behavior across environments.

• Customisable templates: Users can start from pre-built templates, then add their own parameters or chain modules in new ways.

The example below shows a Prism Template YAML snippet that combined two modules in one job: one that builds an ordered sequence column and one that ingests the result into an Iceberg table.

The structure is easy to read and the heavy logic lives in reusable modules.

The Prism control plane: making Spark feel serverless

The first take on a control plane at Snap was lean. It wrapped Dataproc APIs, handled some permissions and exposed separate concepts for clusters and jobs. Orchestration tools such as Airflow still had to own cluster lifecycle, including creation, reuse, teardown and failure handling.

At small scale, that model works. As usage grows, human-managed cluster lifecycle turns into a liability. Teams end up carrying subtle differences in how they handle errors and retries. Operational load rises. Reliability drops.

The team redesigned the control plane around a different principle: one simple job submission interface that hides cluster lifecycle.

The redesigned system:

• Presents a single API endpoint for job submission

• Internally handles cluster provisioning, monitoring, retries, and shutdown

• Uses a workflow engine built on Temporal to orchestrate these steps

The division of responsibilities is clear:

• The control plane manages metadata and configuration

• Temporal workflows manage the runtime orchestration

This improves reliability and reduces cognitive load for users. It also creates a base for smarter features such as autotuning and intelligent retry policies, since the platform now owns the whole lifecycle rather than just parts of it.

Metrics and autotuning: turning Spark signals into smarter defaults

Spark and Dataproc expose a large amount of metrics out of the box. The problem is not availability, it is usability.

The raw metrics:

• Exist at multiple levels: job, stage, task, cluster

• Are not always structured for time-series analysis

• Use inconsistent naming and retention policies across sources

• Are difficult to use as inputs for automation

To fix this, Snap built a dedicated metrics system for Spark workloads in Prism.

This system:

• Ingests selected signals from jobs, clusters, and infrastructure

• Normalizes them into a coherent schema

• Stores them in a centralized Spanner database for durability and consistency

With this foundation, Prism can:

• Show actionable metrics in the UI console

• Back features like autoscaling and intelligent retries

• Support autotuning features that adjust configuration based on observed behavior

Importantly, this metrics work was done in parallel with the UI console, so both evolved together. The result is a unified experience where what the user sees and what the automation uses come from the same underlying system.

How Prism spread across Snap

The clearest sign that a platform is working is usage. Prism’s daily job counts have climbed from single digits to several thousand per day, with peaks above 10,000 jobs.

The pattern of adoption falls into two main buckets.

1. Direct use by advanced Spark teams: Teams with complex Spark needs use Prism directly. Their workloads often involve:

• Large joins

• Tight coupling with specific data models

• Custom logic that does not fit into a narrow “standard pipeline” box

These teams still get value from Prism’s abstractions and control plane, but they stay close to the underlying capabilities.

2. Integration into internal platforms: Other teams do not think in terms of Spark at all. They work with internal tools for:

• ML data preparation

• Feature engineering

• Experimentation

Those tools, in turn, embed Prism. The teams’ users get domain-specific interfaces, while Prism quietly runs the heavy Spark work in the background.

Supporting both direct and embedded use is a crucial design choice. It lets Prism spread across Snap without forcing every user into the same interface or abstraction level.

The full scoop

To learn more about this, check Snapchat's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Lately, I’ve been thinking about starting a new series where people working in data share how they got here, what they’ve learned along the way and what their day-to-day looks like.

So, I’m kicking it off today with , Senior Data Engineer at Workpath and writer of The Pipe and The Line newsletter.

The Pipe & The Line

Hands-on guides, tools, and experiments to sharpen your Data & AI Engineering skills from someone who learned it all in the wild.

By Alejandro Aboy

We talked about how he went from marketing to data engineering, what his workflow looks like, why he was called an octopus and why he thinks “big data” is a fool’s errand for most teams.

So without further ado, let’s get into it!

Read more

Today we will look at how Shopify built a real-time data pipeline at 400TB scale.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to change data capture at Shopify!

Read more

Today we will look at how Grab made real-time processing faster for its users.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to real-time processing at Grab!

Read more

Today we will look at how Expedia Group monitors A/B tests at a large scale

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people and they subscribe to the newsletter.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get the real-time monitoring of A/B tests by Expedia

Read more

Today we will look at how Bolt tracks payments at scale.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people and they subscribe to the newsletter.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to how Bolt deals with processing payments from millions of customers

Read more

Today we will look at how Flipkart solved the problem of calculating delivery date.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Flipkart’s challenge of calculating delivery date at scale

Read more

Quick note before this week’s deep dive. Thanks for reading and subscribing, I really do mean it. If you’ve got feedback, just hit reply. I read every response.

Data Tinkerer has always been about sharing what actually works in data, beyond just tools and tech. The deep dives will keep coming but I want to start spotlighting the stuff we don’t talk about enough: the day-to-day challenges, business outcomes, the challenges and the learnings.

I want to feature stories from people in data roles: senior data engineers, lead analysts, heads of data, you name it. If you’ve got a story, lesson, recent technical win or even a battle scar from the data trenches, let’s get it in front of almost 1,000 smart and engaged peers.

You don’t need to be a “writer”, I’ll help your story shine. Plus, guest contributors get a shoutout in the newsletter and on LinkedIn (If you want).

Keen to share your data story? Just reply to this email or message me on Substack and we’ll tee it up.

Now …

Read more

Today we will look at how Canva solved the surprisingly messy problem of counting at scale

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Canva’s work to build a scalable data pipeline

Read more

TL;DR

Read more

Situation

HubSpot's backend performance team identified that Amazon S3 storage costs accounted for approximately 45% to 50% of daily expenses, with the 'hubspot-live-logs-prod' bucket alone responsible for 20% of these costs.

Task

The team aimed to reduce storage costs by addressing the inefficiencies in their logging system, particularly focusing on the large volumes of raw JSON logs that were not being efficiently compacted.

Action

• Log Retention Review: They discovered that raw JSON logs were retained for 730 days, while compressed ORC logs were kept for 460 days. Aligning the retention period to 460 days for both formats reduced unnecessary storage.​

• Improved Compression: By enhancing their Spark compaction process, they increased the conversion rate of raw JSON logs to the more storage-efficient ORC format, achieving a compression ratio where ORC logs were about 5% the size of the original JSON logs.

Result

These measures led to a 55.7% reduction in monthly JSON log storage costs, translating to annual savings in the seven-figure range. Additionally, engineers experienced faster log query times, with some reporting reductions from 30 minutes to just 36 seconds.

Use Cases

Cost monitoring, Log retention, Log volume reduction

Tech Stack/Framework

AWS Athena, Amazon S3, Apache Spark, Apache Mesos, Redash

Explained Further

Saving Millions on Logging

Read more

Situation

Reddit's advertising platform processes thousands of ad engagement events per second, necessitating real-time validation and enrichment to ensure accurate reporting and prevent budget overdelivery.

Task

Develop a scalable, real-time ad event validation system capable of efficiently handling high event volumes while maintaining performance and reliability.

Action

The engineering team developed the Ad Events Validator (AEV) utilizing Apache Flink to correlate ad server events with user engagement events. To overcome issues related to large state sizes and resource demands, they implemented:

• Field Filtering: Conducted a thorough analysis of downstream data consumption, establishing an allowlist that significantly reduced the event payload size by 90%, leading to CPU and memory usage reductions of 25% and 60%, respectively.

• Tiered State Storage: Integrated Apache Cassandra for external state storage, effectively reducing in-memory state size and enhancing the efficiency of checkpointing and system recovery processes.

Result

These strategic enhancements resulted in a more scalable and cost-efficient AEV system, improving overall performance and operational effectiveness.

Use Cases

Real-Time Event Validation, Data Enrichment, Resource Optimization

Tech Stack/Framework

Apache Flink, Apache Kafka, Apache Cassandra

Explained Further

Background

Reddit processes thousands of ad engagement events per second. These events require validation and enrichment before being sent to downstream systems. Key components of this validation process include applying a standardized look-back window and filtering out suspected invalid traffic.

In addition to a batch validation pipeline, a near real-time pipeline improves budget spend accuracy and provides advertisers with real-time insights into campaign performance. This real-time component, known as the Ad Events Validator (AEV), is built using Apache Flink. AEV matches ad server events with engagement events and writes the validated results to a separate Kafka topic for downstream consumption.

Building and maintaining AEV though, presented several challenges to the Reddit team

1st Challenge: Addressing High State Size Issues

Read more

TL;DR

Situation

Yelp's ad revenue relies on predicting which ads users are likely to click on, using a model called "Wide and Deep Neural Network." Initially, training this model on 450 million data samples took 75 hours per cycle, which was too slow. Yelp wanted to handle 2 billion samples and reduce training time to under an hour per cycle.

Task

The goal was to speed up the training process by improving how data is stored and read, and by using multiple GPUs to handle more data at once.

Action

• Data Storage: Yelp stored the training data in Parquet format on Amazon's S3 storage, which works well with their data processing system, Spark. They found that a tool called Petastorm was too slow for their needs, so they developed their own system called ArrowStreamServer. This new system reads and sends data more efficiently, reducing the time to process 9 million samples from over 13 minutes to about 19 seconds.

• Distributed Training: Yelp initially used a method called MirroredStrategy to train the model on multiple GPUs but found it didn't work well as they added more GPUs. They switched to a tool called Horovod, which allowed them to efficiently use up to 8 GPUs at once, significantly speeding up the training process.

Result

By implementing these changes, Yelp achieved a total speed increase of about 1,400 times in their model training. This means they can now train their ad prediction models much faster, allowing them to handle more data and improve their ad services.

Use Cases

Large-Scale ML Training, ML Training Optimisation, Enhancing Data Pipeline Efficiency

Tech Stack/Framework

TensorFlow, Horovod, Keras, PyArrow, Amazon S3, Apache Spark

Explained Further

The Challenge

Read more

TL;DR

Situation

Meta handles vast amounts of user data across its platforms, requiring strong privacy controls to protect sensitive information. A critical component of this effort is data lineage, which helps trace how data moves across different systems, ensuring compliance with privacy policies like purpose limitation.

Task

Meta needed a scalable and automated way to track data lineage across millions of assets, including databases, web services, and AI systems. This required moving beyond manual data flow documentation to a more robust, automated discovery process.

Action

• Data Flow Collection – Used static code analysis, runtime instrumentation, and input/output matching to track data across stacks (Hack, C++, Python, SQL).

• Privacy Probes – Captured real-time runtime signals, identifying how and where sensitive data is logged, stored, or transformed.

• Automated Lineage Graphs – Created scalable data flow visualizations to streamline privacy control implementation.

• AI & Data Warehouse Integration – Ensured end-to-end traceability across AI models, databases, and batch-processing systems.

• Iterative Filtering Tool – Allowed developers to refine lineage graphs, isolating relevant data flows and removing noise.

Result

Meta’s data lineage system reduced engineering time, improved compliance accuracy, and automated privacy enforcement. It enabled developers to quickly identify and secure sensitive data flows while ensuring continuous monitoring at scale. These innovations enhanced user data protection across Meta’s ecosystem

Use Cases

Privacy Enforcement, Compliance Monitoring, Data Lineage

Tech Stack/Framework

Python, SQL, C++, PyTorch, Presto, Spark

Explained Further

Meta's Privacy Aware Infrastructure (PAI) is designed to embed privacy controls within its systems, ensuring user data is handled responsibly. A foundational element of PAI is data lineage, which traces the journey of data across various platforms, providing a comprehensive view of its flow from collection to processing and storage. This capability is crucial for implementing privacy measures like purpose limitation, which restricts data usage to specific, intended purposes

Understanding Data Lineage at Meta

Data lineage involves mapping out how data moves through Meta's vast ecosystem, connecting source assets (e.g., database tables where data originates) to sink assets (e.g., tables or systems where data is stored or processed).

This mapping is essential for:

Read more

TL;DR

Situation

Expedia Group needed a scalable and cost-effective real-time analytics solution (<15 seconds latency) to process high-volume data (~4500 events/sec) and support global service partners in optimizing operations and enhancing performance.

Task

Design a solution to process and present real-time data with blazing-fast query speeds while addressing limitations of existing tools (e.g., Snowflake and Looker) in terms of scalability, latency, and user experience.

Action

Developed a new architecture using Apache Druid for real-time ingestion, optimized microservices for data processing, and built a custom modular UI library with a Data Resolver API to deliver tailored analytics based on user roles.

Result

The solution achieved a 5x increase in user base, 30-40% reduction in costs, 15-second data latency, and 99.9% SLA uptime. It supported 1,800 users with sub-1-second response times, enhancing decision-making and operational efficiency globally.

Use Cases

Real-Time Insights, Operational Efficiency, Scalability for Concurrent Users

Tech Stack/Framework

Python, Apache Druid, Apache Hive, Apache Kafka, Looker, Snowflake

Explained Further

Real-Time Challenges for Expedia

Read more

TL;DR

Situation

Datadog's time-series database, designed in 2016, struggled to manage a 30x growth in data volume and rising query complexity, resulting in slower performance and higher maintenance overhead.

Task

Develop a scalable indexing system to efficiently process high-cardinality data while improving query speed and reducing operational costs.

Action

The team implemented an inverted index inspired by search engines, mapping tags to time-series IDs. Using RocksDB for storage, they ensured scalability, reliability, and efficient query filtering.

Result

Query performance improved by 99%, enabling support for 20x higher cardinality metrics, reducing query timeouts, and cutting operational costs by nearly 50%.

Use Cases

Real-Time Monitoring, Tag-Based Filtering, Dynamic Schema Handling, Query Execution

Tech Stack/Framework

RocksDB, Apache Kafka, SQLite, Time-Series Database, Rust

Explained Further

Understanding the Problem

Datadog’s timeseries database faced significant challenges as data volumes grew 30x between 2017 and 2022. The increasing complexity of user queries and higher data cardinality strained the existing indexing system, introduced in 2016. The original architecture became a bottleneck for query performance and required substantial maintenance.

Metrics Platform Overview

Read more