Today we will look at how Grab used AI agents to cut repetitive data support from hours to minutes.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get into how Grab turned repetitive data support into an AI agent workflow.

TL;DR

Situation

Grab’s Analytics Data Warehouse team supported 1,000+ monthly users and 15,000+ tables, but repetitive ‘quick questions’ were consuming ~40% of engineering time.

Task

The team needed to reduce manual support work while keeping answers accurate, safe and trusted for production data workflows.

Action

Grab built a multi-agent AI system with specialised agents for data investigation, code search, on-call checks, summarisation and enhancement requests.

Result

Resolution time dropped from hours to minutes, support backlog was largely removed and several FTEs worth of engineering capacity shifted back to roadmap work.

Use Cases

Data quality validation, internal support bot, BI troubleshooting assistant, tracing data lineage

Tech Stack/Framework

FastAPI, LangGraph, Redis, PostgreSQL, tiktoken, Trino, Apache Hive, Delta Lake, Apache Spark, Apache Airflow, RAG

Explained further

Context

Grab’s Analytics Data Warehouse team supports more than 1,000 users every month. Those users rely on a repository of over 15,000 tables, which powers roughly half of all queries across Grab’s data lake.

That scale is useful, obviously. It also creates a very familiar data platform problem: the team becomes the help desk for every ‘quick question’ that is not actually quick.

Someone asks what a column means. Someone else wants to know why a table looks wrong. Another person needs help tracing where a metric came from. Then there are quality checks, source investigations, pipeline questions and basic enhancement requests.

Individually, none of these requests look dramatic. Collectively, they were eating about 40% of the team’s time. That is roughly two days every week spent on repetitive support work instead of building the next layer of the platform.

The Grab team responded by building a multi-agent AI system that could handle simpler questions autonomously and work through more complex investigations with help from specialised agents. The result was a shift from reactive support back to proactive engineering work, with hundreds of hours of productivity unlocked every month.

The Friday Slack message every data engineer fears

It is 5:00 PM on a Friday. A Slack message lands:

“The vehicle_id in our production table looks gibberish. Is the pipeline broken?”

Every data team knows this message. It might sound small but it is not.

To answer it properly, an engineer needs to check the data catalogue, inspect the table, trace the lineage, validate the SQL, look at the code, check pipeline health and maybe search through Slack or Confluence for recent incidents. By the time the stakeholder receives a confident answer, what looked like a five-minute question has quietly consumed a large chunk of engineering time.

The Grab team studied these requests and found a pattern. The problems were different, but the investigation process was often the same. The team had to gather context, check metadata, inspect data, search code, validate assumptions and summarise the answer. That consistency was the opportunity.

Rather than trying to answer every question manually, the team built a multi-agent AI system that could automate the context hunt. Engineers would still stay in the loop where needed, especially for production changes and judgment-heavy decisions. But the repetitive investigation steps could be handled by agents.

Solution

The system was designed around one simple idea: separate the reasoning layer from the action layer.

The large language model handles interpretation and planning. The agents handle specialised work such as querying data, searching code, checking incidents, reviewing pipeline health and drafting code changes. This made the system easier to reason about, easier to debug and safer to operate.

Grab used FastAPI to handle incoming requests and LangGraph to manage the agent workflows. LangGraph was important because this was not a simple one-shot LLM call. The agents needed to loop, pass work to each other, ask for more information and maintain state across a multi-step investigation.

Redis was used for caching and real-time session needs. PostgreSQL served as persistent memory, storing conversation history and agent metadata.

The system also connected into Grab’s internal platforms:

Hubble, the central metadata management platform and data catalogue.

Genchi, the data quality observability platform used to enforce data contracts.

Lighthouse, the platform used to track execution status and monitor pipeline health.

How a Slack question turns into an agent workflow

The journey starts in Slack. When a user sends a request, the system routes it into one of two broad streams.

The first stream is for enhancement requests. These go to the Enhancement Agent, which works with engineering tools such as GitLab, Apache Spark and Airflow to propose and test code changes.

The second stream is for general questions. These go through the investigation pathway, where the system coordinates a small group of specialist agents. The Data Agent queries systems such as Trino, Hive or Delta Lake. The Code Search Agent inspects GitLab. The On-call Agent checks operational context from Confluence, Slack and observability tools.

This design matters because the system is not one giant agent trying to do everything. The ‘brain’ is decoupled from the ‘hands’. The LLM can reason about the request, but the actual work is handled by tools and agents with specific responsibilities.

Why one giant agent was the wrong answer

Grab could have built one large agent and asked it to handle every type of request. That would have been simpler at first glance: one model, one prompt, one inference path.

But that kind of monolithic design becomes hard to maintain as the system grows. Every new tool increases prompt complexity. Every change risks affecting unrelated behaviour. Debugging becomes messy because it is hard to know whether the failure came from classification, data access, code search, reasoning or summarisation.

The multi-agent design traded some latency and coordination complexity for maintainability and accuracy.

A single AI system has the advantage of being simpler to call and manage. But it is also harder to debug, harder to improve safely and more likely to behave like a generalist.

A multi-agent system is more modular. Each agent can be improved independently. The Data Agent can get better at query validation without touching the Code Search Agent. The On-call Agent can be expanded with better incident lookup without changing the Enhancement Agent. Failures are easier to isolate because each part has a narrower job.

For Grab, the choice was obvious. When replacing a manual process that used to take hours, a few minutes of agent coordination is not a serious downside. Accuracy and maintainability mattered more than shaving off a few seconds.

The two-lane system behind Grab’s agent workflow

When a question arrives through Slack, the system first decides which pathway it belongs to.

Pathway 1: Enhancement requests go to the Enhancement Agent.

Pathway 2: Investigation questions go through the Classifier, then one or more specialist agents, then the Summarizer.

The agent that drafts pipeline changes but does not ship them blindly

Some requests are not just questions. They require changes to data pipelines.

For example:

“Can you add a new column for customer_segment?”

Or:

“We need to change the aggregation logic for revenue.”

Previously, this meant an engineer had to gather requirements, inspect schemas, find the right code, check dependencies, write changes, test them and prepare a merge request.

The Enhancement Agent handles much of that heavy lifting.

It gathers context from schemas, lineage, dependencies and the existing codebase. It proposes code changes and creates a merge request. It runs the changes in a test environment. It also flags governance concerns such as PII classification, SLA impact and backward compatibility.

The workflow is intentionally semi-automated.

A user creates a JIRA request. The agent analyses the requirements and gathers extra context through dialogue with the engineer. It creates a Merge Request (MR) with suggested code. The engineer reviews the MR. If the changes are valid, the agent runs them in a test environment. The engineer checks the output against test cases. If everything passes, the engineer merges the MR.

That final approval stays with a human because production pipeline changes need judgment. The agent can do research, write code and run tests, but it should not be blindly changing production systems.

The agent squad for messy data questions

For questions such as “Why does this data look wrong?” or “Where does this metric come from?”, the system uses a coordinated investigation flow.

The Classifier is the first responder. It parses the question, extracts important details such as tables, scripts or columns, detects guardrail violations and decides which specialist agents are needed. It also provides the reasoning and task description for each agent.

For a question like “Why does this ID look wrong?”, the Classifier may route the request to the Data Agent, then the Code Search Agent, then the On-call Agent if production context is needed.

The Data Agent performs the data investigation. It enriches the prompt with table and column metadata, runs SQL queries with guardrails, validates schemas before scanning and retrieves sample data with exploratory comments. This helps confirm whether the user’s observation is real or based on a misunderstanding.

The Code Search Agent inspects the codebase. It traces transformations, follows table lineage across multiple steps and explains transformation logic in plain language. It can also highlight differences between the implementation and what documentation or stakeholders expect.

The On-call Agent checks production context. It searches Slack for outage announcements, checks observability platforms for pipeline status, reviews logs and retry policies, validates quality metrics such as null counts and duplicates, and produces initial incident notes when something looks wrong.

The Summarizer Agent pulls the pieces together. It handles conflicts, turns agent findings into a coherent answer and makes the response concise enough for Slack. It is the final step before the answer is reviewed or posted.

What this looks like in real data team workflows

The clearest way to understand the system is to walk through two common scenarios.

Scenario 1: Adding a new column

A stakeholder creates a JIRA ticket:

“Please add a customer_segment column to the rides table. Source data is available in the user_profiles table.”

In the traditional workflow, this would take a decent chunk of an engineer’s afternoon. There would be requirement checks, code discovery, schema validation, development, testing and MR preparation.

With the Enhancement Agent, the process becomes much faster.

The agent first reads the JIRA ticket to understand what column needs to be added, which target table is involved and where the source data lives.

Then it searches the codebase to find the relevant transformation files. It navigates the repository, identifies the right scripts and locates the parts that need modification.

Before making changes, it validates that the requested column exists in the upstream source table, checks that the column does not already exist in the target table, and confirms schema compatibility and data quality requirements.

It then generates the database schema changes. To do that, it references existing Data Definition Language (DDL) scripts, follows the standard format and creates the required schema modification scripts. These are included in the MR alongside the code changes.

Finally, it creates the MR with documentation. Once the MR is validated, users can interact with the bot to trigger Airflow test runs. They can also specify date ranges or parameters for testing.

The process from ticket to deployable MR is completed in minutes, but the engineer can still inspect the changes before anything moves forward.

Scenario 2: Investigating faulty-looking data

A user asks:

“Why is the ID in the vehicles table unreadable?”

In the old process, an engineer would manually search the catalogue, inspect lineage, write SQL, check logs and review pipeline status.

With agents, the investigation is broken into steps.

First, the Classifier parses the question and decides that all three specialist agents are needed. It plans the sequence as Data Agent, then Code Search Agent, then On-call Agent. Its reasoning is simple: the system needs to verify the data format, trace the transformation logic and check whether there are any production incidents.

The Data Agent starts by retrieving metadata and building a SQL query to inspect sample values. It confirms the user’s observation: the IDs do look unreadable, because they are in UUID format.

It then searches Grab’s data catalogue for dimension tables that could make the values more readable. It finds a suitable dimension table and builds a join query to test whether the UUIDs can be mapped to human-readable vehicle names.

The Data Agent’s conclusion is that the ID column contains valid UUID values. They can be joined with dim_vehicles to get readable vehicle names. The data is not corrupted.

Next, the Code Search Agent traces the lineage. It scans transformation logic in the codebase and finds that the ID is extracted as a raw UUID from a JSON payload from the source system. It queries source samples and confirms that the same pattern exists upstream.

Its conclusion is that the UUID format comes directly from the source system. It is not a bug introduced by Spark transformations.

Then the On-call Agent checks operational health. It reviews Airflow pipeline status, searches Slack for incidents and checks data quality metrics.

Its conclusion is that no production incident is detected. The pipeline is running successfully, quality metrics are normal and no recent complaints have been raised in communication channels.

The Summarizer Agent then turns these findings into a structured answer.

The user’s concern was that ID values looked unreadable. The Data Agent found that the IDs are valid UUIDs and can be joined with dim_vehicles for readable names. The Code Search Agent found that the UUIDs come directly from the source system. The On-call Agent found no production issue.

So the answer is clear: the data looks unfriendly, but it is valid. The pipeline is not broken.

The answer is posted to Slack quickly and marked as unreviewed. This gives users a fast initial response while making it clear that an engineer still needs to review it. The initial response time drops from hours to minutes.

After the answer is posted, anyone can continue the conversation with the agents, which restarts the loop when more clarification is needed.

What Grab had to fix before this worked in production

The prototype worked but making it work in production was harder.

Real users ask messy questions. Conversations get long. Agents use too many tools. Tool outputs can be huge. Some queries are risky. And because this system touches production data workflows, trust matters.

Grab had to optimise the architecture around six major challenges.

1. The agents were drowning in their own context

In multi-agent systems, context grows quickly. Each agent adds messages, tool calls, results and reasoning. If that context is passed forward without discipline, token usage balloons and performance drops.

Grab’s orchestrator maintains a rich execution state, including conversation and tooling history, agent progress and structured responses from each agent. But that state is managed carefully so agents receive the context they need without being flooded.

The system tracks tokens using tiktoken. When limits are exceeded, earlier messages are summarised while preserving information relevant to the original question. Recent messages and critical context remain unsummarised to protect accuracy.

The team also pruned RAG context. Instead of passing entire code files to the Code Search Agent, smaller LLMs extract the most relevant snippets and short descriptions. For database queries, filters are used to retrieve only the most relevant results.

The handoff pattern is also important. Each agent returns its response to a central orchestrator. The orchestrator cleans the context, removes unnecessary tokens and invokes the next agent.

The result is a system that can handle extended investigations without drowning in its own context.

2. Too many tools made the system slower, not smarter

The early version had more than 30 tools, many shaped like generic APIs. That created another problem: tool descriptions and outputs became part of the agent prompt, so the model had to process a lot of irrelevant material before doing useful work.

Grab redesigned tools around real usage. Tool descriptions were shortened. Outputs were aggressively trimmed. Only information needed for decision-making was included.

This reduced the amount of data agents had to process and improved responsiveness.

3. Adding guardrails around SQL and code changes

Agents with database access and code generation powers are useful, but also risky.

They could access sensitive PII, run expensive queries, execute dangerous SQL or generate breaking changes. Grab added multiple layers of safety.

First, the Classifier checks for PII requests and out-of-scope questions before any agent runs.

Second, the Data Agent validates SQL before execution. It checks PII column access, blocks risky DDL and DML operations, detects slow queries and validates schemas before running anything.

Third, all database queries have strict timeout limits.

Fourth, the Enhancement Agent cannot commit directly to master or main. All changes go through merge requests, require human review and run in a test environment first.

The point is not to pretend agents are perfectly safe. The point is to build the system assuming they are not.

4. Designing review into the system

Even with RAG and guardrails, agents can be wrong. A confident wrong answer is worse than no answer, especially in a data platform context.

Grab added human review after the summarised response is generated. Reviewers can approve, reject, refine, re-route to a specific sub-agent or annotate the response.

Approval posts the response with a note that it has been checked. Rejection prevents bad information from reaching users and logs the failure. Refinement lets reviewers add guidance and regenerate the answer. Re-routing sends the question back to a specific agent with more context. Annotation stores structured feedback for future improvement.

This review loop helped build trust while also creating a feedback dataset for improving the system.

5. Letting users move fast without hiding uncertainty

The initial design held back AI-generated answers until an engineer approved them. That was safe, but it created a bottleneck. During busy periods, users could still wait too long.

Grab changed the process. Responses can now be posted immediately, but they are clearly marked as unreviewed. Engineers can still review and modify them afterwards.

This gives users fast answers while setting the right expectation. It also keeps the feedback loop intact.

6. Turning annotations into system improvements

Collecting annotations is useful only if the team learns from them.

Grab turned annotations into an improvement engine. Random annotations are pulled into offline evaluations, so the system is tested against real failure cases rather than only synthetic examples.

The team analyses patterns to see whether the Classifier is routing incorrectly, whether a specific agent has quality issues, whether certain query types cause hallucinations or whether particular table schemas confuse the system.

Quality metrics track annotation rates over time. A sudden rise in rejection rates signals that something needs investigation.

Annotations then guide targeted improvements: better prompts, stronger guardrails, enhanced agents, better few-shot examples, fine-tuning opportunities and regression test suites built from real failures.

The operational impact: faster resolution and more engineering capacity

The multi-agent system changed the team’s operating model.

The bots now autonomously handle most standard user inquiries and a meaningful portion of common enhancement requests. Simple questions that previously required manual investigation can now be answered within minutes.

The resolution time dropped by an order of magnitude. The support backlog was effectively removed. More importantly, the team reclaimed several FTEs worth of engineering capacity.

That capacity shifted from reactive support into proactive roadmap delivery.

This is the real win. The team did not just make support faster. It changed what engineers could spend their time on. Instead of tracing the same definitions, checking the same logs and answering the same data questions, they could return to higher-value platform work. And downstream users got faster answers.

Three principles for applying AI agents in data teams

Grab’s move from manual firefighting to AI-assisted operations came down to three principles.

Multi-agent architecture: Specialists over generalists

Specialised agents worked better than one generalist system because each agent had a narrower job. The Data Agent could focus on SQL and metadata. The Code Search Agent could focus on transformation logic. The On-call Agent could focus on incidents and pipeline health. The Enhancement Agent could focus on code changes.

This made the system easier to maintain, easier to improve and easier to debug.

Strategic human oversight: Building trust through transparency

Human review was not treated as an afterthought. It was part of the product design.

The system could generate fast answers, but reviewers could approve, reject, refine, re-route or annotate them. That created a trust layer for users and a learning loop for the engineering team.

The “unreviewed” label also mattered. It let users benefit from speed without pretending the AI response had the same authority as a checked engineering answer.

Focus on augmentation: Automating repetitive tasks

The agents were not built to replace engineers. They were built to take over the repetitive parts of the work: gathering context, running safe queries, checking logs, tracing code and preparing first-pass answers.

That left humans to focus on the work that actually needs human judgment: architecture, prioritisation, production approvals and building new capabilities.

For Grab’s Analytics Data Warehouse team, that was the shift. Less time spent answering the same operational questions. More time spent building the platform users depend on.

The full scoop

To learn more about this, check Grab's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Airtable cut archive storage costs by 100x and saved millions.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get into how Airtable pulled it off.

TL;DR

Situation

Airtable’s MySQL storage had grown to petabytes, with archive tables driving major cost and scale issues. Some databases were also getting close to the 64TB RDS limit.

Task

The team needed to move cold archive data out of MySQL without breaking revision history or slowing queries. They also had to keep durability, availability and enterprise requirements intact.

Action

Airtable built a two-tier system: recent data stayed in MySQL, old data moved to S3 as Parquet files. They used DataFusion for querying, plus Flink, compaction, validation, caching, indexes and bloom filters.

Result

Parquet made the archive dataset about 10x smaller and S3 was about 10x cheaper than MySQL. That led to roughly 100x lower storage costs and millions in annual savings.

Use Cases

Archiving data, reducing storage cost, Improving query latency

Tech Stack/Framework

Apache DataFusion, AWS MySQL RDS, Apache Flink, AWS SQS

Explained further

Context

Airtable’s storage team went into 2024 with a pretty blunt problem: too much archive data sitting in the wrong place.

Their AWS MySQL RDS footprint had grown to petabytes, some of the biggest databases were getting dangerously close to the 64TB RDS disk limit and one particular class of data was doing most of the damage: cell history and action log tables. Together, these acted as Airtable’s archive layer, powering revision history and helping with internal debugging. For some enterprise customers, that data also had to be retained for up to 10 years.

The issue was not that the data had no value. It clearly did. The issue was that MySQL was an expensive home for a workload that was mostly cold.

Most of this archive data was old, rarely touched and read-only except for hard deletion cases. When it was queried, the access patterns were fairly predictable: point selects and paginated range queries, always scoped to a specific base. That made it a poor match for row-oriented OLTP storage at this scale, especially when Airtable still needed interactive latency and strong durability and availability guarantees.

So the team built a new two-tier storage system. Recent rows would stay in MySQL. Older archive data would move to S3, be stored as Parquet files partitioned by base and queried through a new engine built on Apache DataFusion.

That shift did more than trim costs around the edges. The final archived dataset became 10x smaller than the original data in MySQL thanks to Parquet compression and S3 itself was around 10x cheaper per byte than MySQL storage. Put those together and the result was a storage layer that was about 100x cheaper.

Why Airtable needed a better archive layer

Airtable’s archive data had a few characteristics that mattered a lot:

• the overwhelming majority (trillions of rows) of it was old and infrequently accessed

• most reads were point selects or range queries used for pagination

• queries were always filtered to a single base

• old data was effectively immutable

• the data was keyed by MySQL’s autoincr_id, so it naturally followed insertion order

That combination is useful. It tells you the team did not need a general-purpose database for this layer. They needed something cheaper that still handled a narrow set of read patterns well.

The first key idea was to move archive data from MySQL into S3. S3 was already much cheaper byte-for-byte. The second was to store that data in Parquet and partition it by base. The third was to place a query engine in front of it that could answer interactive requests without forcing full scans.

This became a two-tier system: hot and recent archive rows in MySQL, older rows in S3-backed Parquet. That let Airtable keep the user-facing experience intact while steadily pulling massive amounts of cold data out of an expensive OLTP system.

Architecture overview

At a high level, the architecture is simple enough to explain in one sentence: archive data moved from MySQL into S3 Parquet files, and DataFusion was used to query those files directly with low enough latency to support product features.

What made it work was the alignment between storage layout and query patterns.

The team kept the Parquet schema close to the original MySQL schema. They also preserved ordering by autoincr_id, because a large share of their reads depended on point lookups or ranged access on that field. That meant the query engine could use Parquet metadata to narrow down what bytes to fetch from S3 instead of pulling full files.

They also partitioned files by base, which mattered just as much. Since queries were always scoped to a specific base, partitioning by base meant Airtable could avoid touching unrelated data entirely.

On top of that, Airtable stored S3 file location metadata in DynamoDB. This gave the client layer a clean way to register the right files with the query engine and helped support enterprise requirements such as regional data residency and encryption with customer-provided keys.

How Parquet fit the workload

Parquet was not just a cheaper format choice. It was the thing that made interactive querying on S3 plausible.

Unlike MySQL’s row-oriented InnoDB layout, Parquet is columnar. It stores each column contiguously and groups rows into row groups, with each row group containing column chunks. More importantly, Parquet files include metadata that query engines can exploit for pruning. File metadata carries offsets and sizes, while page-level metadata can include statistics such as min/max values and bloom filters.

That matters because Airtable’s queries were usually not broad analytical scans. They were targeted reads. If a query asks for a narrow range of autoincr_id values within one base, and the files are sorted on that column, the engine can inspect metadata and skip most row groups without reading them.

Airtable leaned directly into that. They kept the original schema mostly intact and preserved sorting by autoincr_id. Because of that, the engine could selectively download the relevant byte ranges from S3 rather than treat each Parquet file like a blob.

There was also a second major upside: compression. Thanks to the columnar layout, the archived dataset ended up about 10x smaller than the original MySQL version. That is a huge result on its own. Pair that with S3’s lower storage cost and the economics really shifted in Airtable’s favor.

Picking the right query engine

Once the storage format was settled, Airtable benchmarked several engines capable of querying Parquet in S3:

• AWS Athena

• DuckDB

• StarRocks

• DataFusion

Athena was ruled out quickly for latency reasons. Its API pattern of starting a query and polling for completion made it better suited to general OLAP workloads than user-facing interactive queries. Airtable was seeing query latencies in the seconds, which was too slow for revision history use cases. It also lacked the strong isolation Airtable cared about across bases.

DuckDB was useful, but not ideal for this workload. They found that query planning did not always use projection pushdowns effectively, which sometimes led to full file downloads. Simple point queries on one autoincr_id could still be subsecond, but overall it trailed DataFusion. The team still used DuckDB heavily during development because it was convenient for debugging Parquet contents from the command line.

StarRocks produced performance results comparable to DataFusion, but it came with the operational burden of running a full-time cluster in Kubernetes to serve relatively low-QPS cold-storage queries. Like Athena, it also did not give Airtable the same kind of strong base-level isolation.

That left DataFusion.

For Airtable, DataFusion hit the sweet spot. It was strong at exploiting Parquet metadata, it was extensible and because it is an embedded Rust library, the team could run it inside their existing worker architecture.

That brought a few clear benefits.

Operationally, there was no extra service to deploy and babysit. Isolation came for free because each base already had its own process boundary. And request affinity stayed high because the same workers kept serving the same bases, which later made caching hit rates excellent.

In other words, DataFusion fit Airtable’s architecture.

Migrating data out of MySQL

Designing the new storage layer was one thing. Moving petabytes of live data into it without breaking anything was another.

Airtable wanted a one-time migration process that would start cutting MySQL storage costs quickly. To get a consistent export view, they used AWS RDS snapshot capabilities, which produced large Parquet files of full tables. They had also prototyped direct SQL extraction into Parquet, but chose not to productionize that approach at scale. Snapshots were preferred because they ran against backup instances and avoided extra pressure on production systems.

The challenge was that these snapshots were massive table-level exports across many shards, while Airtable’s serving model required files partitioned by base.

So the team added a repartitioning and compaction pipeline.

First, Flink jobs parallelized across shard snapshots and repartitioned records by base into intermediate S3 directories. Then AWS Step Functions scanned those intermediate outputs and enqueued bases into SQS. From there, custom compactor code merged the files, merge-sorted them, deduplicated records and produced final serving Parquet files capped at 1GB each.

That 1GB size was not arbitrary. It was chosen through benchmarking as a good serving size with a useful density of page groups per file. Again, a small design detail with big latency implications.

Validating the migration

A migration like this lives or dies on validation.

Airtable first ran bulk validation to confirm that data had not been corrupted during export, repartitioning and compaction. For this, they spun up a StarRocks cluster and compared the serving Parquet files against the original RDS snapshots, finding zero cases of data corruption.

We covered this validation approach in more detail in an earlier piece on how Airtable made archive validation work at petabyte scale.

The important point is that bulk validation gave the team confidence that the migration pipeline itself had preserved the data correctly.

That is a strong result, though it only answered one part of the problem. The bigger challenge was all the new storage client logic that now sat around the data:

• a Rust query engine built with DataFusion

• integration into Node.js using napi-rs

• logic to combine MySQL and S3 results

• logic to identify which S3 files to read

• support for enterprise features like customer keys, data residency and hard deletes

Bulk validation could confirm the data files were right. It could not prove that the full end-to-end user experience would behave exactly as before.

So Airtable moved to shadow validation on live traffic. Requests continued reading from MySQL as normal, while the same queries were also executed in the background through the new system. That let the team compare outputs under real conditions and catch implementation issues before rollout.

The bugs they found were exactly the sort of things that show up when systems cross language runtimes and execution models:

• float precision mismatches between JavaScript and Rust’s serde JSON handling

• a sorting issue where DataFusion used lexicographic rather than numeric ordering

• a crashing SIGABRT issue tied to async napi-rs and Node.js worker threads

• latency problems

They resolved these before launch and before deleting the MySQL copies of the migrated data.

Fixing latency bottlenecks

Once staged rollout began, latency became the main battleground.

That is not surprising. S3-backed query systems usually do not fail because storage is too expensive. They fail because network round trips and scan inefficiencies make them annoyingly slow.

Airtable saw a mix of problems: inefficient query plans, too many S3 requests and cases where sparse filters still caused too much data to be downloaded. They responded with a few targeted optimizations.

Building a tiered cache for archive queries

Caching turned out to be one of the biggest wins.

Under the hood, DataFusion translates SQL queries into S3 GET operations. It fetches Parquet footer metadata, column chunk metadata and then decides what row groups and byte ranges need to be read. If every step involves another network trip, latency stacks up quickly.

So Airtable built a tiered caching system.

The first layer used DataFusion’s built-in cache support to store Parquet file metadata and S3 ListObjects results.

The second layer cached additional Parquet page header metadata in memory. Combined with the first layer, this reduced how often the engine had to round-trip to S3 during query planning. Airtable wrote a custom implementation around DataFusion’s parquet reader interfaces, which let them cache metadata results directly and add instrumentation. The result was a reported 99%+ cache hit ratio.

That number is believable in context because DataFusion ran inside per-base workers and the files themselves were partitioned by base. The system had strong locality by design.

Finally, Airtable added an on-disk cache for full Parquet files. This was reserved for a very small number of heavy bases with bad enough query patterns to justify the extra work and cost. Unlike metadata caching, downloading whole files is not something you want to do casually. But for outlier cases, it gave the team another escape hatch.

Building custom indexes for sparse queries

Not every query could be handled efficiently by the base file layout alone.

Most reads were anchored on autoincr_id, but Airtable also had filters on other fields that could reduce result sets dramatically. Examples included filtering by action type, filtering by row or excluding sync-generated updates.

For some bases, those additional conditions matched only a tiny slice of rows. In those cases, even if autoincr_id helped somewhat, reading broad sections of Parquet files was still wasteful.

So Airtable built a secondary indexing system.

Using DataFusion, they scanned Parquet files and wrote index data out as new Parquet files. The client layer knew how to query those indexes first, then use the result to build a more targeted query against the original archive files.

This was much easier to do because the data was effectively read-only. Airtable did not need to solve the usual headache of synchronizing constantly changing base tables and secondary indexes. Static data makes a lot of index ideas suddenly practical.

Using Bloom Filters for faster lookups

There was one more edge case: lower-QPS point lookups on a different unique identifier that was randomly distributed.

That broke the usual min/max pruning strategy. Since the identifier values were not ordered, Parquet statistics were not helpful. Without another technique, the engine would need to fetch and scan every page group before applying the filter.

Airtable could have solved this with another custom index, but they chose a simpler route: Parquet bloom filters.

Bloom filters are probabilistic membership structures. They can tell you if a value is definitely not present, or maybe present. False positives are possible. False negatives are not.

That property is enough for pruning. If the bloom filter says a page group definitely does not contain the target identifier, the engine can skip it safely. DataFusion already understood Parquet bloom filter metadata, so Airtable could rely on native support instead of bolting on another indexing layer.

Conclusion

Airtable’s storage team took a dataset that had clearly outgrown MySQL’s economics and built a system that matched the workload far better.

They moved petabytes of archive data out of MySQL, kept recent data in the transactional store, archived old rows into base-partitioned Parquet files on S3 and queried those files with an embedded DataFusion engine. Along the way, they layered in DynamoDB metadata registration, a large-scale migration pipeline, bulk and shadow validation, multiple caching layers, custom secondary indexes and bloom filter-based pruning.

The result was a storage system that stayed durable and queryable at interactive latency while cutting storage costs by around 100x and saving millions of dollars per year.

There is still more to do. Airtable’s first implementation focused on bulk migration so they could start saving money quickly. The longer-term goal is incremental archiving, likely through a CDC-style system such as Flink. That opens up a new set of engineering problems around compaction, index rebuilds and operations. There are also other log-like tables that could be migrated onto the same platform.

Still, the core idea is already proven.

If a dataset is mostly cold, mostly read-only and queried through a narrow set of predictable access patterns, keeping it in an expensive OLTP database is often just inertia dressed up as architecture. Airtable looked at the shape of the workload, changed the storage model to match it and got the kind of result every infra team wants: better economics without making the product worse.

The full scoop

To learn more about this, check Airtable's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Notion scaled its AI Q&A to millions of users.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Notion’s AI Q&A level up!

TL;DR

Situation

Notion launched AI Q&A on top of vector search and quickly faced massive demand across millions of workspaces. The initial system hit limits in capacity, onboarding speed and cost.

Task

Scale onboarding, keep indexes fresh and reduce rising infrastructure costs. At the same time, simplify a growingly complex architecture without hurting latency.

Action

They introduced dual ingestion paths, generation-based indexing, serverless architecture and migrated to turbopuffer. Then reduced recomputation with page state tracking and moved embeddings to Ray for unified compute.

Result

600x onboarding growth, 15x workspace growth and major cost reductions across layers. Latency improved and the system became simpler and more efficient.

Use Cases

Real-time search indexing, semantic search, document retrieval

Tech Stack/Framework

Apache Spark, AWS EMR, Apache Airflow, Apache Kafka, AWS S3, DynamoDB, Ray, turbopuffer

Explained further

Context

When Notion launched AI Q&A in November 2023, the core idea sounded simple enough: let people ask natural-language questions and retrieve relevant knowledge from across their workspace and connected tools. In practice, that meant building a vector search system that could ingest huge amounts of content, stay fresh as pages changed and do all of it at a cost that made sense at Notion scale.

That is the real story here. Not just “vector search powers AI” but what happens after launch, when adoption jumps faster than expected and the infrastructure underneath has to keep up. Over two years, the Notion team pushed that system through several big transitions: scaling onboarding, dealing with storage pressure, changing database architecture, reworking indexing logic and moving embeddings workloads onto Ray. The headline numbers are hard to ignore: 10x scale and roughly one-tenth the cost.

This is a good example of how modern AI infrastructure usually evolves. The first version gets the product live. The next few versions are about survival, then simplification, then cost, then latency, then getting rid of all the awkward bits that built up during the rush.

Vector search, explained through Notion’s lens

Traditional keyword search is literal. It works when users type the exact words that exist in the content. It starts falling apart when the wording changes but the meaning stays the same. Someone searching for “team meeting notes” may still want a page called “group standup summary,” but keyword search does not naturally understand that those are closely related.

Vector search solves that by representing text as embeddings. Instead of storing only words, it maps text into a high-dimensional space where semantically similar ideas sit closer together. That means retrieval is based on meaning, not exact phrasing.

For Notion AI, this matters a lot. The system needs to answer questions in natural language by finding useful content across a workspace and even across connected sources like Slack and Google Drive. That is exactly the sort of setup where semantic retrieval becomes more useful than plain lexical matching. A user is not thinking about the title of the page or the exact phrasing inside a paragraph. They are asking a question in their own words and expecting the system to bridge the gap.

That expectation becomes expensive very quickly.

Part 1: Scaling beyond what the original system expected

At launch, Notion’s ingestion and indexing pipeline had two paths.

The first was an offline path. Batch jobs running on Apache Spark would chunk existing documents, generate embeddings through an API and bulk-load those vectors into the vector database. This handled the heavy lifting for backfilling large amounts of existing content.

The second was an online path. Kafka consumers processed page edits in near real time so live workspaces stayed up to date with sub-minute latency.

It is a practical split. The offline side handles the backlog and large initial loads. The online side keeps things fresh once a workspace is active. Together, the two-path setup gave Notion a way to onboard workspaces at scale without sacrificing freshness for day-to-day edits.

The vector database itself ran on dedicated ‘pod’ clusters, where storage and compute were coupled. The Notion team designed sharding in a way that echoed their Postgres setup: workspace ID was the partitioning key, routing used range-based partitioning and a single config referenced all shards.

That all made sense on paper. Then the product launched and demand was overwhelming.

Notion quickly built up a waitlist of millions of workspaces that wanted access to Q&A. The problem was no longer whether the system worked. It was how fast it could onboard people without cracking under the pressure.

When the indexes started to fill up

Only a month after launch, the original indexes were already nearing capacity.

That is the kind of problem that sounds good in product meetings and bad in infrastructure meetings. If the indexes filled up, Notion would have to pause onboarding. That would slow down rollout and delay access for everyone waiting.

The team had two obvious options.

One was to re-shard incrementally. Clone data into another index, delete half, repeat and keep doing that every couple of weeks as new customers came in.

The other was to re-shard for the final expected volume. But their vector database provider charged for uptime, so over-provisioning would have been painfully expensive.

Instead, the Notion team went with a third approach. When a set of indexes got close to full, they provisioned a new set and directed all newly onboarded workspaces there. Each set was assigned a generation ID, which determined where reads and writes should go.

It is not the prettiest long-term design, but it was a smart short-term move. It avoided repeated re-shard operations and kept onboarding moving. Sometimes the right scaling decision is not the most elegant one. It is the one that buys breathing room without stopping the business.

Turning onboarding into a throughput problem

Even with the architecture in place, the initial onboarding rate was nowhere near enough. At launch, Notion could onboard only a few hundred workspaces per day. At that pace, clearing a multi-million waitlist would have taken decades which is obviously not a real option.

So the team pushed hard on throughput. Using Airflow scheduling, pipelining and Spark job tuning, they dramatically increased capacity.

The results were big:

• Daily onboarding capacity increased by 600x

• Active workspaces grew 15x

• Vector database capacity expanded 8x

By April 2024, the Q&A waitlist was cleared.

That is the kind of milestone that looks clean in hindsight but it came with a cost. Managing multiple generations of databases helped during the hypergrowth phase but it also added operational complexity and financial overhead. The team had solved the immediate scaling problem, but the architecture was starting to feel heavy.

That set up the next phase of the story.

Part 2: Cost becomes the next constraint

In May 2024, Notion migrated its embeddings workload from the original dedicated ‘pod’ architecture to a serverless setup that decoupled storage from compute and charged based on usage instead of uptime.

The effect was immediate. Costs dropped by 50 percent from peak usage, translating into several millions of dollars in annual savings.

That alone would have made the migration worthwhile, but the serverless design also fixed two practical problems. First, it removed the storage capacity constraints that had become a serious scaling bottleneck. Second, it simplified operations because the team no longer had to provision capacity ahead of demand.

Still, even after cutting costs in half, the annual run rate for vector database spend was still in the millions. From an engineering point of view, this is where things get interesting. The easy win had already happened. Now the team had to go after deeper structural gains.

A new search foundation (turbopuffer)

While working on the first round of savings, Notion also evaluated alternative search engines. turbopuffer stood out because it offered significantly lower projected costs.

At the time, turbopuffer was a newer player in search. Its architecture was built on object storage with a focus on cost-efficiency and performance. It also supported both managed and bring-your-own-cloud deployment models and it made bulk modification of stored vector objects easier.

That combination lined up well with what Notion needed.

After a successful evaluation, the team decided to migrate its entire multi-billion-object workload to turbopuffer in late 2024. Since they were already making a provider switch, they used the migration as a chance to clean up the broader architecture too.

Several changes happened together.

First, they fully re-indexed the corpus, increasing write throughput in the offline indexing pipeline to rebuild everything in turbopuffer.

Second, they upgraded the embeddings model during the migration to be more performant.

Third, they simplified the architecture. turbopuffer treats each namespace as an independent index which removed the need to think about sharding and generation-based routing in the same way as before.

Finally, they handled the cutover gradually, migrating one generation at a time and validating correctness before moving on.

This is a strong pattern: if a migration is painful anyway, use it to pay off other infrastructure debt at the same time.

The outcome was solid on several fronts:

• 60 percent cost reduction on search engine spend

• 35 percent reduction in AWS EMR compute costs

• p50 production query latency improved from 70–100ms to 50–70ms

That is a meaningful improvement across cost and performance, which is not always easy to pull off together.

Avoiding full reprocessing with page state tracking

The next optimization went after a very expensive inefficiency in the indexing pipeline.

Notion pages can be long, so the team chunks each page into spans, embeds each span and stores those vectors with metadata such as authors and permissions. In the original implementation, any edit to a page or its properties triggered a full re-chunk, full re-embed and full re-upload of all spans on that page.

That meant even a tiny change could trigger a lot of unnecessary work.

The team narrowed the problem down to two things that actually mattered:

1. The page text changes which means embeddings need updating

2. The metadata changes which means metadata needs updating

To detect those cases, they tracked two hashes per span: one hash for the span text and another for the metadata fields. They chose 64-bit xxHash because it offered a good balance of speed, simplicity, low collision risk and storage footprint.

For caching, they used DynamoDB. Each page had one record containing the state of all spans on that page, including text and metadata hashes.

The win came from using that state to avoid unnecessary work.

Case 1: The page text changes

Imagine Herman Melville editing Moby Dick halfway through a page. Before this improvement, the whole page would have been re-embedded and reloaded. After the change, the system chunks the page, fetches the previous state from DynamoDB and compares text hashes span by span. It can then detect which spans actually changed and only re-embed and reload those.

That is the kind of fix that getting the balance right matters. Miss a changed span and search quality suffers. Reprocess too much and cost stays high.

Case 2: The metadata changes

Now imagine Melville updates permissions so the page becomes visible to everyone. The permissions metadata changes but the text does not.

Previously, that still meant re-embedding and reloading the entire page. With the new approach, Notion compares both text and metadata hashes. If the text hashes are unchanged but metadata hashes differ, the system skips embedding entirely and issues a PATCH command to the vector database to update only the metadata. That is much cheaper than recomputing embeddings.

Across these changes, the Page State Project reduced data volume by 70 percent. That saved money on both embeddings API costs and vector database write costs.

Moving embeddings to Ray (indexing)

In July 2025, Notion started migrating its near real-time embeddings pipeline to Ray on Anyscale.

The motivation came from several pain points in the earlier setup.

One was the ‘double compute’ problem. Spark on EMR handled preprocessing like chunking, transformations and API orchestration, but embeddings themselves were still generated through an external provider that charged per token. So the team was paying for both preprocessing infrastructure and embedding API usage.

Another issue was endpoint reliability. Fresh search indexes depended on the stability of an external embeddings API.

The third problem was clunky pipelining. To smooth traffic and avoid API rate limits, the team had built a multi-step handoff process where Spark jobs passed batches through S3. It worked but it was clunky.

Ray and Anyscale gave Notion a cleaner path.

Ray let the team run open-source embedding models directly, which meant more model flexibility and less dependence on external providers. By consolidating preprocessing and inference onto a single compute layer, they could cut out the double-compute setup. Ray also supports pipelining CPU-bound work such as chunking and page-state detection with GPU-bound embedding generation on the same nodes, which helps keep utilization high.

There was also a developer productivity angle. Anyscale workspaces let engineers write and test pipelines from their preferred tools without having to provision infrastructure manually.

And on the product side, self-hosting embeddings removed a third-party API hop from the user-facing path, which helped reduce end-to-end latency.

The rollout is still ongoing, but early results suggest a 90+ percent reduction in embeddings infrastructure costs. That is a major shift in how the economics of the system work.

Real-time query embeddings on Ray (serving)

Indexing is only half the picture. When users or agents search in Notion, queries must also be embedded on the fly before the vector database can be searched.

That makes serving latency-sensitive. The embedding has to happen fast enough that the search still feels responsive.

Hosting large embedding models is not trivial. GPU allocation, ingress routing, replication and autoscaling all matter, especially when traffic is uneven and expectations for responsiveness are high.

Ray Serve helped Notion here by handling much of that operational layer out of the box. The team could wrap open-source embedding models in persistent deployments that stay loaded on GPU, configure request batching and replication and manage the serving setup with normal Python code plus YAML-based infrastructure configuration.

That is a pretty practical endpoint for the broader journey.

What started as a vector search stack built quickly enough to launch AI Q&A turned into a much more refined system: simpler in some places, more selective in others, cheaper across multiple layers and faster where users feel it. The interesting part is not any single tool choice. It is how the Notion team kept removing bottlenecks one by one: storage limits, awkward shard routing, redundant recomputation, external API dependence and fragmented compute layers.

That is usually what mature AI infrastructure looks like in the real world. Not one giant redesign. A sequence of sharp decisions, each fixing the thing that has become too expensive, too slow or too annoying to keep around.

The full scoop

To learn more about this, check Notion's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how LinkedIn ingests data at scale.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Venice: LinkedIn’s data storage platform

TL;DR

Situation

Venice powers LinkedIn’s AI-driven products and has scaled to 2,600+ stores with workloads spanning bulk loads, streaming updates and active/active replication. The ingestion pipeline had to handle throughput-heavy, CPU-heavy and latency-sensitive traffic under eventual consistency.

Task

Redesign ingestion to scale to 230M writes/sec while preserving ordering and protecting read and write SLAs. Support hybrid stores, partial updates and multi–data center replication without destabilizing clusters.

Action

Scaled bulk ingestion with partition tuning, shared consumer/writer pools and direct SST writes; tuned RocksDB via compaction triggers and BlobDB to manage amplification. Optimized CPU-heavy paths using Fast-Avro and parallel processing, then enforced priority pools and adaptive throttling to protect current-version latency.

Result

Venice now handles 175M+ key lookups/sec and 230M+ writes/sec in production. It maintains a write latency SLA under 10 minutes while safeguarding read latency as the top priority.

Use Cases

Large-scale feature stores, real-time recommendation systems, hybrid data serving, low-latency notification

Tech Stack/Framework

Apache Spark, Apache Samza, Apache Kafka, RocksDB, Fast-Avro, Adaptive Throttling

Explained further

Background

Venice is an open-source derived data storage platform and LinkedIn’s default storage layer for online AI use cases. It sits behind products like People You May Know, feed, videos, ads, notifications, the A/B testing platform, LinkedIn Learning and more.

Since Venice launched internally in 2016 it has scaled from a handful of stores to over 2,600 production stores. The workloads also evolved a lot. It started with “just bulk load a dataset” and grew into a mix of:

• Bulk loading huge offline datasets

• Nearline streaming updates

• Active/active replication across data centers

• Partial updates that merge fields and collections

• Deterministic write latency expectations under eventual consistency

This post walks through how the ingestion pipeline was revamped to hit 230 million records per second in production, what changed across the architecture, which optimizations moved the needle and how different workload types get tuned. A lot of these ideas are portable if you run any distributed ingestion system where ordering, throughput and predictable latency all matter at once.

Venice overall ingestion pipeline

At a high level, store owners write to Venice through three paths:

1. Bulk loads from an offline processing platform (example: Spark)

2. Nearline writes from a streaming processing platform (example: Samza)

3. Direct writes from online applications

No matter which path you take, the writes all pass through an intermediate PubSub broker layer. From there, the Venice Storage Node (VSN) consumes messages and persists data locally using RocksDB (an embedded key-value store).

The pipeline sounds straightforward until you operate it at scale. The same ingestion path has to support very different workloads. Some are throughput-driven (bootstrapping a massive store). Some are latency-driven (current-version updates). Some are CPU-heavy (partial updates and conflict resolution). Some are I/O-heavy (compaction, SST churn).

The following sections will look at the challenges and how the LinkedIn team resolved them.

Use case 1: bootstrapping from offline dataset

Venice users can run bulk load jobs using offline processing platforms such as Spark to push new data versions to Venice stores. The hard part is performance for large or massive stores. If you want to find bottlenecks you need to understand the ingestion path end to end.

What happens during a bulk load

• A Venice Push Job (VPJ) creates a new version topic for the new store version, split into multiple partitions

• The Spark job uses a map-reduce framework to produce messages to that version topic

• It keeps one reducer per topic partition so message ordering is preserved

• On the other side, the VSN spins up consumers, reads messages and persists them into RocksDB

• There is one RocksDB instance per topic partition

So you can hit bottlenecks in three obvious places:

1. producing

2. consuming

3. persisting

Production experience says you will hit all three, just not on the same day.

Improving producing and consuming throughput

The usual first lever is increasing the number of partitions for large stores so you can use more of the PubSub cluster capacity. More partitions tends to mean more parallelism and more throughput.

But it comes with trade-offs:

• more partitions means more management overhead across Venice and PubSub

• there is a throughput ceiling per PubSub broker

So partition count is not a free lunch. It’s a knob that buys you throughput and charges you complexity.

Enhancing consumption scalability

To keep up with production, VSN uses shared consumer pools across all hosted stores.

Instead of “one store version, one set of consumers,” each store version can use multiple consumers by distributing hosted partitions among them. The point is to keep multiple connections per PubSub broker to speed up consumption (similar to a Download Manager).

The pool approach also does something boring but important: it sets an upper limit on total consumers which puts a ceiling on cost.

Optimizing I/O performance

VSN uses a shared writer pool to persist changes concurrently across multiple RocksDB instances and use local SSD capacity effectively.

Ordering is critical in Venice so for any given RocksDB instance there is only one writer actively writing to it. You still get concurrency across instances, not inside one instance which is the compromise that keeps ordering intact.

Minimizing memory overhead

Because messages for a partition are strictly ordered (thanks to the map-reduce framework), Venice uses RocksDB’s SSTFileWriter to generate SST files directly. That significantly reduces memory overhead during ingestion.

Ingestion workflow in Venice Server

Put together, the optimized workflow is basically: use the PubSub layer for distribution, use consumer pools for scalable reads, use writer pools for SSD throughput, preserve ordering by design and avoid memory blowups by writing SST files directly.

Use case 2: hybrid store

Venice supports Lambda architecture style use cases by merging updates from both bulk loads and nearline writes. Users query a single store and get a unified view.

Venice hybrid store workflow

How it works:

• each bulk load creates a new store version

• that version has a new Kafka topic and a new database instance

• real-time updates produced by a Samza job via a real-time topic are appended to both version topics to keep them current

• once the new version catches up fully, it is swapped in as the active version to serve reads

The hybrid store is important because it gives you a clean “new version build” story without losing real-time freshness. But it creates a new challenge: the database transitions from read-only to read-write.

That’s where RocksDB tuning matters, because duplicates start showing up more often. Keys get updated or deleted after they were inserted. RocksDB uses log compaction to remove stale entries, but that compaction has overhead: scan, merge, rewrite SST files, consume CPU, I/O and disk.

So the core problem becomes: tune RocksDB so you can balance three competing types of pain.

• Write amplification: bytes written to storage vs bytes written to the DB

• Read amplification: number of disk reads per query

• Space amplification: size of DB files on disk vs the actual data size

Venice uses leveled compaction by default and relies primarily on two methods to balance those trade-offs.

1. Tuning the compaction trigger

The key setting here is:

• level0_file_num_compaction_trigger

This controls the max number of files allowed in Level-0. Once you exceed it, compaction kicks in to push SST files from Level-0 to Level-1 and onward as upper levels fill.

Why it matters:

• higher threshold → fewer compactions → lower write amplification

• but also more Level-0 files → higher read amplification since reads may need to scan multiple files

• plus higher space amplification because duplicates hang around longer

Venice tunes this per cluster because clusters have different bottlenecks:

• memory-serving clusters want data in RAM to speed up lookups. Memory is the limiting resource, so they set a lower threshold to reduce space amplification

• disk-serving clusters are often limited by disk I/O, so they set a higher threshold to reduce compaction frequency and lower disk write rate

This is a practical tuning philosophy: tune to your real bottleneck, not a generic best practice.

2. RocksDB BlobDB integration

BlobDB is aimed at large-value workloads through key-value separation:

• Large values go into blob files

• LSM tree stores small pointers

This avoids copying large values repeatedly during compaction, reducing write amplification. The cost is additional space amplification because blobs can become unreferenced and require garbage collection.

For Venice, BlobDB integration reduced write amplification significantly in multi-tenant clusters, especially for large-value use cases. The reported impact here is big: more than a 50% reduction of disk write throughput. That matters because it avoided scaling out clusters when CPU and storage space were still available.

The win here is: you stop paying the compaction tax over and over on the same large payloads.

Use case 3: Active/active replication with partial update

Venice guarantees eventual consistency, not strong consistency. That matters because it means you cannot just do read-modify-write operations directly due to write delays.

To handle this, Venice introduces partial update, a specialized operation that supports field-level updates and collection merges.

Venice partial update workflow

Inside the Venice server, the leader replica:

• decodes the incoming payload

• applies the update

• re-encodes the result

• writes to the local database

• writes to the Version Topic

• follower replicas consume the merged results

Most of that is CPU-heavy.

Then the platform evolved further with active/active replication across multiple data centers. The key mechanism is deterministic conflict resolution (DCR), similar to CRDTs. Venice tracks update timestamps at row and field levels, compares incoming timestamps with existing ones and decides to apply or skip.

Venice Active/Active workflow

Now the leader replica has even more to do for DCR:

• timestamp metadata lookup

• decoding

• encoding

Again: CPU heavy. So the optimisation below focus on CPU efficiency.

1. Fast-Avro adoption

Fast-Avro was originally developed by RTBHouse but LinkedIn took over maintenance under the LinkedIn namespace and introduced many optimizations.

The key idea: Fast-Avro is an alternative to Apache Avro serialization and deserialization using runtime code generation which performs significantly better than the native implementation. It supports multiple Avro versions at runtime and is widely adopted inside LinkedIn.

Venice fully integrated Fast-Avro and saw, in one major use case, up to a 90% improvement in deserialization latency at p99 on the application side.

2. Parallel processing

In the traditional pipeline, DCR and partial update operations were executed sequentially, record by record within the same partition. That leads to CPU underutilization.

Venice introduced parallel processing so multiple records can be handled concurrently within the same partition before producing them to the version topic, while still preserving strict ordering in the final step.

Result: significantly improved write throughput for these complex record types.

Use Case 4: Active/active replication with deterministic write latency

Eventually consistent systems still get judged by human expectations. People want their writes to show up and they want it to happen predictably.

Venice is versioned and can ingest backup, current and future versions concurrently in a single server instance. In practice though, only the current version serves reads so deterministic write latency guarantees focus mostly there.

To improve determinism, Venice introduced a pooling strategy in ingestion with different priorities for different workload types. The Venice consumer phase is the first phase in the server ingestion pipeline and controlling the polling rate via pools is how prioritization happens.

Broad priority tiers:

• top priority: active/active and partial update workloads for the current version on the leader replica (CPU-intensive and latency-sensitive)

• next: other workload types targeting the current version

• then: active/active or partial update workloads for backup or future versions on the leader replica

• finally: everything else in a lower-priority bucket

This design is trying to do a few practical things:

• isolate CPU-heavy workloads so they don’t slow down lighter ones

• prioritize the current version so the most up-to-date data flows smoothly

• keep the number of pools limited to avoid resource management turning into a second job

The catch is tuning. Clusters see different workloads, store behavior varies widely even within one cluster, throughput swings over time and read traffic changes throughout the day. Static configs force you to tune for worst-case, which wastes resources most of the time.

So Venice introduced adaptive throttling: dynamically adjust ingestion based on recent performance.

• if the system is within agreed SLAs, ingestion rates are adjusted according to priorities

• if an SLA is violated, ingestion is throttled back immediately

Defining the SLAs matters. Venice focuses on two key criteria:

1. Read latency SLA: highest priority. Never violate read latency SLAs, even if it costs ingestion throughput

2. Write latency SLA for the current version: while read latency SLAs are met, write latency for the current version becomes top priority, pools are tuned proportionally to maximize utilization and throughput

Wrapping up

With these optimizations, Venice at LinkedIn handles:

• Over 175 million key lookups per second

• Over 230 million writes per second

• While maintaining a write latency SLA under 10 minutes

The full scoop

To learn more about this, check LinkedIn's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Grab detects data issues in real-time.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Grab’s real-time work!

Read more

Today we will look at how Uber moved from batch to streaming in their data lake.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Uber’s streaming solution

Read more

Today we will look at how Snap unified Spark, ML workflows and 10k+ daily jobs under one platform.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Snap’s ML platform transformation.

Read more

Lately, I’ve been thinking about starting a new series where people working in data share how they got here, what they’ve learned along the way and what their day-to-day looks like.

So, I’m kicking it off today with , Senior Data Engineer at Workpath and writer of The Pipe and The Line newsletter.

The Pipe & The Line

Hands-on guides, tools, and experiments to sharpen your Data & AI Engineering skills from someone who learned it all in the wild.

By Alejandro Aboy

We talked about how he went from marketing to data engineering, what his workflow looks like, why he was called an octopus and why he thinks “big data” is a fool’s errand for most teams.

So without further ado, let’s get into it!

Read more

Today we will look at how Shopify built a real-time data pipeline at 400TB scale.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to change data capture at Shopify!

Read more

Today we will look at how Grab made real-time processing faster for its users.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to real-time processing at Grab!

Read more

Today we will look at how Expedia Group monitors A/B tests at a large scale

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people and they subscribe to the newsletter.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get the real-time monitoring of A/B tests by Expedia

Read more

Today we will look at how Bolt tracks payments at scale.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people and they subscribe to the newsletter.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to how Bolt deals with processing payments from millions of customers

Read more

Today we will look at how Flipkart solved the problem of calculating delivery date.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Flipkart’s challenge of calculating delivery date at scale

Read more

Quick note before this week’s deep dive. Thanks for reading and subscribing, I really do mean it. If you’ve got feedback, just hit reply. I read every response.

Data Tinkerer has always been about sharing what actually works in data, beyond just tools and tech. The deep dives will keep coming but I want to start spotlighting the stuff we don’t talk about enough: the day-to-day challenges, business outcomes, the challenges and the learnings.

I want to feature stories from people in data roles: senior data engineers, lead analysts, heads of data, you name it. If you’ve got a story, lesson, recent technical win or even a battle scar from the data trenches, let’s get it in front of almost 1,000 smart and engaged peers.

You don’t need to be a “writer”, I’ll help your story shine. Plus, guest contributors get a shoutout in the newsletter and on LinkedIn (If you want).

Keen to share your data story? Just reply to this email or message me on Substack and we’ll tee it up.

Now …

Read more

Today we will look at how Canva solved the surprisingly messy problem of counting at scale

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Canva’s work to build a scalable data pipeline

Read more

TL;DR

Read more

Situation

HubSpot's backend performance team identified that Amazon S3 storage costs accounted for approximately 45% to 50% of daily expenses, with the 'hubspot-live-logs-prod' bucket alone responsible for 20% of these costs.

Task

The team aimed to reduce storage costs by addressing the inefficiencies in their logging system, particularly focusing on the large volumes of raw JSON logs that were not being efficiently compacted.

Action

• Log Retention Review: They discovered that raw JSON logs were retained for 730 days, while compressed ORC logs were kept for 460 days. Aligning the retention period to 460 days for both formats reduced unnecessary storage.​

• Improved Compression: By enhancing their Spark compaction process, they increased the conversion rate of raw JSON logs to the more storage-efficient ORC format, achieving a compression ratio where ORC logs were about 5% the size of the original JSON logs.

Result

These measures led to a 55.7% reduction in monthly JSON log storage costs, translating to annual savings in the seven-figure range. Additionally, engineers experienced faster log query times, with some reporting reductions from 30 minutes to just 36 seconds.

Use Cases

Cost monitoring, Log retention, Log volume reduction

Tech Stack/Framework

AWS Athena, Amazon S3, Apache Spark, Apache Mesos, Redash

Explained Further

Saving Millions on Logging

Read more

Situation

Reddit's advertising platform processes thousands of ad engagement events per second, necessitating real-time validation and enrichment to ensure accurate reporting and prevent budget overdelivery.

Task

Develop a scalable, real-time ad event validation system capable of efficiently handling high event volumes while maintaining performance and reliability.

Action

The engineering team developed the Ad Events Validator (AEV) utilizing Apache Flink to correlate ad server events with user engagement events. To overcome issues related to large state sizes and resource demands, they implemented:

• Field Filtering: Conducted a thorough analysis of downstream data consumption, establishing an allowlist that significantly reduced the event payload size by 90%, leading to CPU and memory usage reductions of 25% and 60%, respectively.

• Tiered State Storage: Integrated Apache Cassandra for external state storage, effectively reducing in-memory state size and enhancing the efficiency of checkpointing and system recovery processes.

Result

These strategic enhancements resulted in a more scalable and cost-efficient AEV system, improving overall performance and operational effectiveness.

Use Cases

Real-Time Event Validation, Data Enrichment, Resource Optimization

Tech Stack/Framework

Apache Flink, Apache Kafka, Apache Cassandra

Explained Further

Background

Reddit processes thousands of ad engagement events per second. These events require validation and enrichment before being sent to downstream systems. Key components of this validation process include applying a standardized look-back window and filtering out suspected invalid traffic.

In addition to a batch validation pipeline, a near real-time pipeline improves budget spend accuracy and provides advertisers with real-time insights into campaign performance. This real-time component, known as the Ad Events Validator (AEV), is built using Apache Flink. AEV matches ad server events with engagement events and writes the validated results to a separate Kafka topic for downstream consumption.

Building and maintaining AEV though, presented several challenges to the Reddit team

1st Challenge: Addressing High State Size Issues

Read more

TL;DR

Situation

Yelp's ad revenue relies on predicting which ads users are likely to click on, using a model called "Wide and Deep Neural Network." Initially, training this model on 450 million data samples took 75 hours per cycle, which was too slow. Yelp wanted to handle 2 billion samples and reduce training time to under an hour per cycle.

Task

The goal was to speed up the training process by improving how data is stored and read, and by using multiple GPUs to handle more data at once.

Action

• Data Storage: Yelp stored the training data in Parquet format on Amazon's S3 storage, which works well with their data processing system, Spark. They found that a tool called Petastorm was too slow for their needs, so they developed their own system called ArrowStreamServer. This new system reads and sends data more efficiently, reducing the time to process 9 million samples from over 13 minutes to about 19 seconds.

• Distributed Training: Yelp initially used a method called MirroredStrategy to train the model on multiple GPUs but found it didn't work well as they added more GPUs. They switched to a tool called Horovod, which allowed them to efficiently use up to 8 GPUs at once, significantly speeding up the training process.

Result

By implementing these changes, Yelp achieved a total speed increase of about 1,400 times in their model training. This means they can now train their ad prediction models much faster, allowing them to handle more data and improve their ad services.

Use Cases

Large-Scale ML Training, ML Training Optimisation, Enhancing Data Pipeline Efficiency

Tech Stack/Framework

TensorFlow, Horovod, Keras, PyArrow, Amazon S3, Apache Spark

Explained Further

The Challenge

Read more

TL;DR

Situation

Meta handles vast amounts of user data across its platforms, requiring strong privacy controls to protect sensitive information. A critical component of this effort is data lineage, which helps trace how data moves across different systems, ensuring compliance with privacy policies like purpose limitation.

Task

Meta needed a scalable and automated way to track data lineage across millions of assets, including databases, web services, and AI systems. This required moving beyond manual data flow documentation to a more robust, automated discovery process.

Action

• Data Flow Collection – Used static code analysis, runtime instrumentation, and input/output matching to track data across stacks (Hack, C++, Python, SQL).

• Privacy Probes – Captured real-time runtime signals, identifying how and where sensitive data is logged, stored, or transformed.

• Automated Lineage Graphs – Created scalable data flow visualizations to streamline privacy control implementation.

• AI & Data Warehouse Integration – Ensured end-to-end traceability across AI models, databases, and batch-processing systems.

• Iterative Filtering Tool – Allowed developers to refine lineage graphs, isolating relevant data flows and removing noise.

Result

Meta’s data lineage system reduced engineering time, improved compliance accuracy, and automated privacy enforcement. It enabled developers to quickly identify and secure sensitive data flows while ensuring continuous monitoring at scale. These innovations enhanced user data protection across Meta’s ecosystem

Use Cases

Privacy Enforcement, Compliance Monitoring, Data Lineage

Tech Stack/Framework

Python, SQL, C++, PyTorch, Presto, Spark

Explained Further

Meta's Privacy Aware Infrastructure (PAI) is designed to embed privacy controls within its systems, ensuring user data is handled responsibly. A foundational element of PAI is data lineage, which traces the journey of data across various platforms, providing a comprehensive view of its flow from collection to processing and storage. This capability is crucial for implementing privacy measures like purpose limitation, which restricts data usage to specific, intended purposes

Understanding Data Lineage at Meta

Data lineage involves mapping out how data moves through Meta's vast ecosystem, connecting source assets (e.g., database tables where data originates) to sink assets (e.g., tables or systems where data is stored or processed).

This mapping is essential for:

Read more