It’s time for another round-up on all things data and AI!

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Without further ado, let’s get to the round up for April!

Data science & AI

• Components of a Coding Agent (19 minute read)
  provides a great break down of the six core components of coding agents, showing why tools, repo context, memory, caching and context management often matter as much as the underlying LLM.

  

• The Revenge of the Data Scientist (8 minute read)
  Hamel Husain argues data science is making a comeback in the LLM era because evals, traces, metrics, labels and experimental design are exactly the skills needed to make AI systems work reliably.

• Machine Learning Visualized (4 minute read)
  Gavin Hung’s Machine Learning Visualized is a free, notebook-based guide that derives ML algorithms from first principles and uses interactive visuals to show how models train, learn and converge.

  

• Your AI Might be Lying to Your Boss (17 minute read)
  William O’Connell argues while AI IDEs can help developers save time, their metrics around AI-generated code are inaccurate and can't be trusted because of their self-serving interests.

• Why Most Time Series Models Fail Before They Start (18 minute read)
  This article explains why many time series models fail before they start, showing how stationarity checks, visual diagnostics and transformations like differencing can stop models from chasing trends instead of learning real signal.

• Multi-agent coordination patterns: Five approaches and when to use them (5 minute read)
  Anthropic breaks down five multi-agent coordination patterns, showing when to use generator-verifier, orchestrator-subagent, agent teams, message bus or shared state instead of overcomplicating agent systems too early.

  

• AIs can now often do massive easy-to-verify SWE tasks and I’ve updated towards shorter timelines (20 minute read)
  Ryan Greenblatt argues that AIs are now much better at massive, easy-to-verify software engineering tasks than expected, which has pushed him toward shorter AI timelines even though judgment-heavy work remains a major bottleneck.

• Being a Staff+ Data Scientist in 2026 (14 minute read)
  argues staff+ data science is less about fancy models and more about navigating messy stakeholder incentives, uncertainty, self-serve myths and the human work needed to turn data into decisions.

• How Pinterest Used Multimodal AI to Help Millions of Shoppers (14 minute read)
  Pinterest turned billions of products into 4.2 million shopping landing pages and improved search performance by 35%. This piece breaks down how Pinterest used vision-language models, contrastive learning and distributed inference to make products easier to discover.

Data engineering

• Architectural Foundations & Infrastructure (12 minute read)
  kicks off a good data platform architecture series, arguing that good infrastructure choices start with business requirements, data shape and boring fundamentals like scalability, resilience, modularity, observability and cost.

  

• The 5 Silent Failures in Data Pipelines (11 minute read)
  breaks down five silent pipeline failures, from schema drift to stale data and brittle logic, showing how bad data can reach dashboards without throwing a single error.

  

• Data Observability Fundamentals for Data Engineers (18 minute read)
  Good article by explaining data observability as a tool-agnostic way to catch silent pipeline failures, using freshness, volume, schema, quality and lineage signals to keep data and AI systems trustworthy.

• The Power of Data Sketches: A Comprehensive Guide (19 minute read)
  explains how data sketches trade perfect accuracy for tiny memory, fast queries and bounded error, making them useful when exact counts and aggregations become too expensive at scale.

  

• Stop Letting Tools Lead Your Platform Decisions (5 minute read)
  argues data platform decisions should start with use cases, constraints and users, not fashionable tools, because the simplest architecture is often the one that actually fits.

• Databases Were Not Designed For This (16 minute read)
  argues agentic AI makes databases riskier, so teams need defensive patterns like least privilege, timeouts, idempotency, soft deletes and query tagging before giving agents access to production data.

• The Last Mile to Apache Iceberg - Building a Basement Data Platform (13 minute read)
  shows how he solved the ‘last mile’ into Apache Iceberg by building a tiny low-cost event ingestion path for side-project analytics without Kafka, SaaS pipelines or a scary cloud bill.

• Lyft Data Tech Stack (5 minute read)
  breaks down Lyft’s data stack, showing how Kafka, Flink, Trino, Airflow, Flyte and 100+ PB on S3 support real-time analytics, ML and millions of daily rides.

  

• 30 BI Engineering Interview Questions That Actually Matter in the AI Era (32 minute read)
  Anusha Kovi reframes BI engineering interviews for the AI era around the skills that still matter: SQL judgment, semantic modeling, data governance, trustworthy metrics and turning messy business questions into reliable analytics.

• How Airtable Saved Millions by Cutting Archive Storage Costs by 100x (16 minute read)
  Airtable moved petabytes of archive data out of MySQL, made storage 100x cheaper and kept interactive query latency intact. This piece breaks down how Airtable handled the migration, why it chose DataFusion and the optimizations that made cold storage feel fast.

Data analysis and visualisation

• The data role is being reborn (8 minute read)
  thinks the data role is being reborn, with AI shrinking old output-heavy analyst work while increasing the need for judgment, business context and decision-shaping.

• Stop Coloring Retention Tables the Classic Way (3 minute read)
  shows that retention tables become much more useful when you color cohorts by deviation from the period average, rather than using classic gradients that mostly show the obvious decline over time.

• Dashboard rot as org attention grave markers (10 minute read)
  Randy Au argues dashboard rot is not really a dashboard problem, but a sign of shifting organisational attention, stale priorities and abandoned decisions fossilised as BI clutter.

Other interesting reads

• We’re in 1905: Why Electricity (Not Dot-Com) Is the Right AI Analogy (12 minute read)
  Interesting article by arguing AI is less like the dot-com boom and more like early electricity: the technology works but the real productivity gains will only come when companies redesign their workflows, org structures and data “factories” around it.

• Keeping up with the GPTs (24 minute read)
  thinks compute-poor AI labs can narrow the gap through efficiency tricks like distillation, but compute still dominates, so catching up with frontier labs is much easier than overtaking them

  

• Seeing like a spreadsheet (23 minute read)
  Interesting article by discussing how spreadsheets reshaped American business into a numbers-first optimisation machine and that AI may reorganise work and decision-making in a similar way.

Quick favor - need your take

Was there any standout article or topic from April I missed? Feel free to drop a comment or hit reply, even a quick line helps.

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how decision making can be improved.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

With that out of the way, let’s get to the today’s topic of decision making!

Last week I was reading the bitter lesson by Rich Sutton again. Great read, if you haven’t already. And it got me thinking: are there similar patterns in the world of data? Are there cases where we overvalue the human knowledge of the domain? And I think there is a close analogy in decision-making.

The biggest lesson that can be read from years of judgment and decision-making research is that simple, consistent rules beat human judgement most of the time.

This can be hard to accept because it feels like an attack on expertise but the evidence is blunt: human judgment is often noisy, inconsistent and overconfident. Paul Meehl made the argument in 1954 that statistical rules often outperform expert clinical judgment. Later reviews found the same pattern across many fields.

We like to believe that experience gives us a special ability to see the truth of a case. We like to believe that judgment lives in the rich details, the subtle signals, the human context. But often, when we test that belief against outcomes, the details we trusted were noise, the subtle signals were distractions and the human context gave us more confidence than accuracy.

Daniel Kahneman’s work helps explain why. Intuition is pattern recognition, not magic. It works when the environment is stable and feedback is fast and clear. Chess players can build good intuition because they see repeated patterns and receive immediate feedback. Many business, hiring, policy and strategy decisions are different. Feedback is delayed, messy and often ambiguous. A bad decision can look good because of luck. A good decision can look bad because the environment changed. In that kind of setting, intuition can feel like expertise while behaving more like confidence.

Hiring is a simple example. Managers often believe they can spot talent through conversation. They notice confidence, polish, energy and ‘fit’. But unstructured interviews are noisy. Two interviewers can see the same candidate and walk away with different conclusions. The same interviewer may judge differently depending on mood, fatigue or one memorable answer. A structured process feels less impressive: define criteria, ask similar questions, score each dimension and combine the scores. But that boring process is usually fairer and more predictive.

The same pattern appears in admissions, forecasting, insurance, performance reviews and risk assessment. Kahneman and Sunstein called this problem ‘noise’: unwanted variation in judgments that should be much closer together. In Noise, they describe an insurance company where underwriters independently priced the same fictitious cases and the median variation in premiums was 55%, far higher than executives expected.

Bias gets most of the attention because it has a story. Noise is harder to see because there is no obvious villain. No one needs to be irrational, corrupt or incompetent. People can be smart, experienced and honest and still produce wildly different judgments.

In AI, methods that scaled with computation beat methods that tried to encode human cleverness. In decision-making, methods that scale with evidence and consistency beat methods that try to preserve human intuition.

That does not mean algorithms should replace people. Humans still define the goal, decide what data is legitimate, handle ethical trade-offs and question whether history still applies. The lesson is narrower: stop using intuition for the parts of decision-making where tested, consistent rules do better. Use people to frame the problem but use algorithms for repeated decision making scenarios.

What are your thoughts?

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Airtable cut archive storage costs by 100x and saved millions.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get into how Airtable pulled it off.

TL;DR

Situation

Airtable’s MySQL storage had grown to petabytes, with archive tables driving major cost and scale issues. Some databases were also getting close to the 64TB RDS limit.

Task

The team needed to move cold archive data out of MySQL without breaking revision history or slowing queries. They also had to keep durability, availability and enterprise requirements intact.

Action

Airtable built a two-tier system: recent data stayed in MySQL, old data moved to S3 as Parquet files. They used DataFusion for querying, plus Flink, compaction, validation, caching, indexes and bloom filters.

Result

Parquet made the archive dataset about 10x smaller and S3 was about 10x cheaper than MySQL. That led to roughly 100x lower storage costs and millions in annual savings.

Use Cases

Archiving data, reducing storage cost, Improving query latency

Tech Stack/Framework

Apache DataFusion, AWS MySQL RDS, Apache Flink, AWS SQS

Explained further

Context

Airtable’s storage team went into 2024 with a pretty blunt problem: too much archive data sitting in the wrong place.

Their AWS MySQL RDS footprint had grown to petabytes, some of the biggest databases were getting dangerously close to the 64TB RDS disk limit and one particular class of data was doing most of the damage: cell history and action log tables. Together, these acted as Airtable’s archive layer, powering revision history and helping with internal debugging. For some enterprise customers, that data also had to be retained for up to 10 years.

The issue was not that the data had no value. It clearly did. The issue was that MySQL was an expensive home for a workload that was mostly cold.

Most of this archive data was old, rarely touched and read-only except for hard deletion cases. When it was queried, the access patterns were fairly predictable: point selects and paginated range queries, always scoped to a specific base. That made it a poor match for row-oriented OLTP storage at this scale, especially when Airtable still needed interactive latency and strong durability and availability guarantees.

So the team built a new two-tier storage system. Recent rows would stay in MySQL. Older archive data would move to S3, be stored as Parquet files partitioned by base and queried through a new engine built on Apache DataFusion.

That shift did more than trim costs around the edges. The final archived dataset became 10x smaller than the original data in MySQL thanks to Parquet compression and S3 itself was around 10x cheaper per byte than MySQL storage. Put those together and the result was a storage layer that was about 100x cheaper.

Why Airtable needed a better archive layer

Airtable’s archive data had a few characteristics that mattered a lot:

• the overwhelming majority (trillions of rows) of it was old and infrequently accessed

• most reads were point selects or range queries used for pagination

• queries were always filtered to a single base

• old data was effectively immutable

• the data was keyed by MySQL’s autoincr_id, so it naturally followed insertion order

That combination is useful. It tells you the team did not need a general-purpose database for this layer. They needed something cheaper that still handled a narrow set of read patterns well.

The first key idea was to move archive data from MySQL into S3. S3 was already much cheaper byte-for-byte. The second was to store that data in Parquet and partition it by base. The third was to place a query engine in front of it that could answer interactive requests without forcing full scans.

This became a two-tier system: hot and recent archive rows in MySQL, older rows in S3-backed Parquet. That let Airtable keep the user-facing experience intact while steadily pulling massive amounts of cold data out of an expensive OLTP system.

Architecture overview

At a high level, the architecture is simple enough to explain in one sentence: archive data moved from MySQL into S3 Parquet files, and DataFusion was used to query those files directly with low enough latency to support product features.

What made it work was the alignment between storage layout and query patterns.

The team kept the Parquet schema close to the original MySQL schema. They also preserved ordering by autoincr_id, because a large share of their reads depended on point lookups or ranged access on that field. That meant the query engine could use Parquet metadata to narrow down what bytes to fetch from S3 instead of pulling full files.

They also partitioned files by base, which mattered just as much. Since queries were always scoped to a specific base, partitioning by base meant Airtable could avoid touching unrelated data entirely.

On top of that, Airtable stored S3 file location metadata in DynamoDB. This gave the client layer a clean way to register the right files with the query engine and helped support enterprise requirements such as regional data residency and encryption with customer-provided keys.

How Parquet fit the workload

Parquet was not just a cheaper format choice. It was the thing that made interactive querying on S3 plausible.

Unlike MySQL’s row-oriented InnoDB layout, Parquet is columnar. It stores each column contiguously and groups rows into row groups, with each row group containing column chunks. More importantly, Parquet files include metadata that query engines can exploit for pruning. File metadata carries offsets and sizes, while page-level metadata can include statistics such as min/max values and bloom filters.

That matters because Airtable’s queries were usually not broad analytical scans. They were targeted reads. If a query asks for a narrow range of autoincr_id values within one base, and the files are sorted on that column, the engine can inspect metadata and skip most row groups without reading them.

Airtable leaned directly into that. They kept the original schema mostly intact and preserved sorting by autoincr_id. Because of that, the engine could selectively download the relevant byte ranges from S3 rather than treat each Parquet file like a blob.

There was also a second major upside: compression. Thanks to the columnar layout, the archived dataset ended up about 10x smaller than the original MySQL version. That is a huge result on its own. Pair that with S3’s lower storage cost and the economics really shifted in Airtable’s favor.

Picking the right query engine

Once the storage format was settled, Airtable benchmarked several engines capable of querying Parquet in S3:

• AWS Athena

• DuckDB

• StarRocks

• DataFusion

Athena was ruled out quickly for latency reasons. Its API pattern of starting a query and polling for completion made it better suited to general OLAP workloads than user-facing interactive queries. Airtable was seeing query latencies in the seconds, which was too slow for revision history use cases. It also lacked the strong isolation Airtable cared about across bases.

DuckDB was useful, but not ideal for this workload. They found that query planning did not always use projection pushdowns effectively, which sometimes led to full file downloads. Simple point queries on one autoincr_id could still be subsecond, but overall it trailed DataFusion. The team still used DuckDB heavily during development because it was convenient for debugging Parquet contents from the command line.

StarRocks produced performance results comparable to DataFusion, but it came with the operational burden of running a full-time cluster in Kubernetes to serve relatively low-QPS cold-storage queries. Like Athena, it also did not give Airtable the same kind of strong base-level isolation.

That left DataFusion.

For Airtable, DataFusion hit the sweet spot. It was strong at exploiting Parquet metadata, it was extensible and because it is an embedded Rust library, the team could run it inside their existing worker architecture.

That brought a few clear benefits.

Operationally, there was no extra service to deploy and babysit. Isolation came for free because each base already had its own process boundary. And request affinity stayed high because the same workers kept serving the same bases, which later made caching hit rates excellent.

In other words, DataFusion fit Airtable’s architecture.

Migrating data out of MySQL

Designing the new storage layer was one thing. Moving petabytes of live data into it without breaking anything was another.

Airtable wanted a one-time migration process that would start cutting MySQL storage costs quickly. To get a consistent export view, they used AWS RDS snapshot capabilities, which produced large Parquet files of full tables. They had also prototyped direct SQL extraction into Parquet, but chose not to productionize that approach at scale. Snapshots were preferred because they ran against backup instances and avoided extra pressure on production systems.

The challenge was that these snapshots were massive table-level exports across many shards, while Airtable’s serving model required files partitioned by base.

So the team added a repartitioning and compaction pipeline.

First, Flink jobs parallelized across shard snapshots and repartitioned records by base into intermediate S3 directories. Then AWS Step Functions scanned those intermediate outputs and enqueued bases into SQS. From there, custom compactor code merged the files, merge-sorted them, deduplicated records and produced final serving Parquet files capped at 1GB each.

That 1GB size was not arbitrary. It was chosen through benchmarking as a good serving size with a useful density of page groups per file. Again, a small design detail with big latency implications.

Validating the migration

A migration like this lives or dies on validation.

Airtable first ran bulk validation to confirm that data had not been corrupted during export, repartitioning and compaction. For this, they spun up a StarRocks cluster and compared the serving Parquet files against the original RDS snapshots, finding zero cases of data corruption.

We covered this validation approach in more detail in an earlier piece on how Airtable made archive validation work at petabyte scale.

The important point is that bulk validation gave the team confidence that the migration pipeline itself had preserved the data correctly.

That is a strong result, though it only answered one part of the problem. The bigger challenge was all the new storage client logic that now sat around the data:

• a Rust query engine built with DataFusion

• integration into Node.js using napi-rs

• logic to combine MySQL and S3 results

• logic to identify which S3 files to read

• support for enterprise features like customer keys, data residency and hard deletes

Bulk validation could confirm the data files were right. It could not prove that the full end-to-end user experience would behave exactly as before.

So Airtable moved to shadow validation on live traffic. Requests continued reading from MySQL as normal, while the same queries were also executed in the background through the new system. That let the team compare outputs under real conditions and catch implementation issues before rollout.

The bugs they found were exactly the sort of things that show up when systems cross language runtimes and execution models:

• float precision mismatches between JavaScript and Rust’s serde JSON handling

• a sorting issue where DataFusion used lexicographic rather than numeric ordering

• a crashing SIGABRT issue tied to async napi-rs and Node.js worker threads

• latency problems

They resolved these before launch and before deleting the MySQL copies of the migrated data.

Fixing latency bottlenecks

Once staged rollout began, latency became the main battleground.

That is not surprising. S3-backed query systems usually do not fail because storage is too expensive. They fail because network round trips and scan inefficiencies make them annoyingly slow.

Airtable saw a mix of problems: inefficient query plans, too many S3 requests and cases where sparse filters still caused too much data to be downloaded. They responded with a few targeted optimizations.

Building a tiered cache for archive queries

Caching turned out to be one of the biggest wins.

Under the hood, DataFusion translates SQL queries into S3 GET operations. It fetches Parquet footer metadata, column chunk metadata and then decides what row groups and byte ranges need to be read. If every step involves another network trip, latency stacks up quickly.

So Airtable built a tiered caching system.

The first layer used DataFusion’s built-in cache support to store Parquet file metadata and S3 ListObjects results.

The second layer cached additional Parquet page header metadata in memory. Combined with the first layer, this reduced how often the engine had to round-trip to S3 during query planning. Airtable wrote a custom implementation around DataFusion’s parquet reader interfaces, which let them cache metadata results directly and add instrumentation. The result was a reported 99%+ cache hit ratio.

That number is believable in context because DataFusion ran inside per-base workers and the files themselves were partitioned by base. The system had strong locality by design.

Finally, Airtable added an on-disk cache for full Parquet files. This was reserved for a very small number of heavy bases with bad enough query patterns to justify the extra work and cost. Unlike metadata caching, downloading whole files is not something you want to do casually. But for outlier cases, it gave the team another escape hatch.

Building custom indexes for sparse queries

Not every query could be handled efficiently by the base file layout alone.

Most reads were anchored on autoincr_id, but Airtable also had filters on other fields that could reduce result sets dramatically. Examples included filtering by action type, filtering by row or excluding sync-generated updates.

For some bases, those additional conditions matched only a tiny slice of rows. In those cases, even if autoincr_id helped somewhat, reading broad sections of Parquet files was still wasteful.

So Airtable built a secondary indexing system.

Using DataFusion, they scanned Parquet files and wrote index data out as new Parquet files. The client layer knew how to query those indexes first, then use the result to build a more targeted query against the original archive files.

This was much easier to do because the data was effectively read-only. Airtable did not need to solve the usual headache of synchronizing constantly changing base tables and secondary indexes. Static data makes a lot of index ideas suddenly practical.

Using Bloom Filters for faster lookups

There was one more edge case: lower-QPS point lookups on a different unique identifier that was randomly distributed.

That broke the usual min/max pruning strategy. Since the identifier values were not ordered, Parquet statistics were not helpful. Without another technique, the engine would need to fetch and scan every page group before applying the filter.

Airtable could have solved this with another custom index, but they chose a simpler route: Parquet bloom filters.

Bloom filters are probabilistic membership structures. They can tell you if a value is definitely not present, or maybe present. False positives are possible. False negatives are not.

That property is enough for pruning. If the bloom filter says a page group definitely does not contain the target identifier, the engine can skip it safely. DataFusion already understood Parquet bloom filter metadata, so Airtable could rely on native support instead of bolting on another indexing layer.

Conclusion

Airtable’s storage team took a dataset that had clearly outgrown MySQL’s economics and built a system that matched the workload far better.

They moved petabytes of archive data out of MySQL, kept recent data in the transactional store, archived old rows into base-partitioned Parquet files on S3 and queried those files with an embedded DataFusion engine. Along the way, they layered in DynamoDB metadata registration, a large-scale migration pipeline, bulk and shadow validation, multiple caching layers, custom secondary indexes and bloom filter-based pruning.

The result was a storage system that stayed durable and queryable at interactive latency while cutting storage costs by around 100x and saving millions of dollars per year.

There is still more to do. Airtable’s first implementation focused on bulk migration so they could start saving money quickly. The longer-term goal is incremental archiving, likely through a CDC-style system such as Flink. That opens up a new set of engineering problems around compaction, index rebuilds and operations. There are also other log-like tables that could be migrated onto the same platform.

Still, the core idea is already proven.

If a dataset is mostly cold, mostly read-only and queried through a narrow set of predictable access patterns, keeping it in an expensive OLTP database is often just inertia dressed up as architecture. Airtable looked at the shape of the workload, changed the storage model to match it and got the kind of result every infra team wants: better economics without making the product worse.

The full scoop

To learn more about this, check Airtable's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Pinterest used multimodal AI to organise products into relevant shopping collections

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to PinLanding: Pinterest’s AI system for shopping collections

Situation

Traditional shopping collections built from search logs and manual curation were not enough to cover the growing mix of detailed, compositional and AI-native shopping queries.

Task

Build a scalable system that could generate precise, searchable shopping collections directly from product content while still aligning with real user shopping intent.

Action

Pinterest built PinLanding, a multimodal pipeline that used search behavior to identify intent gaps, vision-language models to extract product attributes, clustering and LLM-as-judge to clean and validate a shopping vocabulary, a CLIP-style model to assign curated attributes at scale and Ray plus Spark to run large-scale inference and feed construction.

Result

The system generated 4.2 million shopping landing pages, expanded unique topics by 4x, improved average Precision@10 from 0.84 to 0.96 and delivered a 35% lift in search performance.

Use Cases

Search relevance improvement, attribute tagging, catalog enrichment, cold-start collection generation

Tech Stack/Framework

Apache Spark, Ray, PyArrow, CLIP, VLM

Explained further

Context

Online retailers and social platforms now manage catalogs with billions of items. Pinterest is one example, but the underlying problem is much broader: how do you organize massive product inventories into shopping collections that people can actually browse?

Pinterest has been dealing with exactly that. Historically, shopping collections were mostly shaped by user search history and manual curation. That worked well enough when search behavior was more predictable and the long tail was smaller. It works a lot less well when people start searching in full sentences, asking for vibes, aesthetics and combinations of constraints all at once.

That shift is what makes PinLanding interesting.

Instead of waiting for search logs and human curation to define the collection space, Pinterest’s team flips the process around. The system starts from the product content itself, then builds shopping collections from that foundation while still staying grounded in how people actually search. In other words, it is content-first, but not content-only.

The system is built around four components: understanding user search patterns, building and validating a shopping collection vocabulary using multimodal LLMs and LLM-as-judge, constructing feeds from attributes and evaluating the system while adapting to AI-native search behavior.

1. Mapping what shoppers actually want

The system starts with something fairly practical: understanding what users are actually trying to do.

Pinterest aggregates signals from search history, autocomplete interactions, filter usage and browse paths to estimate the distribution of shopping intents across the catalog. The goal here is not to build a giant list of every possible topic. It is to get a grounded picture of current demand, current coverage and where the system is falling short.

Two clear patterns show up.

The first is the head of the distribution: high-volume, well-formed shopping queries like ‘black cocktail dress’ or ‘white linen pants.’ These are the kinds of searches traditional collection systems already handle reasonably well. They are structured, frequent and easy to tie back to existing ranking and merchandising logic.

The second is where things get more interesting. There is a growing long tail of conversational and compositional queries, especially as users get more comfortable interacting with AI systems. These are requests that blend style, occasion, constraints and subjective framing like ‘what to wear for Italian summer vacation’ or ‘long red satin dress with lace trim under $200’. They are describing a scenario or aesthetic.

That distinction matters because it reveals the limitations of a purely query-driven collection pipeline. The head is manageable. The tail is messy, sparse and expanding.

Pinterest uses these behavioral signals in three ways. First, they highlight product spaces where user demand is strong but collection coverage is still thin. Second, they reveal which attribute dimensions users care about, including things like color, occasion, style, fit, price and brand, across 20 categories. Third, they create the baseline against which the rest of the pipeline is evaluated.

That last point is important. PinLanding is not trying to throw away query understanding. It is trying to improve topical coverage and precision relative to the old query-driven baseline. Search behavior still matters. The difference is that the system no longer depends on search logs alone to decide what collections can exist.

2. From raw product content to searchable shopping topics

Once Pinterest understands the behavior surface, the next step is to describe each product in a structured way that is useful for collection generation.

Each product is modeled as a multimodal tuple made up of an image plus metadata like title, description, merchant tags and price. A vision-language model (VLM) is then used to generate candidate attributes for that product. Rather than producing free-form descriptions, the model is prompted to return normalized key-value pairs. That design choice makes downstream processing far easier because it turns messy visual and textual information into something closer to structured data.

The initial output is broad and useful in one sense because it has high recall. It captures lots of possible attributes. The problem is that it is too messy to use directly.

The raw VLM output tends to overproduce highly specific descriptors like ‘black insoles’ or ‘lace-trim hem with side slit.’ It also produces near-duplicate variants such as ‘boho,’ ‘bohemian’ and ‘boho-chic’ for what is basically the same shopping concept. That creates a sparse attribute space where too many labels apply to too few products. Once that happens, collection quality starts to fall apart. You end up with lots of tiny fragments instead of reusable shopping topics.

Pinterest addresses this with a curation pipeline designed to build a compact and reusable attribute vocabulary.

The first step is frequency filtering. Attributes that appear only on a tiny number of products are rarely useful as collection keys, so they are removed. This strips out many ultra-specific descriptors while keeping the attributes that are more likely to reflect reusable shopping concepts.

The second step is embedding-based clustering. Dense text embeddings are generated for the remaining attributes and highly similar attributes are merged. When multiple variants express the same idea, the more frequent surface form becomes the canonical one.

The final step uses LLM-as-judge to score candidate topic-query pairs. Given an attribute tuple and its generated query, a second LLM evaluates whether the pair is semantically coherent, whether it sounds like a plausible shopping intent and whether it matches typical search phrasing. This helps rank and filter candidate topics not just for internal consistency but for searchability.

That last part is the key difference between extracting attributes and building shopping topics. Attributes by themselves are too granular and too raw. The curation pipeline turns them into a vocabulary that is compact enough to scale and natural enough to match how people actually search.

2.1 Scaling attribute assignment with a CLIP-style model

Once the curated attribute vocabulary exists, the next question is how to assign those attributes across the full product catalog.

Running the original vision-language model on every product at production scale would be expensive and brittle. So Pinterest trains a dual-encoder model inspired by CLIP instead.

One encoder takes product image and text as input and produces a product embedding. The other encoder takes an attribute phrase and produces an attribute embedding in the same vector space. During training, product-attribute pairs generated from the VLM are treated as positive examples, while non-matching pairs act as negatives.

The training objective is bidirectional contrastive loss. Matching product-attribute pairs are pulled closer together in embedding space and mismatched pairs are pushed apart.

At inference time, the system embeds all products and all attributes once. An attribute is assigned to a product when the similarity between the two embeddings crosses a calibrated threshold. That threshold can also be adjusted with frequency-based weighting to counter long-tail smoothing.

This architecture solves a few problems at once.

It is cheaper than running the VLM everywhere. It creates a denser and more consistent attribute graph. And it reduces the explosion of distinct attributes that came from raw model outputs.

Interestingly, they note that the resulting system ends up with fewer distinct attributes overall, while increasing the average number of attributes per product. That is a useful tradeoff. Fewer labels does not mean less information here. It means the information is compressed into a vocabulary that is actually reusable.

That dense attribute graph becomes the foundation for everything that follows.

3. Building shopping feeds at catalog scale

Once products have been tagged with curated attributes, Pinterest needs to construct feeds at catalog scale. That means dealing with millions of Pins and millions of possible topics.

That is where Ray and large-scale batch inference come in.

Attribute inference runs as a Ray streaming job with three main stages.

The first stage handles data loading and preprocessing. Product images and metadata are downloaded, tokenized and serialized into PyArrow tables, then sharded across a CPU cluster.

The second stage handles ML inference. Ray schedules preprocessed batches onto a GPU pool where the CLIP-based classifier performs forward passes and produces attribute scores.

Because execution is streamed, data loading, preprocessing and inference can overlap instead of waiting on one another. The use of heterogeneous clusters also allows CPU-heavy preprocessing and GPU-heavy inference to scale independently. The training and inference pipeline for the classifier completes in roughly 12 hours on 8 NVIDIA A100 GPUs at an estimated cost of about $500 per training run. For a system operating at this scale, that is pretty reasonable.

Once attribute assignments are available, feed construction moves to a matching layer built around ANN-style and strict attribute matching via Spark.

Each shopping topic is defined as an attribute tuple. For example: category equals dress, color equals yellow, season equals summer, occasion equals party.

Spark is then used to compute relevance scores between topics and products. The scoring aggregates shared attributes between topic and product, weighted by attribute-level confidence.

To keep this tractable, candidate joins are pruned through attribute-based partitioning and minimum-overlap prefilters.

This part is easy to underestimate. Generating a topic vocabulary is one thing. Turning that vocabulary into feeds across millions of products is where many systems fall apart. Pinterest’s design works because the modeling and infrastructure choices line up. The attribute space is structured enough to match efficiently and the batch inference stack is built to operate at the right scale.

4. Measuring whether the system actually works

Pinterest evaluates the system at both the attribute level and the collection level.

For attribute quality, the CLIP-based model is tested on Fashion200K, a standard benchmark for fashion attribute prediction. The reported result is 99.7 percent Recall@10, which substantially exceeds prior methods that sit in the 50 percent range on the same metric. That suggests the model has learned a strong mapping between product imagery and fashion attributes.

Pinterest also looks at the distribution of generated attributes. Compared with raw GPT-4-V outputs, the CLIP-based system produces a much more usable attribute space. There are fewer extremely rare attributes and more attributes that consistently apply across many products.

That is important because quality here is not only about benchmark performance. It is also about whether the resulting label space can support downstream shopping tasks without collapsing into noise.

For collection quality, human raters compare feeds produced by the content-first pipeline against a traditional search-log-derived baseline. The evaluation uses Precision@10, measured as the fraction of the top ten products in a collection that match the collection’s title attributes.

Across attribute families including color, main material, fit and stretch, shape, style, season and festival, occasion and brand, the new system improves average Precision@10 from 0.84 to 0.96. Several categories, including style and brand, reach 1.00.

That is a meaningful jump. It suggests the collections are not just broader in coverage, but also more precise in the items they surface.

In production, the pipeline generates 4.2 million shopping landing pages. That represents a fourfold increase in unique topics relative to the previous search-log-based approach and leads to a 35% improvement in search performance.

That combination is what makes PinLanding interesting because it is not only producing cleaner labels in an offline benchmark but also expanding coverage and improving relevance in production.

Summary

PinLanding is a strong example of what multimodal AI looks like when it is pushed into production with actual constraints.

The system does not rely on a single model to do everything. Instead, Pinterest breaks the problem into stages: measure intent, generate structured attributes, compress them into a usable vocabulary, scale assignment with a dual encoder and then construct collections with distributed matching.

That decomposition is probably the most useful lesson here.

There are plenty of teams experimenting with multimodal models for product understanding. The harder part is turning those outputs into something dense, normalized and operationally stable enough to support search and discovery at web scale. Pinterest’s team shows a fairly clear path for doing that.

The broader shift is also worth noting. Search-log-derived collections are not going away, but they are no longer enough on their own. As people search in more conversational, compositional and trend-driven ways, systems need to infer collections from the content itself while still staying anchored to user intent.

That is the balance PinLanding is aiming for.

And for platforms sitting on billions of products, that balance is probably becoming less of a nice-to-have and more of a survival requirement.

The full scoop

To learn more about this, check Pinterest's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

It’s time for another round-up on all things data and AI!

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Without further ado, let’s get to the round up for March!

Data science & AI

• State of Context Engineering in 2026 (12 minute read)
  argues that context engineering is evolving from prompt tinkering into a structured discipline where managing memory, retrieval and state becomes the core challenge of building reliable AI systems.

  

• Bayesian statistics for confused data scientists (15 minute read)
  Nicolas Chagnet explains Bayesian statistics in plain terms by showing that its real strength is not mathematical elegance but giving data scientists a cleaner way to reason about uncertainty and sparse real-world data.

• Agentic AI Engineering Guide (10 minute read)
  argues that most agentic AI systems fail not because the model is weak, but because teams make avoidable engineering mistakes around context, architecture, planning and evaluation.

  

• The Context Problem (29 minute read)
  argues that the AI industry has turned context into a token-priced billing unit even though context should really mean the relational structure that makes information coherent and useful.

• A Visual Guide to Attention Variants in Modern LLMs (26 minute read)
  shows how modern LLM attention has evolved from standard multi-head attention into a growing set of variants each designed to balance quality, memory use and inference efficiency.

  

• The Boon of Dimensionality (6 minute read)
  Amandeep Singh shows that high-dimensional space creates the geometric conditions that make embeddings, random projections and feature separation work in modern machine learning.

• GPT 5.4 is a big step for Codex (9 minute read)
  writes that GPT 5.4 feels like a real step forward for Codex, with gains in usability, speed, context handling and agent reliability that matter more in practice than benchmark scores alone.

• How Shopify Scales Taxonomy Evolution Across 10,000+ Categories With Multi-Agent AI (14 minute read)
  This piece breaks down how Shopify moved from reactive manual updates to a multi-agent system that scans taxonomy branches in parallel, proposes new categories/attributes from merchant data, detects duplicates and runs automated QA through domain-specific judges.

Data engineering

• Layer by Layer, We Built Data Systems No One Understands (9 minute read)
  writes that modern data stacks keep piling on layers in the name of simplicity but the result is often more sprawl, more cost and systems that are harder to understand or tie back to business outcomes.

  

• ETL is Dead (12 minute read)
  argues that ETL is not disappearing in volume but it is fading as the core identity of data engineering as AI shifts the real work toward context and semantics.

• The Data Engineer’s GitHub Portfolio (2026 Edition) (10 minute read)
  writes that a strong data engineering GitHub portfolio should prove technical taste, system thinking and real-world problem solving, not just show a pile of tutorial projects.

  

• The Data Engineering Mindset Every AI Builder Needs (14 minute read)
  writes that most AI products do not break because of the model but because builders ignore the data foundations early on, especially data flow design, data quality and monitoring.

• The absolute beginners guide to databasemaxxing (18 minute read)
  This article walks through database internals from a beginner’s perspective, showing how concepts like parsing, binding, scans and index seeks fit together under the hood.

• Your Data Model Isn’t Broken, Part I: Why Refactoring Beats Rebuilding (12 minute read)
  Chris Hillman makes the case that most broken data models are really bundles of hard-won business knowledge and that careful refactoring is usually smarter than blowing everything up and starting again.

• ClickHouse -> Real-time insight in 15 minutes (19 minute read)
  shows that ClickHouse is built for real-time analytics but getting that performance in production usually means handling enough operational complexity that makes platforms like Tinybird appealing.

  

• How Notion Scaled AI Q&A to Millions of Workspaces (14 minute read)
  This article walks through how Notion scaled AI Q&A to millions of workspaces while increasing onboarding throughput 600x and cutting costs by up to 90%.

Other interesting reads

• Relevance of Business Models for Data (10 minute read)
  makes the case that data teams should start with the business model first, because strategy, architecture, governance and analytics all work better when they are tied to how the company actually creates and captures value.

  

• It’s about the strategy, stupid (14 minute read)
  Timo Dechau makes the case that data work becomes far more useful when it starts with business strategy, not with dashboards, tracking audits or whatever tactic happens to be fashionable.

• What do frontier AI companies’ job postings reveal about their plans? (9 minute read)
  Interesting article suggesting that frontier labs’ job postings reveal where the market is heading, with hiring patterns pointing to heavier go-to-market pushes, new product bets and different strategies for securing compute and data.

Quick favor - need your take

Was there any standout article or topic from March I missed? Feel free to drop a comment or hit reply, even a quick line helps.

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Notion scaled its AI Q&A to millions of users.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Notion’s AI Q&A level up!

TL;DR

Situation

Notion launched AI Q&A on top of vector search and quickly faced massive demand across millions of workspaces. The initial system hit limits in capacity, onboarding speed and cost.

Task

Scale onboarding, keep indexes fresh and reduce rising infrastructure costs. At the same time, simplify a growingly complex architecture without hurting latency.

Action

They introduced dual ingestion paths, generation-based indexing, serverless architecture and migrated to turbopuffer. Then reduced recomputation with page state tracking and moved embeddings to Ray for unified compute.

Result

600x onboarding growth, 15x workspace growth and major cost reductions across layers. Latency improved and the system became simpler and more efficient.

Use Cases

Real-time search indexing, semantic search, document retrieval

Tech Stack/Framework

Apache Spark, AWS EMR, Apache Airflow, Apache Kafka, AWS S3, DynamoDB, Ray, turbopuffer

Explained further

Context

When Notion launched AI Q&A in November 2023, the core idea sounded simple enough: let people ask natural-language questions and retrieve relevant knowledge from across their workspace and connected tools. In practice, that meant building a vector search system that could ingest huge amounts of content, stay fresh as pages changed and do all of it at a cost that made sense at Notion scale.

That is the real story here. Not just “vector search powers AI” but what happens after launch, when adoption jumps faster than expected and the infrastructure underneath has to keep up. Over two years, the Notion team pushed that system through several big transitions: scaling onboarding, dealing with storage pressure, changing database architecture, reworking indexing logic and moving embeddings workloads onto Ray. The headline numbers are hard to ignore: 10x scale and roughly one-tenth the cost.

This is a good example of how modern AI infrastructure usually evolves. The first version gets the product live. The next few versions are about survival, then simplification, then cost, then latency, then getting rid of all the awkward bits that built up during the rush.

Vector search, explained through Notion’s lens

Traditional keyword search is literal. It works when users type the exact words that exist in the content. It starts falling apart when the wording changes but the meaning stays the same. Someone searching for “team meeting notes” may still want a page called “group standup summary,” but keyword search does not naturally understand that those are closely related.

Vector search solves that by representing text as embeddings. Instead of storing only words, it maps text into a high-dimensional space where semantically similar ideas sit closer together. That means retrieval is based on meaning, not exact phrasing.

For Notion AI, this matters a lot. The system needs to answer questions in natural language by finding useful content across a workspace and even across connected sources like Slack and Google Drive. That is exactly the sort of setup where semantic retrieval becomes more useful than plain lexical matching. A user is not thinking about the title of the page or the exact phrasing inside a paragraph. They are asking a question in their own words and expecting the system to bridge the gap.

That expectation becomes expensive very quickly.

Part 1: Scaling beyond what the original system expected

At launch, Notion’s ingestion and indexing pipeline had two paths.

The first was an offline path. Batch jobs running on Apache Spark would chunk existing documents, generate embeddings through an API and bulk-load those vectors into the vector database. This handled the heavy lifting for backfilling large amounts of existing content.

The second was an online path. Kafka consumers processed page edits in near real time so live workspaces stayed up to date with sub-minute latency.

It is a practical split. The offline side handles the backlog and large initial loads. The online side keeps things fresh once a workspace is active. Together, the two-path setup gave Notion a way to onboard workspaces at scale without sacrificing freshness for day-to-day edits.

The vector database itself ran on dedicated ‘pod’ clusters, where storage and compute were coupled. The Notion team designed sharding in a way that echoed their Postgres setup: workspace ID was the partitioning key, routing used range-based partitioning and a single config referenced all shards.

That all made sense on paper. Then the product launched and demand was overwhelming.

Notion quickly built up a waitlist of millions of workspaces that wanted access to Q&A. The problem was no longer whether the system worked. It was how fast it could onboard people without cracking under the pressure.

When the indexes started to fill up

Only a month after launch, the original indexes were already nearing capacity.

That is the kind of problem that sounds good in product meetings and bad in infrastructure meetings. If the indexes filled up, Notion would have to pause onboarding. That would slow down rollout and delay access for everyone waiting.

The team had two obvious options.

One was to re-shard incrementally. Clone data into another index, delete half, repeat and keep doing that every couple of weeks as new customers came in.

The other was to re-shard for the final expected volume. But their vector database provider charged for uptime, so over-provisioning would have been painfully expensive.

Instead, the Notion team went with a third approach. When a set of indexes got close to full, they provisioned a new set and directed all newly onboarded workspaces there. Each set was assigned a generation ID, which determined where reads and writes should go.

It is not the prettiest long-term design, but it was a smart short-term move. It avoided repeated re-shard operations and kept onboarding moving. Sometimes the right scaling decision is not the most elegant one. It is the one that buys breathing room without stopping the business.

Turning onboarding into a throughput problem

Even with the architecture in place, the initial onboarding rate was nowhere near enough. At launch, Notion could onboard only a few hundred workspaces per day. At that pace, clearing a multi-million waitlist would have taken decades which is obviously not a real option.

So the team pushed hard on throughput. Using Airflow scheduling, pipelining and Spark job tuning, they dramatically increased capacity.

The results were big:

• Daily onboarding capacity increased by 600x

• Active workspaces grew 15x

• Vector database capacity expanded 8x

By April 2024, the Q&A waitlist was cleared.

That is the kind of milestone that looks clean in hindsight but it came with a cost. Managing multiple generations of databases helped during the hypergrowth phase but it also added operational complexity and financial overhead. The team had solved the immediate scaling problem, but the architecture was starting to feel heavy.

That set up the next phase of the story.

Part 2: Cost becomes the next constraint

In May 2024, Notion migrated its embeddings workload from the original dedicated ‘pod’ architecture to a serverless setup that decoupled storage from compute and charged based on usage instead of uptime.

The effect was immediate. Costs dropped by 50 percent from peak usage, translating into several millions of dollars in annual savings.

That alone would have made the migration worthwhile, but the serverless design also fixed two practical problems. First, it removed the storage capacity constraints that had become a serious scaling bottleneck. Second, it simplified operations because the team no longer had to provision capacity ahead of demand.

Still, even after cutting costs in half, the annual run rate for vector database spend was still in the millions. From an engineering point of view, this is where things get interesting. The easy win had already happened. Now the team had to go after deeper structural gains.

A new search foundation (turbopuffer)

While working on the first round of savings, Notion also evaluated alternative search engines. turbopuffer stood out because it offered significantly lower projected costs.

At the time, turbopuffer was a newer player in search. Its architecture was built on object storage with a focus on cost-efficiency and performance. It also supported both managed and bring-your-own-cloud deployment models and it made bulk modification of stored vector objects easier.

That combination lined up well with what Notion needed.

After a successful evaluation, the team decided to migrate its entire multi-billion-object workload to turbopuffer in late 2024. Since they were already making a provider switch, they used the migration as a chance to clean up the broader architecture too.

Several changes happened together.

First, they fully re-indexed the corpus, increasing write throughput in the offline indexing pipeline to rebuild everything in turbopuffer.

Second, they upgraded the embeddings model during the migration to be more performant.

Third, they simplified the architecture. turbopuffer treats each namespace as an independent index which removed the need to think about sharding and generation-based routing in the same way as before.

Finally, they handled the cutover gradually, migrating one generation at a time and validating correctness before moving on.

This is a strong pattern: if a migration is painful anyway, use it to pay off other infrastructure debt at the same time.

The outcome was solid on several fronts:

• 60 percent cost reduction on search engine spend

• 35 percent reduction in AWS EMR compute costs

• p50 production query latency improved from 70–100ms to 50–70ms

That is a meaningful improvement across cost and performance, which is not always easy to pull off together.

Avoiding full reprocessing with page state tracking

The next optimization went after a very expensive inefficiency in the indexing pipeline.

Notion pages can be long, so the team chunks each page into spans, embeds each span and stores those vectors with metadata such as authors and permissions. In the original implementation, any edit to a page or its properties triggered a full re-chunk, full re-embed and full re-upload of all spans on that page.

That meant even a tiny change could trigger a lot of unnecessary work.

The team narrowed the problem down to two things that actually mattered:

1. The page text changes which means embeddings need updating

2. The metadata changes which means metadata needs updating

To detect those cases, they tracked two hashes per span: one hash for the span text and another for the metadata fields. They chose 64-bit xxHash because it offered a good balance of speed, simplicity, low collision risk and storage footprint.

For caching, they used DynamoDB. Each page had one record containing the state of all spans on that page, including text and metadata hashes.

The win came from using that state to avoid unnecessary work.

Case 1: The page text changes

Imagine Herman Melville editing Moby Dick halfway through a page. Before this improvement, the whole page would have been re-embedded and reloaded. After the change, the system chunks the page, fetches the previous state from DynamoDB and compares text hashes span by span. It can then detect which spans actually changed and only re-embed and reload those.

That is the kind of fix that getting the balance right matters. Miss a changed span and search quality suffers. Reprocess too much and cost stays high.

Case 2: The metadata changes

Now imagine Melville updates permissions so the page becomes visible to everyone. The permissions metadata changes but the text does not.

Previously, that still meant re-embedding and reloading the entire page. With the new approach, Notion compares both text and metadata hashes. If the text hashes are unchanged but metadata hashes differ, the system skips embedding entirely and issues a PATCH command to the vector database to update only the metadata. That is much cheaper than recomputing embeddings.

Across these changes, the Page State Project reduced data volume by 70 percent. That saved money on both embeddings API costs and vector database write costs.

Moving embeddings to Ray (indexing)

In July 2025, Notion started migrating its near real-time embeddings pipeline to Ray on Anyscale.

The motivation came from several pain points in the earlier setup.

One was the ‘double compute’ problem. Spark on EMR handled preprocessing like chunking, transformations and API orchestration, but embeddings themselves were still generated through an external provider that charged per token. So the team was paying for both preprocessing infrastructure and embedding API usage.

Another issue was endpoint reliability. Fresh search indexes depended on the stability of an external embeddings API.

The third problem was clunky pipelining. To smooth traffic and avoid API rate limits, the team had built a multi-step handoff process where Spark jobs passed batches through S3. It worked but it was clunky.

Ray and Anyscale gave Notion a cleaner path.

Ray let the team run open-source embedding models directly, which meant more model flexibility and less dependence on external providers. By consolidating preprocessing and inference onto a single compute layer, they could cut out the double-compute setup. Ray also supports pipelining CPU-bound work such as chunking and page-state detection with GPU-bound embedding generation on the same nodes, which helps keep utilization high.

There was also a developer productivity angle. Anyscale workspaces let engineers write and test pipelines from their preferred tools without having to provision infrastructure manually.

And on the product side, self-hosting embeddings removed a third-party API hop from the user-facing path, which helped reduce end-to-end latency.

The rollout is still ongoing, but early results suggest a 90+ percent reduction in embeddings infrastructure costs. That is a major shift in how the economics of the system work.

Real-time query embeddings on Ray (serving)

Indexing is only half the picture. When users or agents search in Notion, queries must also be embedded on the fly before the vector database can be searched.

That makes serving latency-sensitive. The embedding has to happen fast enough that the search still feels responsive.

Hosting large embedding models is not trivial. GPU allocation, ingress routing, replication and autoscaling all matter, especially when traffic is uneven and expectations for responsiveness are high.

Ray Serve helped Notion here by handling much of that operational layer out of the box. The team could wrap open-source embedding models in persistent deployments that stay loaded on GPU, configure request batching and replication and manage the serving setup with normal Python code plus YAML-based infrastructure configuration.

That is a pretty practical endpoint for the broader journey.

What started as a vector search stack built quickly enough to launch AI Q&A turned into a much more refined system: simpler in some places, more selective in others, cheaper across multiple layers and faster where users feel it. The interesting part is not any single tool choice. It is how the Notion team kept removing bottlenecks one by one: storage limits, awkward shard routing, redundant recomputation, external API dependence and fragmented compute layers.

That is usually what mature AI infrastructure looks like in the real world. Not one giant redesign. A sequence of sharp decisions, each fixing the thing that has become too expensive, too slow or too annoying to keep around.

The full scoop

To learn more about this, check Notion's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

It’s time for another round-up on all things data and AI!

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Without further ado, let’s get to the round up for February!

Data science & AI

• Inside OpenAI’s in-house data agent (14 minute read)
  OpenAI explains how its in-house data agent combines rich internal context, live querying and self-learning memory to help employees go from vague business questions to trustworthy analysis in minutes.

• A Guide to Which AI to Use in the Agentic Era (17 minute read)
  breaks down the current AI tool landscape into a simple question: which model is best for this specific task, not which one wins the internet on any given day.

• Judgment isn’t uniquely human (19 minute read)
  argues judgment and taste are not uniquely human, and that treating them as off-limits to AI is another case of people underestimating how quickly models can learn high-level cognitive tasks.

  

• I spent $10,000 to automate my research at OpenAI with Codex (6 minute read)
  A researcher from OpenAI argues that people still underestimate what Codex can do in real workflows, sharing a high-usage setup and the practical lessons from using it at serious scale.

• Semantic Linking: the Aboutness of Data (12 minute read)
  expalins that semantic linking is the missing connection between data and meaning, where the real job is not adding labels to tables but explicitly mapping data objects to shared business concepts.

  

• AI Is Finally Eating Software’s Total Market: Here’s What’s Next (11 minute read)
  argues that AI won’t just disrupt software products, it will collapse whole layers of software value unless companies control the workflow, the customer relationship or the data moat.

• World Models and the Data Problem in Robotics (13 minute read)
  Nvidia researcher makes the case that robotics hits a data wall long before an algorithm wall and that world models learned from human first-person video are the most plausible route to scalable robot intelligence.

• Lessons learned from scaling data scientists with AI (10 minute read)
  Whatnot’s lesson from deploying AI for data science is that LLMs don’t remove the need for data scientists, they force teams to get serious about semantic layers and production-grade context management.

  

• How Shopify Scales Taxonomy Evolution Across 10,000+ Categories With Multi-Agent AI (14 minute read)
  This piece breaks down how Shopify moved from reactive manual updates to a multi-agent system that scans taxonomy branches in parallel, proposes new categories/attributes from merchant data, detects duplicates and runs automated QA through domain-specific judges.

Data engineering

• A Portable Analytics Stack (13 minute read)
  shows how a portable analytics stack built on DuckDB, DuckLake, dlt and SQLMesh can replace warehouse-heavy setups with lightweight, version-controlled pipelines that run locally or on cheap scheduled compute.

  

• Healing Tables: When Day-by-Day Backfills Become a Slow-Motion Disaster (24 minute read)
  Chris Hillman shows why incremental historical backfills corrupt dimensions and proposes a healing table pattern that separates change detection from period building so history can be rebuilt cleanly.

• 2028 - THE GREAT DATA RECKONING (16 minute read)
  A speculative but funny take by where he imagines a 2028 data industry shakeout where AI wipes out much of the tooling and data theater while the people who survive are the ones with real business context and architecture skills.

• Data Engineers Are Becoming MetadataOps Engineers (10 minute read)
  An interesting take by that the next layer of data engineering is MetadataOps: building AI-ready metadata, semantic structure and agent-facing governance so LLMs stop guessing and start using data reliably.

  

• How Data Engineers Can Leverage AI Tools Without Losing Fundamentals (13 minute read)
  makes the case that data engineers should use AI to accelerate the boilerplate, not outsource the fundamentals because the real leverage still comes from owning modeling, architecture and performance

• Backfills - The Necessary Evil of Data Engineering (12 minute read)
  A practical look by at why backfills happen, why engineers hate them, and how better parameterization, rerunnability, and storage-aware design can make them less painful.

  

• Why Your 5-Second BigQuery Query Isn’t Cheap (13 minute read)
  A practical breakdown of BigQuery pricing by that shows why short query runtimes are a misleading proxy for cost, and why slots are the compute metric that actually matters.

• How LinkedIn Built a Pipeline That Scales to 230M Records/sec Without Breaking SLAs (12 minute read)
  LinkedIn pushed Venice to handle 175M+ lookups per second while ingesting 230M writes per second. This piece breaks down how they balanced compaction, CPU bottlenecks and adaptive throttling to scale ingestion under eventual consistency.

Data analysis and visualisation

• The State of Machine Learning Competitions (34 minute read)
  This report maps the 2025 competition landscape and finds that winning solutions are getting more compute-hungry, transformer-heavy and increasingly shaped by Qwen in NLP while classic tabular methods still hold their ground.

  

Other interesting reads

• AI Doesn’t Reduce Work - It Intensifies It (9 minute read)
  Interesting findings in HBR that AI doesn’t really remove work so much as intensify it, speeding up expectations and raising the risk of burnout instead of delivering the productivity gains companies hoped for.

• The Anthropic Hive Mind (21 minute read)
  Steve Yegge’s take is that Anthropic’s real edge is not just Claude but a hive-mind way of working where humans and AI operate in a shared, high-speed loop that most companies aren’t built for yet.

• The least understood driver of AI progress (36 minute read)
  Anson Ho highlights that software progress, not just bigger chips or more spending, is a major and underappreciated reason AI keeps getting better faster than many people expect.

Quick favor - need your take

Was there any standout article or topic from February I missed? Feel free to drop a comment or hit reply, even a quick line helps.

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Shopify scales its product categorisation using agentic AI

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Shopify’s multi-agent taxonomy

TL;DR

Situation

Shopify’s product classification system makes tens of millions of predictions daily, across a taxonomy with 10,000+ categories and 2,000+ attributes. Commerce changes fast, the taxonomy has to keep up or the whole stack starts drifting.

Task

Keep the taxonomy current at scale without relying on slow, reactive, manual curation. Fix volume, expertise and consistency problems before they hit merchants, customers and model quality.

Action

Built an AI multi-agent system: structural analysis + product-driven analysis, then intelligent synthesis. Added equivalence detection (category = broader category + attribute filters) plus automated QA via domain-specific AI judges.

Result

Taxonomy branches can be analyzed in parallel: hundreds of categories instead of a few per day. Quality improved via grounded merchant data + structural consistency, with judges filtering proposals (example: “MagSafe compatible” approved at 93% confidence).

Use Cases

Category discovery, attribute gap detection, taxonomy maintenance, search and filtering improvement

Tech Stack/Framework

AI agent, equivalence detection, multi-agent system

Explained further

Context

Last year, over 875 million people bought items from Shopify merchants. Shopify already runs a product classification system that makes tens of millions of predictions daily with a high degree of accuracy.

But classification is the easy part compared to the thing underneath it: taxonomy. Because the model doesn’t just need to be right, it also needs a clean, consistent set of labels to be right about.

That’s the challenge for Shopify: once you have 10,000+ categories and 2,000+ attributes, the taxonomy becomes its own product with its own failure modes. It can get stale. It can get inconsistent. It can drift away from how merchants actually describe products. And when that happens, the classifier quality takes the blame for what is basically a taxonomy debt problem.

So this post is about what Shopify did next: they built an AI multi-agent system that doesn’t just classify products, it actively improves the taxonomy labels themselves so the system stays agile as commerce changes.

The challenge: scaling taxonomy without losing accuracy

A taxonomy is a contract between three groups that rarely agree:

• Merchants describing products the way they think about them

• Customers searching and filtering with their own mental model

• Platform systems trying to enforce structure so everything stays queryable and comparable

Now add the reality that commerce never sits still. New products appear. Old categories split. Entire verticals get reshaped by trends, tech and regulation. The taxonomy has to keep up or the platform drifts away from how people actually shop and sell.

Shopify frames the challenge as three problems.

The volume problem: manual updates can’t keep up

A global product taxonomy needs constant attention. Every new product type, emerging technology category and seasonal trend potentially triggers taxonomy updates.

Manual curation becomes a bottleneck because taxonomy work is not one change. It is usually a bundle: a category addition, a hierarchy decision, a set of attributes, naming alignment and a check for duplicates or conflicts.

For example, consider the emergence of categories like smart home devices or remote work equipment. Each category represents not just new categories but also entirely new attribute sets.

Smart home devices for instance need connectivity types, power requirements and compatibility. Those are specs that did not exist in the taxonomy before.

So the work isn’t a one-off. It’s continuous expansion and adjustment across a giant tree of concepts.

The expertise problem: every vertical has its own rules

Good taxonomy design is domain-heavy. You do not get it right by being generally smart. You get it right by knowing what matters in that product world. For example, there are nuanced differences between types of guitar pickups or appropriate attributes for skincare products.

A taxonomy team can’t realistically maintain deep expertise across every vertical that merchants sell into. But if the taxonomy is inconsistent or poorly structured, merchants pay for it through reduced discoverability, suboptimal search results and ineffective filters for customers.

The consistency problem: one concept, five different labels

As the taxonomy grows organically, inconsistencies creep in:

• similar concepts represented differently across categories

• naming conventions inconsistent

• discrepancies between merchant categorization and customer expectations

Those inconsistencies compound. Merchants get confused when listing. Customers get frustrated when filtering and comparing. And the classifier quality drops because labels stop being reliably meaningful across the tree.

This is the part most teams underestimate. In a taxonomy, small inconsistencies behave like small data quality issues: they don’t stay small.

From manual taxonomy work to agent-led evolution

Shopify’s taxonomy management evolved from a manual workflow into an AI-driven system.

The old way: Expert review, slow throughput

The traditional pattern is familiar:

1. domain experts analyze product data

2. identify gaps or inconsistencies

3. propose changes

4. implement changes via careful review

It ensures quality but it also creates bottlenecks.

The biggest problem was the reactive nature of it: Shopify would only recognize the need for new categories or attributes after merchants began listing products that didn’t fit. By then, the system had already missed chances to give merchants and customers a better experience.

So even when you do great manual work, you’re always late.

The breakthrough: Two lenses, one system

Advanced language models opened a door: not to replace human experts, but to augment them with scale and consistency.

The key insight was that taxonomy improvement comes from two different angles:

• structural analysis: the logical structure of the taxonomy, gaps in hierarchies, missing relationships

• product-driven analysis: what real product data says merchants actually sell and how they describe it

Each angle catches different issues. Shopify’s breakthrough was combining them into a system that can continuously propose improvements then filter them through quality checks before human review.

Inside the system: How the agents work

The new architecture rests on three principles:

• specialized analysis

• intelligent coordination

• quality assurance

And the intent is clear: continuous evolution, not one-time taxonomy construction.

What’s different: continuous evolution, not one-time creation

AI’s been used for product categorisation and one-off taxonomy builds for a while. The difference here is instead of building it once and hoping it holds, Shopify uses specialised AI agents to keep the taxonomy evolving continuously. There are 3 core components to this approach:

1- Real product grounding: The system integrates actual merchant product data so proposals reflect how merchants describe and categorize products. This keeps decisions grounded in commerce reality rather than only theory.

In other words: if merchants are consistently describing a differentiator, it probably belongs in the taxonomy, even if it offends someone’s idea of a “pure” category tree.

2- Multi-agent specialization: Multiple specialized agents run different analyses. One focuses on structural consistency. Another focuses on product-driven insights. Then those outputs are synthesized. The claim here is that the combination finds improvements that neither agent would find alone.

That makes sense structurally. Taxonomy is both a graph problem and a language problem.

3- Sophisticated equivalence discovery: This is the most interesting component. detecting equivalence relationships where a specific category equals a broader category filtered by attribute values.

This matters because merchants should be able to organize their catalogs however they want, while the platform still understands what products ‘mean’ underneath the merchant’s choices.

So instead of forcing everyone into one rigid structure, Shopify tries to learn mappings that preserve flexibility and still support search, recommendations, and analytics.

Architecture flow

The AI agent workflow works like this:

• enable agents to explore the taxonomy

• run multi-stage analysis (structural + product-driven)

• synthesize and resolve conflicts

• detect equivalences

• run automated QA using judges

• send refined proposals to humans

• update the taxonomy in production

Enabling agent-taxonomy interaction

Before agents can improve anything, they need to ‘read’ the taxonomy like a human would.

Shopify implemented a system that allows agents to:

• search for related categories

• examine hierarchical relationships

• verify whether proposed changes conflict with existing elements

A good example: an agent analyzing guitar-related categories can explore the full musical instruments hierarchy, inspect related attributes across instruments and look for patterns that suggest better structure.

In other words, the agent doesn’t just look at one node. It roams the neighborhood.

The pipeline: specialised agents, staged decisions

For the AI Agent to be work properly, different specialised agents are at work to provide specific insights:

Structural analysis: This agent looks at the taxonomy itself for logical consistency, completeness, gaps in category hierarchies, naming convention inconsistencies and opportunities to reorganize related concepts.

It operates purely on the taxonomy structure and aims to keep the whole thing coherent.

Product-driven analysis: This agent integrates real merchant data and examines how products are described and categorized on the platform.

Specifically, it looks at patterns in product titles, product descriptions and merchant-defined categories. The goal is to find gaps between how merchants think about products and how the taxonomy represents them.

This is an important distinction. A taxonomy can be structurally perfect and still be useless if it doesn’t match merchant reality.

Intelligent synthesis: Now we have two streams of recommendations:

• structure-driven improvements

• product-driven improvements

They can conflict. They can overlap. They can propose redundant changes.

The synthesis step merges insights, resolves conflicts, and eliminates redundancies. And sometimes the best answer is not pick one, it’s combine both.

Equivalence detection: This agent solves a practical commerce problem: merchants want flexibility but platform systems need consistency.

Consider golf shoes:

• Merchant A uses a specific ‘Golf Shoes’ category

• Merchant B uses ‘Athletic Shoes’ with an ‘Activity Type = Golf attribute

Both are valid for the merchant. But search, recommendations and analytics benefit from understanding these represent the same product set.

So the system detects attribute-based equivalences of the form:

specific category = broader category + one or more attribute filters

This lets merchants organize however makes sense for their business while keeping platform intelligence consistent across different catalog structures.

If you’ve ever tried to do cross-merchant analytics at scale, you can probably feel why Shopify cared enough to build an entire agent for this.

Automated QA: judges before humans

After proposals are generated, Shopify adds automated QA through specialized AI judges.

These judges evaluate proposed changes using reasoning capabilities and taxonomy design principles to filter and refine suggestions before human review.

The important detail is that evaluation differs by change type:

• adding new attributes

• creating category hierarchies

• modifying existing structures

Different changes require different criteria, so one generic ‘judge prompt’ would be weak. So instead, they use domain-specific judges.

An electronics-focused judge applies electronics expertise. A musical instruments judge applies that domain’s patterns and rules. The goal is consistent domain-aware evaluation across verticals.

Results

The system can analyze taxonomy branches in parallel, identifying improvement opportunities that used to take weeks of manual work.

Where experts might analyze a few categories per day, the system can evaluate hundreds of categories, checking both:

• structural consistency

• alignment with real product data

This matters most for emerging product categories. When new product types become popular on the platform, the system can quickly identify taxonomy gaps and propose comprehensive solutions, instead of reactive patches that build up debt.

Quality improvements

The multi-agent design improves consistency and comprehensiveness because it combines two lenses:

• structural analysis keeps hierarchy organization logical and consistent

• product-driven analysis keeps categories and attributes aligned with merchant reality

The automated QA layer reduces iteration cycles by catching issues before human review and applying domain expertise consistently.

Example: mobile phone accessories and MagSafe compatibility

Product analysis identified that merchants frequently advertise “MagSafe support” for accessories such as chargers, cases and wallets.

So the agent proposed adding a boolean attribute: ‘MagSafe compatible.’

A specialized electronics judge evaluated the proposal and checked:

• no duplicate attribute already exists

• boolean type is appropriate

• while brand-specific, MagSafe is treated as a legitimate technical standard similar to Bluetooth or Qi

The judge approved the attribute with 93% confidence, noting it would improve customer filtering for MagSafe-ready products.

This example matters because it demonstrates the full loop:

• merchant reality creates a signal

• the agent proposes a structured change

• a domain judge validates it with rule checks and domain framing

• humans get a higher quality proposal to review

Scaling development: from reactive fixes to proactive evolution

The biggest shift is strategic: taxonomy development becomes proactive, not reactive.

Instead of waiting for a merchant pain point or a platform limitation to trigger a change, the system can identify and address gaps earlier.

The system can also reason over the entire taxonomy structure, which supports cross-category consistency. That helps avoid the fragmentation you get when teams fix issues in isolation.

To validate the approach, they applied it to a specific area: Electronics > Communications > Telephony (called “Telephony AI” in their analysis) and compared it against their previous manual expansion method.

As you can see from the chart, the AI-assisted method can compress years of work into weeks for the taxonomy area if the agents are applied across all verticals.

The full scoop

To learn more about this, check Shopify's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how LinkedIn ingests data at scale.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Venice: LinkedIn’s data storage platform

TL;DR

Situation

Venice powers LinkedIn’s AI-driven products and has scaled to 2,600+ stores with workloads spanning bulk loads, streaming updates and active/active replication. The ingestion pipeline had to handle throughput-heavy, CPU-heavy and latency-sensitive traffic under eventual consistency.

Task

Redesign ingestion to scale to 230M writes/sec while preserving ordering and protecting read and write SLAs. Support hybrid stores, partial updates and multi–data center replication without destabilizing clusters.

Action

Scaled bulk ingestion with partition tuning, shared consumer/writer pools and direct SST writes; tuned RocksDB via compaction triggers and BlobDB to manage amplification. Optimized CPU-heavy paths using Fast-Avro and parallel processing, then enforced priority pools and adaptive throttling to protect current-version latency.

Result

Venice now handles 175M+ key lookups/sec and 230M+ writes/sec in production. It maintains a write latency SLA under 10 minutes while safeguarding read latency as the top priority.

Use Cases

Large-scale feature stores, real-time recommendation systems, hybrid data serving, low-latency notification

Tech Stack/Framework

Apache Spark, Apache Samza, Apache Kafka, RocksDB, Fast-Avro, Adaptive Throttling

Explained further

Background

Venice is an open-source derived data storage platform and LinkedIn’s default storage layer for online AI use cases. It sits behind products like People You May Know, feed, videos, ads, notifications, the A/B testing platform, LinkedIn Learning and more.

Since Venice launched internally in 2016 it has scaled from a handful of stores to over 2,600 production stores. The workloads also evolved a lot. It started with “just bulk load a dataset” and grew into a mix of:

• Bulk loading huge offline datasets

• Nearline streaming updates

• Active/active replication across data centers

• Partial updates that merge fields and collections

• Deterministic write latency expectations under eventual consistency

This post walks through how the ingestion pipeline was revamped to hit 230 million records per second in production, what changed across the architecture, which optimizations moved the needle and how different workload types get tuned. A lot of these ideas are portable if you run any distributed ingestion system where ordering, throughput and predictable latency all matter at once.

Venice overall ingestion pipeline

At a high level, store owners write to Venice through three paths:

1. Bulk loads from an offline processing platform (example: Spark)

2. Nearline writes from a streaming processing platform (example: Samza)

3. Direct writes from online applications

No matter which path you take, the writes all pass through an intermediate PubSub broker layer. From there, the Venice Storage Node (VSN) consumes messages and persists data locally using RocksDB (an embedded key-value store).

The pipeline sounds straightforward until you operate it at scale. The same ingestion path has to support very different workloads. Some are throughput-driven (bootstrapping a massive store). Some are latency-driven (current-version updates). Some are CPU-heavy (partial updates and conflict resolution). Some are I/O-heavy (compaction, SST churn).

The following sections will look at the challenges and how the LinkedIn team resolved them.

Use case 1: bootstrapping from offline dataset

Venice users can run bulk load jobs using offline processing platforms such as Spark to push new data versions to Venice stores. The hard part is performance for large or massive stores. If you want to find bottlenecks you need to understand the ingestion path end to end.

What happens during a bulk load

• A Venice Push Job (VPJ) creates a new version topic for the new store version, split into multiple partitions

• The Spark job uses a map-reduce framework to produce messages to that version topic

• It keeps one reducer per topic partition so message ordering is preserved

• On the other side, the VSN spins up consumers, reads messages and persists them into RocksDB

• There is one RocksDB instance per topic partition

So you can hit bottlenecks in three obvious places:

1. producing

2. consuming

3. persisting

Production experience says you will hit all three, just not on the same day.

Improving producing and consuming throughput

The usual first lever is increasing the number of partitions for large stores so you can use more of the PubSub cluster capacity. More partitions tends to mean more parallelism and more throughput.

But it comes with trade-offs:

• more partitions means more management overhead across Venice and PubSub

• there is a throughput ceiling per PubSub broker

So partition count is not a free lunch. It’s a knob that buys you throughput and charges you complexity.

Enhancing consumption scalability

To keep up with production, VSN uses shared consumer pools across all hosted stores.

Instead of “one store version, one set of consumers,” each store version can use multiple consumers by distributing hosted partitions among them. The point is to keep multiple connections per PubSub broker to speed up consumption (similar to a Download Manager).

The pool approach also does something boring but important: it sets an upper limit on total consumers which puts a ceiling on cost.

Optimizing I/O performance

VSN uses a shared writer pool to persist changes concurrently across multiple RocksDB instances and use local SSD capacity effectively.

Ordering is critical in Venice so for any given RocksDB instance there is only one writer actively writing to it. You still get concurrency across instances, not inside one instance which is the compromise that keeps ordering intact.

Minimizing memory overhead

Because messages for a partition are strictly ordered (thanks to the map-reduce framework), Venice uses RocksDB’s SSTFileWriter to generate SST files directly. That significantly reduces memory overhead during ingestion.

Ingestion workflow in Venice Server

Put together, the optimized workflow is basically: use the PubSub layer for distribution, use consumer pools for scalable reads, use writer pools for SSD throughput, preserve ordering by design and avoid memory blowups by writing SST files directly.

Use case 2: hybrid store

Venice supports Lambda architecture style use cases by merging updates from both bulk loads and nearline writes. Users query a single store and get a unified view.

Venice hybrid store workflow

How it works:

• each bulk load creates a new store version

• that version has a new Kafka topic and a new database instance

• real-time updates produced by a Samza job via a real-time topic are appended to both version topics to keep them current

• once the new version catches up fully, it is swapped in as the active version to serve reads

The hybrid store is important because it gives you a clean “new version build” story without losing real-time freshness. But it creates a new challenge: the database transitions from read-only to read-write.

That’s where RocksDB tuning matters, because duplicates start showing up more often. Keys get updated or deleted after they were inserted. RocksDB uses log compaction to remove stale entries, but that compaction has overhead: scan, merge, rewrite SST files, consume CPU, I/O and disk.

So the core problem becomes: tune RocksDB so you can balance three competing types of pain.

• Write amplification: bytes written to storage vs bytes written to the DB

• Read amplification: number of disk reads per query

• Space amplification: size of DB files on disk vs the actual data size

Venice uses leveled compaction by default and relies primarily on two methods to balance those trade-offs.

1. Tuning the compaction trigger

The key setting here is:

• level0_file_num_compaction_trigger

This controls the max number of files allowed in Level-0. Once you exceed it, compaction kicks in to push SST files from Level-0 to Level-1 and onward as upper levels fill.

Why it matters:

• higher threshold → fewer compactions → lower write amplification

• but also more Level-0 files → higher read amplification since reads may need to scan multiple files

• plus higher space amplification because duplicates hang around longer

Venice tunes this per cluster because clusters have different bottlenecks:

• memory-serving clusters want data in RAM to speed up lookups. Memory is the limiting resource, so they set a lower threshold to reduce space amplification

• disk-serving clusters are often limited by disk I/O, so they set a higher threshold to reduce compaction frequency and lower disk write rate

This is a practical tuning philosophy: tune to your real bottleneck, not a generic best practice.

2. RocksDB BlobDB integration

BlobDB is aimed at large-value workloads through key-value separation:

• Large values go into blob files

• LSM tree stores small pointers

This avoids copying large values repeatedly during compaction, reducing write amplification. The cost is additional space amplification because blobs can become unreferenced and require garbage collection.

For Venice, BlobDB integration reduced write amplification significantly in multi-tenant clusters, especially for large-value use cases. The reported impact here is big: more than a 50% reduction of disk write throughput. That matters because it avoided scaling out clusters when CPU and storage space were still available.

The win here is: you stop paying the compaction tax over and over on the same large payloads.

Use case 3: Active/active replication with partial update

Venice guarantees eventual consistency, not strong consistency. That matters because it means you cannot just do read-modify-write operations directly due to write delays.

To handle this, Venice introduces partial update, a specialized operation that supports field-level updates and collection merges.

Venice partial update workflow

Inside the Venice server, the leader replica:

• decodes the incoming payload

• applies the update

• re-encodes the result

• writes to the local database

• writes to the Version Topic

• follower replicas consume the merged results

Most of that is CPU-heavy.

Then the platform evolved further with active/active replication across multiple data centers. The key mechanism is deterministic conflict resolution (DCR), similar to CRDTs. Venice tracks update timestamps at row and field levels, compares incoming timestamps with existing ones and decides to apply or skip.

Venice Active/Active workflow

Now the leader replica has even more to do for DCR:

• timestamp metadata lookup

• decoding

• encoding

Again: CPU heavy. So the optimisation below focus on CPU efficiency.

1. Fast-Avro adoption

Fast-Avro was originally developed by RTBHouse but LinkedIn took over maintenance under the LinkedIn namespace and introduced many optimizations.

The key idea: Fast-Avro is an alternative to Apache Avro serialization and deserialization using runtime code generation which performs significantly better than the native implementation. It supports multiple Avro versions at runtime and is widely adopted inside LinkedIn.

Venice fully integrated Fast-Avro and saw, in one major use case, up to a 90% improvement in deserialization latency at p99 on the application side.

2. Parallel processing

In the traditional pipeline, DCR and partial update operations were executed sequentially, record by record within the same partition. That leads to CPU underutilization.

Venice introduced parallel processing so multiple records can be handled concurrently within the same partition before producing them to the version topic, while still preserving strict ordering in the final step.

Result: significantly improved write throughput for these complex record types.

Use Case 4: Active/active replication with deterministic write latency

Eventually consistent systems still get judged by human expectations. People want their writes to show up and they want it to happen predictably.

Venice is versioned and can ingest backup, current and future versions concurrently in a single server instance. In practice though, only the current version serves reads so deterministic write latency guarantees focus mostly there.

To improve determinism, Venice introduced a pooling strategy in ingestion with different priorities for different workload types. The Venice consumer phase is the first phase in the server ingestion pipeline and controlling the polling rate via pools is how prioritization happens.

Broad priority tiers:

• top priority: active/active and partial update workloads for the current version on the leader replica (CPU-intensive and latency-sensitive)

• next: other workload types targeting the current version

• then: active/active or partial update workloads for backup or future versions on the leader replica

• finally: everything else in a lower-priority bucket

This design is trying to do a few practical things:

• isolate CPU-heavy workloads so they don’t slow down lighter ones

• prioritize the current version so the most up-to-date data flows smoothly

• keep the number of pools limited to avoid resource management turning into a second job

The catch is tuning. Clusters see different workloads, store behavior varies widely even within one cluster, throughput swings over time and read traffic changes throughout the day. Static configs force you to tune for worst-case, which wastes resources most of the time.

So Venice introduced adaptive throttling: dynamically adjust ingestion based on recent performance.

• if the system is within agreed SLAs, ingestion rates are adjusted according to priorities

• if an SLA is violated, ingestion is throttled back immediately

Defining the SLAs matters. Venice focuses on two key criteria:

1. Read latency SLA: highest priority. Never violate read latency SLAs, even if it costs ingestion throughput

2. Write latency SLA for the current version: while read latency SLAs are met, write latency for the current version becomes top priority, pools are tuned proportionally to maximize utilization and throughput

Wrapping up

With these optimizations, Venice at LinkedIn handles:

• Over 175 million key lookups per second

• Over 230 million writes per second

• While maintaining a write latency SLA under 10 minutes

The full scoop

To learn more about this, check LinkedIn's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

It’s time for another round-up on all things data and AI!

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Without further ado, let’s get to the round up for January!

Data science & AI

• Agent design patterns (8 minute read)
  Anthropic engineer provides a grounded guide to designing AI agents that separates real, reliable architectures from overcomplicated agent hype that doesn’t survive production.

  

• Use Agents or Be Left Behind? A Personal Guide to Automating Your Own Work (31 minute read)
  Tim Dettmers cuts through the agent hype, arguing the real value isn’t autonomous magic but practical agents that reliably coordinate tools, memory and execution.

• Piecewise Regression for Time Series Forecasting (7 minute read)
  shares a practical walkthrough of using piecewise regression on time series to detect structural breaks, regime changes and trend shifts that single global models tend to smooth over.

  

• AI is Hitting a Measurement Wall (27 minute read)
  Makes the case that today’s AI benchmarks are saturated and misleading, masking the growing gap between model performance on tests and value in real applications.

• Drift Detection in Robust Machine Learning Systems (18 minute read)
  The article shows how unnoticed drift can quietly degrade model performance and outlines practical techniques to detect it early in production.

• 8 plots that explain the state of open models (7 minute read)
  Eight charts by cut through the noise to show that Chinese open models, led by Qwen, dominate real-world adoption and benchmarks, while Western challengers only compete at the very top end.

  

• LLMs as Judges: Measuring Bias, Hinting Effects, and Tier Preferences (10 minute read)
  and examine when LLMs can act as evaluators, showing how bias, prompt framing and hinting can distort model-as-judge benchmarks.

• How to Build a Recommendation System at Scale: Insights from Instacart (10 minute read)
  A practical walk-through of how large-scale recommendation systems are actually built in production by , covering modeling choices and the tradeoffs that matter once you move past toy examples.

Data engineering

• I spent 5 hours learning Unity Catalog. Here’s everything you need to know (10 minute read)
  provides a breakdown of how Databricks’ open-sourced Unity Catalog works under the hood.

  

• Databricks Lakeflow vs Apache Airflow (13 minute read)
  A candid comparison by showing how Databricks Lakeflow trades Airflow’s flexibility and openness for tighter platform integration, simpler ops and better defaults if you’re already all-in on Databricks.

• The Certifications Scam (7 minute read)
  A blunt takedown of data certifications by , arguing they mostly signal marketing and gatekeeping rather than real skills, experience or on-the-job impact.

• End To End Agentic Data Modeling: Using AI and OpenMetadata MCP for Impact Analysis (8 minute read)
  A hands-on look at building end-to-end agentic data modeling by combining OpenMetadata with MCP-style agents to automate lineage, context sharing and model evolution across the data stack by with

  

• Autofilling the Boring Semantic Layer: From Sakila to Chat-BI with dltHub (9 minute read)
  Adrian Brudaru explores how LLMs can help generate and maintain semantic models on top of data pipelines, reducing manual modeling effort while keeping analytics definitions consistent.

• A Diary of a Data Engineer (13 minute read)
  A candid, day-in-the-life reflection on what data engineering actually looks like in practice by , highlighting the unglamorous but essential work that keeps data systems running day to day.

• Database Development with AI in 2026 (11 minute read)
  Brent Ozar argues that in 2026 AI will meaningfully speed up database development tasks like query writing and troubleshooting but real impact still depends on human judgment and understanding production constraints.

• How Uber Cut Data Lake Freshness From Hours to Minutes With Flink (11 minute read)
  Uber rebuilt its data lake ingestion to move freshness from hours to minutes. This piece breaks down how they replaced batch Spark jobs with Flink streaming, cut compute by 25% and dealt with the very real problems that show up at petabyte scale.

Data analysis and visualisation

• Best Data Visualization Projects of 2025 (3 minute read)
  FlowingData shares the best data visualisations of 2025

  

• The book that finally taught me how to tell stories with data (12 minute read)
  reviews Storytelling with Data, highlighting that impact comes from framing the message and audience first, not from visualisation tricks.

• How to create a more accessible line chart (10 minute read)
  Nicola Rennie shows how small design choices in line charts (color, contrast, labeling and annotations) dramatically improve accessibility without sacrificing clarity or insight.

• 5 Rules for Dashboard Filter Placement (6 minute read)
  breaks down five practical rules for placing dashboard filters so users understand what they’re controlling without adding cognitive load or breaking trust.

Other interesting reads

• ONTOLOGIES - SOME PERSPECTIVES (20 minute read)
  A great intro and explanation of ontologies by (Bill Inmon) and . Really worth a read if you have heard the term a lot but are not sure what it means and how it can be applied

  

• Lessons from Building AI Agents for Financial Services (23 minute read)
  breaks down what building AI agents actually looks like in production, separating real engineering constraints from agent hype.

• Introducing the AI Chip Sales Data Explorer (3 minute read)
  Epoch AI introduces an interactive dataset tracking global AI chip sales, shedding light on who’s actually buying compute and how hardware demand is shaping the AI race.

  

Quick favor - need your take

Was there any standout article or topic from January I missed? Feel free to drop a comment or hit reply, even a quick line helps.

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Following on from previous posts talking to people in the field, today I will be talking with Ahsaas Bajaj who is a Senior Machine Learning Engineer at Instacart. He works on large-scale recommendation systems that serves millions of customers.

We talked about his rise from software engineering to machine learning at Instacart, how does he decide between rules based vs ML approaches and how he approaches the work now as a more senior stakeholder.

So without further ado, let’s get into it!

Can you tell us a bit about your role?

I’m a Senior ML Engineer at Instacart, working across customer and shopper experiences on large-scale recommendation systems that make millions of decisions each day. For the past three years, I’ve led the technical strategy for the Product Substitutions ML system, focused on solving the out-of-stock problem.

The goal is simple: when an item isn’t available, suggest a replacement that preserves customer intent and keeps the order intact. My role spans system design, modeling and evaluation, balancing customer satisfaction, shopper efficiency and business impact at scale.

How did you get into machine learning?

My path into ML wasn’t a straight line. I started as a software engineer at Samsung Research on the on-device search team, which pushed me deep into information retrieval and search system design. That work sparked an interest in research and led me to pursue a graduate degree in computer science.

It shaped how I approach ML today: less focus on models in isolation, more on how systems behave in production. I wanted that work to have real user impact, which took me to Walmart Labs and eventually to Instacart.

What does a ‘typical’ week look like for you?

As I’ve moved into a more senior role, the balance has shifted from pure coding to a mix of execution and direction. My week usually breaks down into three buckets:

Alignment (30%): The glue work. I spend time with product, backend engineering, and leadership aligning on roadmaps. The focus isn’t just what we’re building, but why, making sure ML work ties directly to business goals.

Deep work (30%): Hands-on modeling, coding and system design. Staying close to the code is non-negotiable for me, even at a senior level.

Analysis and “the why” (40%): This is where I spend the most time. I dig into model errors, read raw customer complaints about failed substitutions and sanity-check improvement ideas. This is also where I write proposal docs. In my view, the highest-leverage work a senior MLE does is deciding what problems to solve next, not just executing on what’s assigned.

How do you decide when a problem actually needs ML or if rules-based is good enough?

I think about it in terms of complexity versus value.

If a problem can be solved deterministically with clear rules and those rules are stable and understandable, that’s often the right solution. Machine learning becomes useful when the space of behaviors is too large, nuanced, or context-dependent for rules to scale.

Good data is also a prerequisite. Without reliable signals and feedback loops, even the most sophisticated model won’t perform well in production.

You have written about your work on a recommendation model at Instacart. Can you share a summary of what you have done?

I’ve spent the past three years leading the technical development of Instacart’s Product Substitutions system, which handles millions of replacement decisions daily. The core challenge is deceptively simple: when a customer’s requested item is out of stock, what should we suggest instead?

What makes this interesting from an ML perspective is that it’s fundamentally a relevance problem, not a search problem. We’re not just matching product attributes—we’re trying to understand what the customer actually wanted and find alternatives that preserve that intent. This required rethinking how we model the relationship between items, how we define “good” substitutions, and how we evaluate success in a way that maps to real customer satisfaction.

The system has evolved significantly over time, moving from simpler heuristics to more sophisticated learned representations. But the north star has always been the same: keep orders complete while respecting what customers actually care about.

And what has been the impact on the business?

Substitutions sit at a critical junction in the order lifecycle. When done well, they’re invisible - customers get what they need and the order stays intact. When done poorly, they create friction everywhere: customers reject items or request refunds, shoppers waste time on unsuccessful suggestions, and order values drop.

Our work has meaningfully moved the needle on the metrics that matter: replacement acceptance rates, refund frequency, and what we call “perfect order fill rate”—the percentage of orders where every item was either found or successfully replaced. These improvements compound across millions of weekly orders.

Beyond the immediate transactional metrics, we’ve also seen positive signals in repeat ordering behavior and customer satisfaction scores, particularly for orders that required multiple substitutions. Instacart has referenced this system publicly when discussing operational improvements at scale.

For me, the real validation is when customers don’t notice the algorithm at all - they just notice their groceries arrived complete.

What does the tech stack look like for ML at Instacart?

Instacart’s ML stack is built around an internal platform called Griffin, which standardizes the end-to-end ML lifecycle, from feature engineering and training to deployment and real-time inference. A core piece of this is a shared Feature Marketplace, where teams define, version and reuse batch and streaming features with strong offline-to-online consistency.

Workflows are orchestrated with Apache Airflow and model training runs through a unified abstraction that supports multiple compute backends and common ML frameworks. With Griffin 2.0, the platform moved to a Kubernetes-based setup and added distributed training with Ray, which significantly improved scalability and iteration speed.

Griffin also includes a centralized model registry and metadata store, making experiments easier to track and reproduce. In production, models are deployed as standardized services that handle feature loading and low-latency inference across both customer and shopper experiences.

The main benefit is focus: teams spend less time on infrastructure and more time on modeling, evaluation and trade-offs.

How do you use AI in your day-to-day work and where do you find it genuinely valuable?

I’ve integrated GenAI primarily to shift my focus from execution to decision-making. It’s useful for routine tasks like scaffolding data pipelines or optimizing SQL queries, but I find the highest leverage comes from qualitative analysis.

I routinely feed thousands of customer comments and shopper notes about bad substitutions into LLM-driven pipelines that cluster feedback into coherent themes. What used to be unstructured noise becomes a prioritized list of failure modes. This allows me to spend less time parsing data and more time solving the specific problems that actually impact customer trust.

How has your perspective changed moving to a more senior role?

The biggest shift is realizing that Judgment > Code. Early in my career, I obsessed over the how - the architecture, the libraries, the latency. Now, I obsess over the what and the why. The real work is filtering ideas. In a sea of seemingly good ideas, my job is to find the most bullish one - the one with the highest ROI - and kill the others.

I’ve also learned that Writing is Engineering. You cannot build big things alone. To get buy-in from leadership and cross-functional teams, you must be able to write crisp, narrative-driven proposals that explain why this mathematical solution solves a human problem.

What’s one thing you wish you’d known earlier about machine learning?

The value of error analysis. It’s easy to celebrate aggregate metrics like accuracy or F1 but the real breakthroughs come from studying the “horror cases,” where the model is confidently wrong. Those examples are uncomfortable to look at but they’re where the most useful ideas come from. You can’t fix what you don’t deeply understand.

If you enjoyed reading this, check out Ahsaas’s original article about his work at Instacart

Was there a question that you would like to ask?

Let me know your thoughts by replying to the email or leaving a comment below!

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep reading

Today we will look at how DoorDash uses anomaly detection to save millions of dollars by flagging fraud trends early.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to DoorDash’s fraud detection!

TL;DR

Situation

Fraud trends at DoorDash often blended into normal delivery noise and went unnoticed for weeks, causing avoidable losses. Existing detection was reactive and too slow.

Task

Detect emerging fraud trends early across millions of users and segments, before they materially impact top-line metrics.

Action

Build a daily anomaly detection platform that segments key fraud metrics across millions of overlapping dimensions, applies time-series z-score detection, clusters related anomalies and routes them into an ops investigation workflow.

Result

Cut average fraud detection time from 100+ days to under 3 days, surfaced 60%+ of new fraud trends early, and saved tens of millions annually.

Use Cases

Anomaly detection, fraud detection, payment monitoring, policy change impact monitoring

Tech Stack/Framework

Apache Airflow, DuckDB, Apache Spark, Python

Explained further

Fraud trend detection before it becomes a headline

Fraud doesn’t always kick the door down. Sometimes it slips in through the side window and blends into the noise of millions of legitimate deliveries.

A small spike in refund claims. A pattern in high-risk charges linked to a specific bank. A subtle shift in behavior that looks like randomness until it isn’t. Left alone, those early signals can snowball into a large trend with real top-line impact.

DoorDash’s fraud team wanted to flip the script. Instead of reacting after a new fraud trend has had weeks to grow unchecked, how could they spot it as early as possible, before significant damage is done?

This post shares how the DoorDash team built an anomaly detection platform that scans for emerging patterns across millions of user segments and surfaces the ones that matter before they spiral into major losses.

Terminology

‘Anomaly detection’ is a broad term. Even within fraud, people can mean very different things by it. For this system, DoorDash defined two categories up front:

Anomalous trend detection

Looking for anomalous behavior in a collection of users that may represent a new fraud or false-positive trend.

Here, no single datapoint needs to be weird. The anomaly is the time-series pattern that emerges from many points together, like a growing fraud segment over time.

Anomalous outlier detection

Looking for individual outliers, like a specific user or transaction that is rare or deviates sharply from normal behavior.

In this case, the datapoint is the anomaly. It might be part of a broader trend, or it might be a one-off.

This post focuses how DoorDash built a system to detect anomalous trends.

Here are some terms used within the article and their definitions and examples to make them easier to understand.

Designing the system around real fraud failures

The DoorDash team started the way you’d hope a fraud platform starts: by talking to the people who have to use it.

They met with frontline fraud teams responsible for tracking and fighting new fraud trends and asked for concrete historical examples of trends that simmered longer than ideal before being discovered and mitigated. These became the positive test cases.

Next, the teams were asked for:

• Their most useful early-warning indicator metrics

• The dimensions they commonly use to slice data when investigating a new fraud trend

That produced a working set of:

• Positive examples (historical missed or late-found fraud trends)

• A set of metrics that act as early-warning signals

• A set of dimensions that represent how investigators naturally segment the world

Then the DoorDash team built the system and backtested it. Tuning came next, but the tuning goal was very specific:

1- Maintain 100% recall on the test trends
2- Minimise the number of non-fraudulent anomalies per day

One observation stood out from this phase. The system was fairly insensitive to exact tuning values. What mattered more was upstream: choosing thoughtful metrics and dimensions that can actually capture fraud trends in the first place.

In other words: the math is important but the slices you choose decide what you can even see.

Architecture overview

The anomaly detection platform runs as a daily job coordinated by Airflow. It looks for fraud trends growing on a day-to-week timescale.

DoorDash currently runs anomaly detection jobs for both consumer fraud and Dasher fraud, with plans to expand to more applications over time.

The platform has five steps:

1. Preparing daily fraud snapshots

2. Metric aggregation on multi-dimensional segments

3. Time-series anomaly detection

4. Hierarchical clustering on anomalous segments

5. Turning clusters into investigations and containment

Step 1: Preparing daily fraud snapshots

DoorDash chose a daily batch job for the initial implementation because the fraud trends they historically missed developed over a few days to a few weeks.

An Airflow DAG prepares a dataset for each anomaly detection job containing the day’s data snapshot in a wide-table format.

If the trends you historically missed unfold across days and weeks, you do not need sub-second streaming to get meaningful wins. You need consistency, coverage and a reliable cadence.

Step 2: Metric aggregation on multi-dimensional segments

This is the scale step. Once the daily snapshot is ready, DoorDash loads the single date’s data into a Python environment via Spark, then computes metric aggregates across segments.

For each metric, they track both:

• Absolute value of the metric

  • Example: dollar value of credit and refund claims

• Relative (normalized) value of the metric

  • Example: credit and refund claims divided by dollar value of orders

Why both? because absolute values catch ‘this is costing real money’ and relative values catch ‘this is spiking compared to what is normal for this slice’.

Then comes segmentation. Segments are formed from single, double and triple product combinations of all dimensions. That quickly becomes huge and can run into 100s of millions of segments at Doordash scale and compute becomes important

DuckDB for aggregation

DoorDash computes metric aggregates using DuckDB, an in-memory Python database optimised for fast OLAP-style operations.

They chose DuckDB because it was:

• Much faster (less than 10 minutes)

• More memory efficient than Pandas

The system also excludes dimensional products with cardinality greater than 10^7 to reduce the total number of segments to a manageable size.

Finally, storage format.

The day’s metrics aggregated across hundreds of millions of segments are stored in the data warehouse in sparse tall table format.

In plain English: if a segment has a metric value of zero, DoorDash drops it. That cuts storage and keeps both DuckDB and the downstream warehouse from filling up with rows that say ‘nothing happened here.’

Step 3: Time-series anomaly detection

After Step 2, DoorDash has daily metric aggregates by segment. They keep the previous 28 days of data in the data warehouse, so the platform now has several hundred million metric time series, each of length 28.

DoorDash chose a simple moving-window z-score approach, because it performed well in testing and detected all historical fraudulent trends they used as positive examples.

Baseline and test setup

• First 21 days form the baseline

• The 28th day is the test day

• There is a 7-day gap between the baseline and the test day

That gap exists for a very specific reason. The team noticed many historical fraud trends had a noisy phase when they first started scaling. By leaving a gap, the baseline variance better reflects ‘normal before the trend’ which reduces missed trends.

What counts as an anomaly

A segment’s time series is flagged as anomalous if it meets both:

1. Statistical significance: The 28th-day relative metric is greater than X standard deviations above the mean of the 21-day baseline. DoorDash found 6 standard deviations worked well empirically.

2. Business significance: The 28th-day absolute metric exceeds the 21-day baseline by a dollar value and/or count that is meaningful for that metric. Thresholds vary by metric and were chosen with operations partners.

That two-part rule matters. Statistical significance alone finds weirdness. Business significance filters it down to weirdness that’s worth a human’s time.

Step 4: Hierarchical clustering on anomalous segments

Real fraud trends rarely show up as a single clean segment anomaly. A single trend often triggers anomalous increases across many partially overlapping segments. Example:

A spike in credit and refund claims at ‘Retailer One’ could cause anomalies in segments like:

• {business_name='Retailer One'}

• {country='US', business_name='Retailer One'}

• {business_vertical='retail', business_name='Retailer One'}

So Step 4 exists to shrink ‘thousands of anomalies’ into ‘a few dozen things to look at’.

Segment graph structure

Dimensional segments have a natural structure that can be represented as a three-layer graph:

• Top layer: singlets

  • {business_name='Retailer One'}

• Middle layer: pairs

  • {business_name='Retailer One', country='US'}

• Bottom layer: triplets

  • {business_name='Retailer One', country='US', checkout_platform='iOS'}

DoorDash further partitions the graph by METRIC_NAME so clustering happens within a metric type.

Clustering rules

To connect anomalies within the same metric type:

1. Connect parent anomalies with child anomalies

   • {business_name='Retailer One'} is parent of {country='US', business_name='Retailer One'}

   • {country='US', business_name='Retailer One'} is parent of {business_name='Retailer One', country='US', checkout_platform='iOS'}

2. Connect sibling anomaly triplets if they share 2/3 of their keys and values

   • {business_name='Retailer One', country='US', checkout_platform='iOS'}
     connects with
     {business_name='Retailer One', country='US', business_vertical='retail'}

Then DoorDash runs a graph partition algorithm to find connected anomaly clusters.

Picking a representative segment

Ops teams review a cluster starting from a single representative segment chosen using a fitness function:

fitness = abs_anom_amt * rel_amt / level^1.2

Where:

• abs_anom_amt = 28th-day metric minus the 21-day baseline

• rel_amt = relative (normalized) 28th-day metric within the segment

• level = 0 for singlets, 1 for pairs, 2 for triplets

The intuition:

• abs_anom_amt behaves a bit like ‘how much impact’ (think recall)

• rel_amt behaves a bit like ‘how concentrated’ (think precision)

• dividing by a weak function of level biases toward simpler segments

So the representative is usually a segment that is impactful, unusually high relative to its baseline and not needlessly specific.

What volume looks like in practice

In real operation, DoorDash typically sees anomalies in several thousand segments per day. Clustering reduces that to 20 to 60 anomalous clusters per day across consumer and Dasher fraud areas, which is a volume the operations team can realistically investigate.

Step 5: Turning clusters into investigations and containment

Detection is not the finish line, it is just the trigger.

The representative anomalous segments, along with all other segments in the cluster and example events (deliveries and Dasher assignments), are accessible in a workflow tool for ops investigation.

Ops agents review example deliveries or assignments within the representative segment, looking for trends or patterns that may represent a new fraud trend.

Sometimes the pattern is non-fraudulent, like a new promotion causing a spike in refunds. Other times it is fraudulent.

When a trend is deemed fraudulent:

• it is root-caused in partnership with engineering and product teams so the root cause can be addressed

• a separate containment team runs queries to identify and stop fraudsters matching the trend pattern until product fixes land

So the system is not just detection. It’s detection wired into investigation, containment and longer-term remediation.

Results

DoorDash now uses the anomaly detection platform as its primary early-warning source for new fraudulent trends.

Key results reported by the team:

• More than 60% of all new fraud trends today are found through anomaly detection, and that share is growing as coverage expands.

• Average time-to-detect new fraud trends dropped from more than 100 days to less than three days over the past year.

• The platform saves tens of millions of dollars per year by flagging small but growing fraud trends before they get out of control.

The full scoop

To learn more about this, check DoorDash's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Grab detects data issues in real-time.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Grab’s real-time work!

TL;DR

Situation

Grab runs critical systems on Kafka streams, where bad data can spread and break downstream consumers. Existing checks were slow and mostly limited to schemas, making issues hard to catch and debug.

Task

Detect bad streaming data early, cover both schema and value-level issues and give stream owners fast, actionable visibility without centralising ownership.

Action

Grab built contract-driven stream checks on Coban, turning schemas, field rules and ownership into real-time FlinkSQL tests with Slack alerts and UI-based inspection of bad records.

Result

The system now monitors 100+ Kafka topics in real time, surfaces poison data quickly and helps teams stop issues before they cascade downstream.

Use Cases

Root cause analysis, real-time monitoring, real-time alerting

Tech Stack/Framework

Apache Kafka, Apache Flink, Amazon S3, Slack, LLM

Explained further

About Grab

Grab is often called the Uber of Southeast Asia but that might be selling it short. What started as a ride-hailing app now powers food delivery, groceries, payments and even insurance all bundled into one super app. They run across over 800 cities in 8 Southeast Asian countries. Behind the rides, meals, and payments lies an enormous stream of events flowing through Grab’s systems.

Background

Grab runs a lot of business on streaming data. Kafka topics feed online systems, offline analytics and machine learning pipelines. When those streams are clean, life is good: teams can move faster, models behave, dashboards run smoothly. But when they’re not clean, it’s a major headache.

The tricky part is that ‘bad data’ in Kafka isn’t always obvious. Sometimes it’s quiet: the stream still parses but key fields are wrong, missing or shaped differently than what downstream teams assume.

That’s why Grab decided to introduce a platform-level solution: Kafka stream contracts that let stream stakeholders define what ‘good’ looks like, then automatically test streams in real time, catch issues as they happen and alert the owners quickly.

The core idea is simple:

• Let users define a data contract for a Kafka topic

• Convert that contract into executable tests

• Run those tests continuously

• Capture the poison data plus context

• Notify the right people with enough detail to act

This supports a more decentralized, data-mesh style world where teams own their data products while still keeping the overall system reliable for everyone else.

What wasn’t working before

Historically, monitoring Kafka stream data processing didn’t have a strong, end-to-end solution for data quality validation. That created three big issues: detecting bad data, speed of detection and lack of visibility.

1- Detecting bad data

This can be broken down into two further categories:

1.1 Schema issues

These are schema mismatches between producers and consumers that can trigger deserialization errors. Even if schema backward compatibility is validated during schema evolution, the data inside the Kafka topic can still drift from the defined schema.

One concrete example: a rogue producer writes to a topic without using the expected schema. Now you’ve got a topic that ‘has a schema’ but real events don’t match it. The painful bit is not just knowing something broke, it’s identifying which fields are causing the mismatch.

1.2 Rule and value issues
These are disagreements about what a field means or what shape it should take. Kafka stream schemas define structure but they don’t enforce rules like:

• expected length for an identifier

• expected string pattern

• valid numeric ranges

• constant values that should never change

There wasn’t an existing framework where stakeholders could define and enforce field-level semantic rules for streams.

2- Speed of detection

The second issue was speed of detection. There was no real-time mechanism to automatically validate data against predefined rules, identify issues quickly and alert stakeholders promptly.

Without real-time validation, issues could stick around for a while, quietly impacting multiple online and offline downstream systems before being discovered.

3- Lack of visibility

Even when teams did detect a problem, it was hard to pinpoint the exact ‘poison data’ and understand what violated the schema or the semantic expectations.

Root cause analysis becomes painful when you cannot easily answer:

• Which records were bad?

• Which fields failed?

• What did the bad values look like?

• When did it start and how frequent is it?

The fix

Grab’s Coban platform provides a standardized, platform-level data quality testing and observability setup for Kafka streams. It’s built around four core ideas:

1. Data Contract Definition: Stream stakeholders define a contract that includes schema agreements, semantic rules the topic data must follow, and ownership metadata for alerts and notifications.

2. Automated Test Execution: A long-running test runner automatically executes real-time tests based on that contract.

3. Real-time Data Quality Issue Identification: The system detects data issues in real time at both schema and rules/values levels.

4. Alerts and Result Observability: It alerts the right people and makes it easier to observe issues through the platform UI and downstream tooling.

Put simply: define the rules once, then let the platform watch the stream continuously.

The architecture has three main components:

1. Data contract definition

2. Test execution and data quality issue identification

3. Result observability

All Flow mentions after this refer to those diagrammed steps above

Data contract definition

Coban’s contract acts as a formal agreement among Kafka stream stakeholders. It includes a few building blocks.

Kafka Stream Schema (Flow 1.1)

The contract includes the schema used by the Kafka topic under test. This helps the Test Runner validate schema compatibility across data streams.

Importantly, this is not only about “did the schema change.” It’s also about “does the data actually match what everyone believes the schema is.”

Kafka Stream Configuration (Flow 1.2)

This includes essential config like endpoint and topic name. Coban automatically populates this so users don’t have to wire everything manually.

Observability Metadata (Flow 1.3)

This is where ownership becomes real. The contract includes contact details for stream stakeholders and alert configurations so the right people get notified when issues show up.

Kafka Stream Semantic Test Rules (Flow 1.5)

This is the heart of the semantic side. Users can define intuitive field-level rules such as:

• string pattern checks

• number range checks

• constant value checks

The point is to make the “meaning” of fields enforceable, not just their data types.

LLM-Based Semantic Test Rules Recommendation (Flow 1.4)

Defining dozens or hundreds of field rules can overwhelm people. To reduce that setup burden, Coban uses an LLM-based feature that recommends semantic test rules based on:

• the provided Kafka stream schema

• anonymized sample data

This feature helps users set up semantic rules efficiently, as demonstrated below

The practical benefit: users get a starting point quickly, instead of staring at a schema and trying to invent rules from scratch.

Data contract transformation

Once a contract is defined, Coban’s transformation engine converts it into configurations the Test Runner can interpret (Flow 2.1).

This transformation covers four things:

Kafka Stream Schema: The contract schema is translated into a schema reference format the Test Runner can parse.

Kafka Stream Configuration: The Kafka stream is set up as a source for the Test Runner.

Observability metadata: Contact information is turned into runtime configs for alerting and routing.

Kafka Stream Semantic Test Rules: Human-readable semantic rules are transformed into an inverse SQL query that captures data violating the rules.

‘Inverse SQL’ here means the query is designed to return the bad rows, not the good ones. That’s a smart design choice because it keeps the output focused on what needs investigation.

Test execution & data quality issue identification

Once the transformation engine generates the configuration, the platform automatically deploys the Test Runner.

Test runner

The Test Runner uses FlinkSQL as its compute engine. FlinkSQL was chosen because it makes defining rules straightforward using SQL statements, which also makes it easier for the platform to convert contracts into enforceable checks.

Test execution workflow and problematic data identification

Below are the 4 steps undertaken to execute the test and identify problematic data:

1. Consume Kafka data (Flow 2.2)
   FlinkSQL consumes data from the Kafka topic under test using its own consumer group. This is important because it avoids impacting other consumers.

2. Run inverse SQL (Flow 2.3)
   The Test Runner runs the inverse SQL query to identify:

   • data that violates semantic rules

   • data that is syntactically incorrect “in the first place”

3. Publish data quality issue events (Flow 3.2)
   When bad data is found, the Test Runner packages it into a data quality issue event enriched with:

   • a test summary

   • total count of bad records

   • sample bad data

   Then it publishes the event to a dedicated Kafka topic.

4. Sink events to S3 (Flow 3.1)
   The platform also sinks all data quality events to an AWS S3 bucket for deeper observability and analysis.

This combo (Kafka for realtime events, S3 for deeper inspection) gives both fast alerting and a more durable store for later analysis.

Result observability

Grab’s in-house data quality observability platform, Genchi, consumes the problematic data captured by the Test Runner (Flow 3.3).

Alerting

Genchi sends Slack notifications to stream owners listed in the contract’s observability metadata (Flow 3.5).

Those notifications include useful debugging context such as:

• links to sample data in the Coban UI

• observed time windows

• counts of bad records

• other relevant details

The key point is that alerts are not just ‘something broke’, they include the information you need to start investigating.

Observability

Users can access the Coban UI (Flow 3.4) to see:

• Kafka stream test rules

• sample bad records

• highlighted fields and values that violate rules

That UI piece matters because it shortens the path from ‘alert received’ to ‘I know what field is failing and what the bad values look like.’

Results so far

Since deploying earlier in the year, this solution enabled Kafka stream users to:

• define contracts with both schema and semantic rules

• automate real-time test execution

• alert stakeholders when problematic data is detected so they can act quickly

It has been actively monitoring data quality across 100+ critical Kafka topics.

The solution also offers the capability to immediately identify and halt the propagation of invalid data across multiple streams.

Wrapping up

Grab implemented and rolled out a real-time data quality monitoring solution for Kafka streams through the Coban platform.

The key outcomes include:

• engineers can define syntactic and semantic tests through a data contract

• tests run automatically in real time via a long-running Test Runner based on FlinkSQL

• issues trigger fast Slack alerts through Genchi using ownership metadata in the contract

• teams get better visibility into exactly which data fields violate rules via the Coban UI

In short: Coban turned data quality from a vague hope into something stream owners can specify, enforce and observe in real time.

The full scoop

To learn more about this, check Grab's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

It’s time for another round-up on all things data and AI!

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Without further ado, let’s get to the round up for December!

Data science & AI

• The State Of LLMs 2025: Progress, Problems, and Predictions (34 minute read)
  provides a great recap of main developments in 2025 and a couple of predictions for 2026 (like classical RAG slowly fading away)

  

• Building a Data Cleaning Agent with LangGraph (7 minute read)
  shows how to build a LangGraph-based data cleaning agent that auto-generates, executes, and fixes Python cleaning code to cut down manual data prep.

• Making Sense of Memory in AI Agents (10 minute read)
  This post breaks down how different memory types (short-term, long-term, and structured) let AI agents retain context across steps so they can act coherently instead of responding statelessly.

  

• Exploring TabPFN: A Foundation Model Built for Tabular Data (12 minute read)
  The article explores TabPFN, a foundation model pretrained on synthetic tasks that delivers strong tabular ML performance with near-zero tuning by reframing tabular prediction as conditional inference.

• How to Use Simple Data Contracts in Python for Data Scientists (8 minute read)
  Eirik Berge walks through a lightweight data contract implementation in Python to catch schema breakages early

• How AI is transforming work at Anthropic (35 minute read)
  An interesting look at how engineers and researchers at Anthropic actually use AI day to day and which parts of their work it genuinely helps with.

  

• We removed 80% of our agent’s tools (4 minute read)
  Vercel rebuilt their text-to-SQL agent by stripping away complex tooling and giving Claude direct file-system access, discovering that fewer tools, better documentation and ‘doing less’ made the agent faster, cheaper and more reliable.

  

Data engineering

• Opinionated Data Platforms vs. Open-Source (18 minute read)
  Good article by breaking down the tradeoffs between open-source and ‘opinionated’ and when it makes sense to go for the latter.

  

• LLMs for {PDF} Data Pipelines (8 minute read)
  experiments with using LLMs as part of a data pipeline, showing that agent-style PDF-to-JSON extraction can work in practice despite slowness and may be good enough for real-world automation.

• Snowflake vs Databricks Is the Wrong Debate (9 minute read)
  argues that the Snowflake vs Databricks debate is a distraction, with Databricks deliberately expanding role by role to own the full data stack and compete with cloud and enterprise platforms.

• Data Quality Design Patterns (11 minute read)
  breaks down practical data quality design patterns like WAP, AWAP, TAP and signal tables, showing how teams balance safety, cost and speed to keep bad data out of production pipelines.

  

• DuckDB: The Swiss Army Knife For Data Engineers (8 minute read)
  argues that DuckDB can replace most pandas, Spark, and Airflow workflows by letting data engineers run fast, scalable analytics and ETL directly with SQL, zero infrastructure and minimal complexity.

• How Snap Rebuilt Its ML Platform to Handle 10,000+ Daily Spark Jobs (14 minute read)
  This post breaks down how Snap rebuilt its ML platform with a unified Spark layer to tame spiky workloads, standardise pipelines, and reliably run 10,000+ production jobs a day without blowing up clusters.

Data analysis and visualisation

• Saloni’s guide to data visualization (41 minute read)
  Great and comprehensive post by where she distils data visualization down to first principles, showing how to choose charts, reduce clutter and design visuals that communicate insight instead of just decorating dashboards.

• Broken Chart: discover 9 visualization alternatives (10 minute read)
  Dominic Royé breaks down how common chart design mistakes distort interpretation, showing why many broken charts mislead viewers and how to fix them with clearer scales, context, and visual discipline.

Other interesting reads

• The Most Useful, Timeless Skill to Learn as a Data Professional (11 minute read)
  makes the case that real impact in data comes from using leverage and not just more lines of code

• 2026 - General Thoughts on What’s Ahead (6 minute read)
  thinks that 2026 will be a deliberately ‘boring’ year where AI hype cools off and teams are forced to focus on fundamentals that actually make AI work.

• The next data bottleneck (11 minute read)
  argues that as tools and models get better, the real constraint shifts to human bottlenecks like decision-making, ownership and organisational ability to turn data into action.

Quick favor - need your take

Was there any standout article or topic from November I missed? Feel free to drop a comment or hit reply, even a quick line helps.

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Uber moved from batch to streaming in their data lake.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Uber’s streaming solution

TL;DR

Situation

Batch-based ingestion meant data freshness was hours to days, slowing experimentation, analytics and ML across Uber’s core business domains.

Task

Move ingestion to minutes-level freshness at petabyte scale while lowering compute cost and keeping operations reliable across thousands of datasets.

Action

Built IngestionNext using Flink streaming from Kafka to Hudi, plus a control plane for operating ingestion at scale. Solved streaming bottlenecks (small files, partition skew, checkpoint vs commit alignment) to keep performance and correctness intact.

Result

• Freshness improved from hours to minutes-level.

• Compute usage reduced by ~25% vs batch ingestion.

• Compaction performance improved by ~10x with row-group merging.

Use Cases

Near-real time analytics, personalisation, operational analytics

Tech Stack/Framework

Apache Spark, Apache Kafka, Apache Flink, Apache Hudi, Apache Parquet

Explained further

Why data freshness became a platform priority at Uber?

Uber’s data lake sits underneath a lot of the company’s analytics and machine learning. If a team wants to measure an experiment, monitor performance, train a model or sanity-check a business change, it usually starts with is the data in the lake yet?

Historically, ingestion into the lake was batch-based. Freshness was measured in hours. That was fine when decisions moved at daily report speed. It starts to hurt when the business wants near-real-time loops: faster experiments, faster model iteration, faster detection of issues.

Over the past year, the team built and validated IngestionNext, a new ingestion system that switches the default mindset from batch to streaming. It’s centered on Apache Flink, reads events from Kafka, writes to the data lake in Apache Hudi format and operates at petabyte scale. Along the way, they had to solve the stuff that makes streaming annoying in practice: small files, partition skew, checkpoint vs commit alignment and the operational problem of running thousands of jobs reliably.

Why batch ingestion became a bottleneck?

Two main reasons: freshness and efficiency.

Freshness

As the business sped up, teams across Delivery, Rider, Mobility, Finance and Marketing Analytics kept asking the same thing: “Can we get the data sooner?”

Batch ingestion creates delays measured in hours and sometimes days. That lag slows down iteration and decision-making. In a world of continuous experimentation and fast model cycles, hours of latency is basically a tax on everything.

By moving ingestion to Flink-based streaming, the team reduced freshness from hours to minutes. That directly supports faster model launches, quicker experiments and more accurate analytics because the lake stays closer to what’s happening now.

Efficiency

Batch ingestion with Apache Spark is heavy by nature. Jobs run on a schedule, kick off distributed work at fixed intervals and keep doing that even when the workload is uneven. At Uber’s scale, with thousands of datasets and hundreds of petabytes, that adds up to hundreds of thousands of CPU cores running daily.

Streaming smooths this out. Instead of repeatedly spinning up large batch work, resources can scale with traffic in a more continuous way. Less overhead from scheduling, less big bang compute and more efficient usage overall.

IngestionNext: A streaming ingestion platform built for scale

At the data plane, events land in Apache Kafka. Flink jobs consume those events and write them into the data lake using Apache Hudi. Hudi provides transactional behavior like commits, rollbacks and time travel. Freshness and completeness are measured end-to-end from source to sink, not just “did the job run.”

Operating ingestion at this scale is not a set it and forget it situation. So the team built a control plane focused on automation and safety. It manages the ingestion job lifecycle (create, deploy, restart, stop, delete), handles config changes and runs health verification. The goal is simple: run thousands of ingestion jobs consistently without turning the platform into a giant manual babysitting exercise.

The system also supports regional failover and fallback strategies. If there’s an outage, ingestion can shift across regions. If needed, jobs can temporarily fall back to batch mode so ingestion stays available and data is not lost.

Solving the hard parts of streaming ingestion

Streaming buys freshness but it also introduces new failure modes. The team highlighted three major ones: small files, partition skew and checkpoint/commit synchronization.

Small files

Streaming writes data continuously. That tends to create lots of small Parquet files. Small files are a classic way to make query performance worse while also increasing metadata and storage overhead. You get fresher data, then you pay for it every time someone queries.

The common compaction approach merges Parquet files record by record. That means each file gets decompressed, decoded from columnar format into rows, merged, then encoded and compressed again. It works but it’s expensive and slow because you keep doing encode/decode work over and over.

To fix this, the team introduced row-group-level merging. Instead of dropping down into row format, the merge operates directly on Parquet’s native columnar structure. That avoids the expensive recompression path and improves compaction performance by more than an order of magnitude, around 10x.

There are open-source efforts exploring schema-evolution-aware merging using padding and masking to align schemas but that comes with added implementation complexity and maintenance risk.

So the team took a simpler path: enforce schema consistency during merging. Only files with identical schema are merged together. No masking, no low-level code modifications, less engineering overhead and still faster, more efficient and more reliable compaction.

Partition skew

Streaming ingestion depends on steady consumption from Kafka across Flink subtasks. The messy reality is that short-lived downstream slowdowns, like garbage collection pauses can unbalance consumption. Some partitions get read more than others. You end up with skew.

Skew doesn’t just look ugly on a dashboard. It can reduce compression efficiency and lead to slower queries downstream.

The fixes came from three angles:

• Operational tuning: aligning Flink parallelism with Kafka partitions and adjusting fetch parameters.

• Connector-level fairness: adding mechanisms like round-robin polling, pause/resume for heavy partitions and per-partition quotas.

• Observability: exposing per-partition lag metrics, adding skew-aware autoscaling and setting targeted alerts.

This is a good reminder that streaming issues often show up first as weird lag and then become why are queries slower now” If you can’t see skew clearly, you’ll chase symptoms forever.

Checkpoint and commit synchronization

Flink and Hudi each track progress but they track different things.

• Flink checkpoints track consumed offsets.

• Hudi commits track writes.

If failures happen and these drift out of sync, the system can skip data or duplicate it. In ingestion, either outcome is a serious problem.

The team solved this by extending Hudi commit metadata to embed Flink checkpoint IDs. With that linkage, recovery becomes deterministic during rollbacks or failovers. The system can reason about which checkpoint corresponds to which commit and recover without guessing.

Production results: faster data with lower cost

The team onboarded datasets to the Flink-based ingestion platform and validated performance on some of Uber’s largest datasets.

The early results:

• Freshness: improved from hours to minutes-level freshness.

• Efficiency: 25% reduction in compute usage compared to batch ingestion.

Extending real-time beyond ingestion

IngestionNext improves ingestion latency from online Kafka into the offline raw data lake. That’s a big step but it’s not the full story.

Freshness still stalls downstream in transformation and analytics layers. If ingestion is minutes but transformation is still slow, the point of decision is still stale.

The next frontier for Uber is extending real-time capability end-to-end: ingestion → transformation → real-time insights and analytics. This matters because Uber’s lake powers a long list of domains: Delivery, Mobility, Machine Learning, Rider, Marketplace, Maps, Finance and Marketing Analytics. Freshness is a cross-cutting requirement.

Conclusion

Uber’s shift from batch to streaming ingestion is a meaningful platform milestone. By re-architecting ingestion around Apache Flink, IngestionNext delivers fresher data, stronger reliability and scalable efficiency across a petabyte-scale lake.

The design is not just run Flink jobs. It includes operational foundations like an automated control plane, resiliency strategies and streaming-specific engineering work: faster compaction via row-group merging, skew controls and deterministic recovery by linking Flink checkpoints to Hudi commits.

The bigger idea is the mindset shift: treating freshness as a first-class dimension of data quality. With IngestionNext proven in production, the next push is clear: bring streaming into downstream transformation and analytics so the company can close the real-time loop, not just ingest data faster.

The full scoop

To learn more about this, check Uber's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today I will be talking with who writes the The Data Governance Playbook newsletter and works with companies on implementing data governance.

The Data Governance Playbook

Your go-to resource for mastering Data Governance through practical tips, expert insights, and a touch of humour !

By Charlotte Ledoux

I discovered her work through the CDO game (worth trying if you haven’t!). It reminded me how often data governance is misunderstood, despite becoming essential as AI takes off.

We talked about her move from analytics to governance, how the real value comes from clarity and ownership rather than tools and why the smartest governance programs start with listening long before they start with policies.

So without further ado, let’s get into it!

Can you tell us about your role?

I’m a Data & AI Governance expert. In practice, that means I make sure an organisation’s data is trustworthy, secure and responsibly used, especially as AI adoption accelerates. I help define the roles, responsibilities, processes and tools that state how data is collected, shared, protected and used so that teams can innovate with confidence rather than chaos.

How did you break into data governance?

Before specializing in data governance, I worked more hands-on in the data ecosystem : collaborating with data teams on data science, analytics and data strategy. Over time, I realized that the biggest blockers to effective data use weren’t tools or skills but rather unclear ownership, missing standards and a lack of trust.

Governance drew me in because it sits at the intersection of strategy, quality, ethics and business value. It’s the discipline that creates the structure needed for data to actually deliver impact.

So what is data governance? How do you explain it simply?

Data governance is the framework that ensures data is reliable, secure and used appropriately. It defines the rules, responsibilities and processes that allow an organization to manage data (and now AI!) in a controlled and value-driven way.

A simpler version I often use: it’s about enabling people to do great things with data.

What’s a common misunderstanding about data governance?

Read more

It’s time for another round-up on all things data!

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Without further ado, let’s get to the round up for November.

Read more

Today we will look at how Snap unified Spark, ML workflows and 10k+ daily jobs under one platform.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Snap’s ML platform transformation.

Read more

Following on from previous posts talking to people in the field, today I will be talking with who is a Senior Data Science Manager at Skyscanner and writer of the Senior Data Science Lead newsletter.

Senior Data Science Lead

Helping managers build world-class teams, data professionals master storytelling and guiding those looking to break into Data Science. I have built teams from scratch and lead 50+ data scientists. Now, I share my experience with you.

By Jose Parreño Garcia

We talked about his rise from data analyst to Senior DS Manager at Skyscanner, what “production-ready” really means and why the real intelligence in data science lives before and after the model.

So without further ado, let’s get into it!

Read more

It’s time for another round-up on all things data!

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Without further ado, let’s get to the round up for October.

Read more