Today we will look at how Shopify scales its product categorisation using agentic AI

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Shopify’s multi-agent taxonomy

TL;DR

Situation

Shopify’s product classification system makes tens of millions of predictions daily, across a taxonomy with 10,000+ categories and 2,000+ attributes. Commerce changes fast, the taxonomy has to keep up or the whole stack starts drifting.

Task

Keep the taxonomy current at scale without relying on slow, reactive, manual curation. Fix volume, expertise and consistency problems before they hit merchants, customers and model quality.

Action

Built an AI multi-agent system: structural analysis + product-driven analysis, then intelligent synthesis. Added equivalence detection (category = broader category + attribute filters) plus automated QA via domain-specific AI judges.

Result

Taxonomy branches can be analyzed in parallel: hundreds of categories instead of a few per day. Quality improved via grounded merchant data + structural consistency, with judges filtering proposals (example: “MagSafe compatible” approved at 93% confidence).

Use Cases

Category discovery, attribute gap detection, taxonomy maintenance, search and filtering improvement

Tech Stack/Framework

AI agent, equivalence detection, multi-agent system

Explained further

Context

Last year, over 875 million people bought items from Shopify merchants. Shopify already runs a product classification system that makes tens of millions of predictions daily with a high degree of accuracy.

But classification is the easy part compared to the thing underneath it: taxonomy. Because the model doesn’t just need to be right, it also needs a clean, consistent set of labels to be right about.

That’s the challenge for Shopify: once you have 10,000+ categories and 2,000+ attributes, the taxonomy becomes its own product with its own failure modes. It can get stale. It can get inconsistent. It can drift away from how merchants actually describe products. And when that happens, the classifier quality takes the blame for what is basically a taxonomy debt problem.

So this post is about what Shopify did next: they built an AI multi-agent system that doesn’t just classify products, it actively improves the taxonomy labels themselves so the system stays agile as commerce changes.

The challenge: scaling taxonomy without losing accuracy

A taxonomy is a contract between three groups that rarely agree:

• Merchants describing products the way they think about them

• Customers searching and filtering with their own mental model

• Platform systems trying to enforce structure so everything stays queryable and comparable

Now add the reality that commerce never sits still. New products appear. Old categories split. Entire verticals get reshaped by trends, tech and regulation. The taxonomy has to keep up or the platform drifts away from how people actually shop and sell.

Shopify frames the challenge as three problems.

The volume problem: manual updates can’t keep up

A global product taxonomy needs constant attention. Every new product type, emerging technology category and seasonal trend potentially triggers taxonomy updates.

Manual curation becomes a bottleneck because taxonomy work is not one change. It is usually a bundle: a category addition, a hierarchy decision, a set of attributes, naming alignment and a check for duplicates or conflicts.

For example, consider the emergence of categories like smart home devices or remote work equipment. Each category represents not just new categories but also entirely new attribute sets.

Smart home devices for instance need connectivity types, power requirements and compatibility. Those are specs that did not exist in the taxonomy before.

So the work isn’t a one-off. It’s continuous expansion and adjustment across a giant tree of concepts.

The expertise problem: every vertical has its own rules

Good taxonomy design is domain-heavy. You do not get it right by being generally smart. You get it right by knowing what matters in that product world. For example, there are nuanced differences between types of guitar pickups or appropriate attributes for skincare products.

A taxonomy team can’t realistically maintain deep expertise across every vertical that merchants sell into. But if the taxonomy is inconsistent or poorly structured, merchants pay for it through reduced discoverability, suboptimal search results and ineffective filters for customers.

The consistency problem: one concept, five different labels

As the taxonomy grows organically, inconsistencies creep in:

• similar concepts represented differently across categories

• naming conventions inconsistent

• discrepancies between merchant categorization and customer expectations

Those inconsistencies compound. Merchants get confused when listing. Customers get frustrated when filtering and comparing. And the classifier quality drops because labels stop being reliably meaningful across the tree.

This is the part most teams underestimate. In a taxonomy, small inconsistencies behave like small data quality issues: they don’t stay small.

From manual taxonomy work to agent-led evolution

Shopify’s taxonomy management evolved from a manual workflow into an AI-driven system.

The old way: Expert review, slow throughput

The traditional pattern is familiar:

1. domain experts analyze product data

2. identify gaps or inconsistencies

3. propose changes

4. implement changes via careful review

It ensures quality but it also creates bottlenecks.

The biggest problem was the reactive nature of it: Shopify would only recognize the need for new categories or attributes after merchants began listing products that didn’t fit. By then, the system had already missed chances to give merchants and customers a better experience.

So even when you do great manual work, you’re always late.

The breakthrough: Two lenses, one system

Advanced language models opened a door: not to replace human experts, but to augment them with scale and consistency.

The key insight was that taxonomy improvement comes from two different angles:

• structural analysis: the logical structure of the taxonomy, gaps in hierarchies, missing relationships

• product-driven analysis: what real product data says merchants actually sell and how they describe it

Each angle catches different issues. Shopify’s breakthrough was combining them into a system that can continuously propose improvements then filter them through quality checks before human review.

Inside the system: How the agents work

The new architecture rests on three principles:

• specialized analysis

• intelligent coordination

• quality assurance

And the intent is clear: continuous evolution, not one-time taxonomy construction.

What’s different: continuous evolution, not one-time creation

AI’s been used for product categorisation and one-off taxonomy builds for a while. The difference here is instead of building it once and hoping it holds, Shopify uses specialised AI agents to keep the taxonomy evolving continuously. There are 3 core components to this approach:

1- Real product grounding: The system integrates actual merchant product data so proposals reflect how merchants describe and categorize products. This keeps decisions grounded in commerce reality rather than only theory.

In other words: if merchants are consistently describing a differentiator, it probably belongs in the taxonomy, even if it offends someone’s idea of a “pure” category tree.

2- Multi-agent specialization: Multiple specialized agents run different analyses. One focuses on structural consistency. Another focuses on product-driven insights. Then those outputs are synthesized. The claim here is that the combination finds improvements that neither agent would find alone.

That makes sense structurally. Taxonomy is both a graph problem and a language problem.

3- Sophisticated equivalence discovery: This is the most interesting component. detecting equivalence relationships where a specific category equals a broader category filtered by attribute values.

This matters because merchants should be able to organize their catalogs however they want, while the platform still understands what products ‘mean’ underneath the merchant’s choices.

So instead of forcing everyone into one rigid structure, Shopify tries to learn mappings that preserve flexibility and still support search, recommendations, and analytics.

Architecture flow

The AI agent workflow works like this:

• enable agents to explore the taxonomy

• run multi-stage analysis (structural + product-driven)

• synthesize and resolve conflicts

• detect equivalences

• run automated QA using judges

• send refined proposals to humans

• update the taxonomy in production

Enabling agent-taxonomy interaction

Before agents can improve anything, they need to ‘read’ the taxonomy like a human would.

Shopify implemented a system that allows agents to:

• search for related categories

• examine hierarchical relationships

• verify whether proposed changes conflict with existing elements

A good example: an agent analyzing guitar-related categories can explore the full musical instruments hierarchy, inspect related attributes across instruments and look for patterns that suggest better structure.

In other words, the agent doesn’t just look at one node. It roams the neighborhood.

The pipeline: specialised agents, staged decisions

For the AI Agent to be work properly, different specialised agents are at work to provide specific insights:

Structural analysis: This agent looks at the taxonomy itself for logical consistency, completeness, gaps in category hierarchies, naming convention inconsistencies and opportunities to reorganize related concepts.

It operates purely on the taxonomy structure and aims to keep the whole thing coherent.

Product-driven analysis: This agent integrates real merchant data and examines how products are described and categorized on the platform.

Specifically, it looks at patterns in product titles, product descriptions and merchant-defined categories. The goal is to find gaps between how merchants think about products and how the taxonomy represents them.

This is an important distinction. A taxonomy can be structurally perfect and still be useless if it doesn’t match merchant reality.

Intelligent synthesis: Now we have two streams of recommendations:

• structure-driven improvements

• product-driven improvements

They can conflict. They can overlap. They can propose redundant changes.

The synthesis step merges insights, resolves conflicts, and eliminates redundancies. And sometimes the best answer is not pick one, it’s combine both.

Equivalence detection: This agent solves a practical commerce problem: merchants want flexibility but platform systems need consistency.

Consider golf shoes:

• Merchant A uses a specific ‘Golf Shoes’ category

• Merchant B uses ‘Athletic Shoes’ with an ‘Activity Type = Golf attribute

Both are valid for the merchant. But search, recommendations and analytics benefit from understanding these represent the same product set.

So the system detects attribute-based equivalences of the form:

specific category = broader category + one or more attribute filters

This lets merchants organize however makes sense for their business while keeping platform intelligence consistent across different catalog structures.

If you’ve ever tried to do cross-merchant analytics at scale, you can probably feel why Shopify cared enough to build an entire agent for this.

Automated QA: judges before humans

After proposals are generated, Shopify adds automated QA through specialized AI judges.

These judges evaluate proposed changes using reasoning capabilities and taxonomy design principles to filter and refine suggestions before human review.

The important detail is that evaluation differs by change type:

• adding new attributes

• creating category hierarchies

• modifying existing structures

Different changes require different criteria, so one generic ‘judge prompt’ would be weak. So instead, they use domain-specific judges.

An electronics-focused judge applies electronics expertise. A musical instruments judge applies that domain’s patterns and rules. The goal is consistent domain-aware evaluation across verticals.

Results

The system can analyze taxonomy branches in parallel, identifying improvement opportunities that used to take weeks of manual work.

Where experts might analyze a few categories per day, the system can evaluate hundreds of categories, checking both:

• structural consistency

• alignment with real product data

This matters most for emerging product categories. When new product types become popular on the platform, the system can quickly identify taxonomy gaps and propose comprehensive solutions, instead of reactive patches that build up debt.

Quality improvements

The multi-agent design improves consistency and comprehensiveness because it combines two lenses:

• structural analysis keeps hierarchy organization logical and consistent

• product-driven analysis keeps categories and attributes aligned with merchant reality

The automated QA layer reduces iteration cycles by catching issues before human review and applying domain expertise consistently.

Example: mobile phone accessories and MagSafe compatibility

Product analysis identified that merchants frequently advertise “MagSafe support” for accessories such as chargers, cases and wallets.

So the agent proposed adding a boolean attribute: ‘MagSafe compatible.’

A specialized electronics judge evaluated the proposal and checked:

• no duplicate attribute already exists

• boolean type is appropriate

• while brand-specific, MagSafe is treated as a legitimate technical standard similar to Bluetooth or Qi

The judge approved the attribute with 93% confidence, noting it would improve customer filtering for MagSafe-ready products.

This example matters because it demonstrates the full loop:

• merchant reality creates a signal

• the agent proposes a structured change

• a domain judge validates it with rule checks and domain framing

• humans get a higher quality proposal to review

Scaling development: from reactive fixes to proactive evolution

The biggest shift is strategic: taxonomy development becomes proactive, not reactive.

Instead of waiting for a merchant pain point or a platform limitation to trigger a change, the system can identify and address gaps earlier.

The system can also reason over the entire taxonomy structure, which supports cross-category consistency. That helps avoid the fragmentation you get when teams fix issues in isolation.

To validate the approach, they applied it to a specific area: Electronics > Communications > Telephony (called “Telephony AI” in their analysis) and compared it against their previous manual expansion method.

As you can see from the chart, the AI-assisted method can compress years of work into weeks for the taxonomy area if the agents are applied across all verticals.

The full scoop

To learn more about this, check Shopify's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Following on from previous posts talking to people in the field, today I will be talking with Ahsaas Bajaj who is a Senior Machine Learning Engineer at Instacart. He works on large-scale recommendation systems that serves millions of customers.

We talked about his rise from software engineering to machine learning at Instacart, how does he decide between rules based vs ML approaches and how he approaches the work now as a more senior stakeholder.

So without further ado, let’s get into it!

Can you tell us a bit about your role?

I’m a Senior ML Engineer at Instacart, working across customer and shopper experiences on large-scale recommendation systems that make millions of decisions each day. For the past three years, I’ve led the technical strategy for the Product Substitutions ML system, focused on solving the out-of-stock problem.

The goal is simple: when an item isn’t available, suggest a replacement that preserves customer intent and keeps the order intact. My role spans system design, modeling and evaluation, balancing customer satisfaction, shopper efficiency and business impact at scale.

How did you get into machine learning?

My path into ML wasn’t a straight line. I started as a software engineer at Samsung Research on the on-device search team, which pushed me deep into information retrieval and search system design. That work sparked an interest in research and led me to pursue a graduate degree in computer science.

It shaped how I approach ML today: less focus on models in isolation, more on how systems behave in production. I wanted that work to have real user impact, which took me to Walmart Labs and eventually to Instacart.

What does a ‘typical’ week look like for you?

As I’ve moved into a more senior role, the balance has shifted from pure coding to a mix of execution and direction. My week usually breaks down into three buckets:

Alignment (30%): The glue work. I spend time with product, backend engineering, and leadership aligning on roadmaps. The focus isn’t just what we’re building, but why, making sure ML work ties directly to business goals.

Deep work (30%): Hands-on modeling, coding and system design. Staying close to the code is non-negotiable for me, even at a senior level.

Analysis and “the why” (40%): This is where I spend the most time. I dig into model errors, read raw customer complaints about failed substitutions and sanity-check improvement ideas. This is also where I write proposal docs. In my view, the highest-leverage work a senior MLE does is deciding what problems to solve next, not just executing on what’s assigned.

How do you decide when a problem actually needs ML or if rules-based is good enough?

I think about it in terms of complexity versus value.

If a problem can be solved deterministically with clear rules and those rules are stable and understandable, that’s often the right solution. Machine learning becomes useful when the space of behaviors is too large, nuanced, or context-dependent for rules to scale.

Good data is also a prerequisite. Without reliable signals and feedback loops, even the most sophisticated model won’t perform well in production.

You have written about your work on a recommendation model at Instacart. Can you share a summary of what you have done?

I’ve spent the past three years leading the technical development of Instacart’s Product Substitutions system, which handles millions of replacement decisions daily. The core challenge is deceptively simple: when a customer’s requested item is out of stock, what should we suggest instead?

What makes this interesting from an ML perspective is that it’s fundamentally a relevance problem, not a search problem. We’re not just matching product attributes—we’re trying to understand what the customer actually wanted and find alternatives that preserve that intent. This required rethinking how we model the relationship between items, how we define “good” substitutions, and how we evaluate success in a way that maps to real customer satisfaction.

The system has evolved significantly over time, moving from simpler heuristics to more sophisticated learned representations. But the north star has always been the same: keep orders complete while respecting what customers actually care about.

And what has been the impact on the business?

Substitutions sit at a critical junction in the order lifecycle. When done well, they’re invisible - customers get what they need and the order stays intact. When done poorly, they create friction everywhere: customers reject items or request refunds, shoppers waste time on unsuccessful suggestions, and order values drop.

Our work has meaningfully moved the needle on the metrics that matter: replacement acceptance rates, refund frequency, and what we call “perfect order fill rate”—the percentage of orders where every item was either found or successfully replaced. These improvements compound across millions of weekly orders.

Beyond the immediate transactional metrics, we’ve also seen positive signals in repeat ordering behavior and customer satisfaction scores, particularly for orders that required multiple substitutions. Instacart has referenced this system publicly when discussing operational improvements at scale.

For me, the real validation is when customers don’t notice the algorithm at all - they just notice their groceries arrived complete.

What does the tech stack look like for ML at Instacart?

Instacart’s ML stack is built around an internal platform called Griffin, which standardizes the end-to-end ML lifecycle, from feature engineering and training to deployment and real-time inference. A core piece of this is a shared Feature Marketplace, where teams define, version and reuse batch and streaming features with strong offline-to-online consistency.

Workflows are orchestrated with Apache Airflow and model training runs through a unified abstraction that supports multiple compute backends and common ML frameworks. With Griffin 2.0, the platform moved to a Kubernetes-based setup and added distributed training with Ray, which significantly improved scalability and iteration speed.

Griffin also includes a centralized model registry and metadata store, making experiments easier to track and reproduce. In production, models are deployed as standardized services that handle feature loading and low-latency inference across both customer and shopper experiences.

The main benefit is focus: teams spend less time on infrastructure and more time on modeling, evaluation and trade-offs.

How do you use AI in your day-to-day work and where do you find it genuinely valuable?

I’ve integrated GenAI primarily to shift my focus from execution to decision-making. It’s useful for routine tasks like scaffolding data pipelines or optimizing SQL queries, but I find the highest leverage comes from qualitative analysis.

I routinely feed thousands of customer comments and shopper notes about bad substitutions into LLM-driven pipelines that cluster feedback into coherent themes. What used to be unstructured noise becomes a prioritized list of failure modes. This allows me to spend less time parsing data and more time solving the specific problems that actually impact customer trust.

How has your perspective changed moving to a more senior role?

The biggest shift is realizing that Judgment > Code. Early in my career, I obsessed over the how - the architecture, the libraries, the latency. Now, I obsess over the what and the why. The real work is filtering ideas. In a sea of seemingly good ideas, my job is to find the most bullish one - the one with the highest ROI - and kill the others.

I’ve also learned that Writing is Engineering. You cannot build big things alone. To get buy-in from leadership and cross-functional teams, you must be able to write crisp, narrative-driven proposals that explain why this mathematical solution solves a human problem.

What’s one thing you wish you’d known earlier about machine learning?

The value of error analysis. It’s easy to celebrate aggregate metrics like accuracy or F1 but the real breakthroughs come from studying the “horror cases,” where the model is confidently wrong. Those examples are uncomfortable to look at but they’re where the most useful ideas come from. You can’t fix what you don’t deeply understand.

If you enjoyed reading this, check out Ahsaas’s original article about his work at Instacart

Was there a question that you would like to ask?

Let me know your thoughts by replying to the email or leaving a comment below!

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep reading

Today we will look at how DoorDash uses anomaly detection to save millions of dollars by flagging fraud trends early.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to DoorDash’s fraud detection!

TL;DR

Situation

Fraud trends at DoorDash often blended into normal delivery noise and went unnoticed for weeks, causing avoidable losses. Existing detection was reactive and too slow.

Task

Detect emerging fraud trends early across millions of users and segments, before they materially impact top-line metrics.

Action

Build a daily anomaly detection platform that segments key fraud metrics across millions of overlapping dimensions, applies time-series z-score detection, clusters related anomalies and routes them into an ops investigation workflow.

Result

Cut average fraud detection time from 100+ days to under 3 days, surfaced 60%+ of new fraud trends early, and saved tens of millions annually.

Use Cases

Anomaly detection, fraud detection, payment monitoring, policy change impact monitoring

Tech Stack/Framework

Apache Airflow, DuckDB, Apache Spark, Python

Explained further

Fraud trend detection before it becomes a headline

Fraud doesn’t always kick the door down. Sometimes it slips in through the side window and blends into the noise of millions of legitimate deliveries.

A small spike in refund claims. A pattern in high-risk charges linked to a specific bank. A subtle shift in behavior that looks like randomness until it isn’t. Left alone, those early signals can snowball into a large trend with real top-line impact.

DoorDash’s fraud team wanted to flip the script. Instead of reacting after a new fraud trend has had weeks to grow unchecked, how could they spot it as early as possible, before significant damage is done?

This post shares how the DoorDash team built an anomaly detection platform that scans for emerging patterns across millions of user segments and surfaces the ones that matter before they spiral into major losses.

Terminology

‘Anomaly detection’ is a broad term. Even within fraud, people can mean very different things by it. For this system, DoorDash defined two categories up front:

Anomalous trend detection

Looking for anomalous behavior in a collection of users that may represent a new fraud or false-positive trend.

Here, no single datapoint needs to be weird. The anomaly is the time-series pattern that emerges from many points together, like a growing fraud segment over time.

Anomalous outlier detection

Looking for individual outliers, like a specific user or transaction that is rare or deviates sharply from normal behavior.

In this case, the datapoint is the anomaly. It might be part of a broader trend, or it might be a one-off.

This post focuses how DoorDash built a system to detect anomalous trends.

Here are some terms used within the article and their definitions and examples to make them easier to understand.

Designing the system around real fraud failures

The DoorDash team started the way you’d hope a fraud platform starts: by talking to the people who have to use it.

They met with frontline fraud teams responsible for tracking and fighting new fraud trends and asked for concrete historical examples of trends that simmered longer than ideal before being discovered and mitigated. These became the positive test cases.

Next, the teams were asked for:

• Their most useful early-warning indicator metrics

• The dimensions they commonly use to slice data when investigating a new fraud trend

That produced a working set of:

• Positive examples (historical missed or late-found fraud trends)

• A set of metrics that act as early-warning signals

• A set of dimensions that represent how investigators naturally segment the world

Then the DoorDash team built the system and backtested it. Tuning came next, but the tuning goal was very specific:

1- Maintain 100% recall on the test trends
2- Minimise the number of non-fraudulent anomalies per day

One observation stood out from this phase. The system was fairly insensitive to exact tuning values. What mattered more was upstream: choosing thoughtful metrics and dimensions that can actually capture fraud trends in the first place.

In other words: the math is important but the slices you choose decide what you can even see.

Architecture overview

The anomaly detection platform runs as a daily job coordinated by Airflow. It looks for fraud trends growing on a day-to-week timescale.

DoorDash currently runs anomaly detection jobs for both consumer fraud and Dasher fraud, with plans to expand to more applications over time.

The platform has five steps:

1. Preparing daily fraud snapshots

2. Metric aggregation on multi-dimensional segments

3. Time-series anomaly detection

4. Hierarchical clustering on anomalous segments

5. Turning clusters into investigations and containment

Step 1: Preparing daily fraud snapshots

DoorDash chose a daily batch job for the initial implementation because the fraud trends they historically missed developed over a few days to a few weeks.

An Airflow DAG prepares a dataset for each anomaly detection job containing the day’s data snapshot in a wide-table format.

If the trends you historically missed unfold across days and weeks, you do not need sub-second streaming to get meaningful wins. You need consistency, coverage and a reliable cadence.

Step 2: Metric aggregation on multi-dimensional segments

This is the scale step. Once the daily snapshot is ready, DoorDash loads the single date’s data into a Python environment via Spark, then computes metric aggregates across segments.

For each metric, they track both:

• Absolute value of the metric

  • Example: dollar value of credit and refund claims

• Relative (normalized) value of the metric

  • Example: credit and refund claims divided by dollar value of orders

Why both? because absolute values catch ‘this is costing real money’ and relative values catch ‘this is spiking compared to what is normal for this slice’.

Then comes segmentation. Segments are formed from single, double and triple product combinations of all dimensions. That quickly becomes huge and can run into 100s of millions of segments at Doordash scale and compute becomes important

DuckDB for aggregation

DoorDash computes metric aggregates using DuckDB, an in-memory Python database optimised for fast OLAP-style operations.

They chose DuckDB because it was:

• Much faster (less than 10 minutes)

• More memory efficient than Pandas

The system also excludes dimensional products with cardinality greater than 10^7 to reduce the total number of segments to a manageable size.

Finally, storage format.

The day’s metrics aggregated across hundreds of millions of segments are stored in the data warehouse in sparse tall table format.

In plain English: if a segment has a metric value of zero, DoorDash drops it. That cuts storage and keeps both DuckDB and the downstream warehouse from filling up with rows that say ‘nothing happened here.’

Step 3: Time-series anomaly detection

After Step 2, DoorDash has daily metric aggregates by segment. They keep the previous 28 days of data in the data warehouse, so the platform now has several hundred million metric time series, each of length 28.

DoorDash chose a simple moving-window z-score approach, because it performed well in testing and detected all historical fraudulent trends they used as positive examples.

Baseline and test setup

• First 21 days form the baseline

• The 28th day is the test day

• There is a 7-day gap between the baseline and the test day

That gap exists for a very specific reason. The team noticed many historical fraud trends had a noisy phase when they first started scaling. By leaving a gap, the baseline variance better reflects ‘normal before the trend’ which reduces missed trends.

What counts as an anomaly

A segment’s time series is flagged as anomalous if it meets both:

1. Statistical significance: The 28th-day relative metric is greater than X standard deviations above the mean of the 21-day baseline. DoorDash found 6 standard deviations worked well empirically.

2. Business significance: The 28th-day absolute metric exceeds the 21-day baseline by a dollar value and/or count that is meaningful for that metric. Thresholds vary by metric and were chosen with operations partners.

That two-part rule matters. Statistical significance alone finds weirdness. Business significance filters it down to weirdness that’s worth a human’s time.

Step 4: Hierarchical clustering on anomalous segments

Real fraud trends rarely show up as a single clean segment anomaly. A single trend often triggers anomalous increases across many partially overlapping segments. Example:

A spike in credit and refund claims at ‘Retailer One’ could cause anomalies in segments like:

• {business_name='Retailer One'}

• {country='US', business_name='Retailer One'}

• {business_vertical='retail', business_name='Retailer One'}

So Step 4 exists to shrink ‘thousands of anomalies’ into ‘a few dozen things to look at’.

Segment graph structure

Dimensional segments have a natural structure that can be represented as a three-layer graph:

• Top layer: singlets

  • {business_name='Retailer One'}

• Middle layer: pairs

  • {business_name='Retailer One', country='US'}

• Bottom layer: triplets

  • {business_name='Retailer One', country='US', checkout_platform='iOS'}

DoorDash further partitions the graph by METRIC_NAME so clustering happens within a metric type.

Clustering rules

To connect anomalies within the same metric type:

1. Connect parent anomalies with child anomalies

   • {business_name='Retailer One'} is parent of {country='US', business_name='Retailer One'}

   • {country='US', business_name='Retailer One'} is parent of {business_name='Retailer One', country='US', checkout_platform='iOS'}

2. Connect sibling anomaly triplets if they share 2/3 of their keys and values

   • {business_name='Retailer One', country='US', checkout_platform='iOS'}
     connects with
     {business_name='Retailer One', country='US', business_vertical='retail'}

Then DoorDash runs a graph partition algorithm to find connected anomaly clusters.

Picking a representative segment

Ops teams review a cluster starting from a single representative segment chosen using a fitness function:

fitness = abs_anom_amt * rel_amt / level^1.2

Where:

• abs_anom_amt = 28th-day metric minus the 21-day baseline

• rel_amt = relative (normalized) 28th-day metric within the segment

• level = 0 for singlets, 1 for pairs, 2 for triplets

The intuition:

• abs_anom_amt behaves a bit like ‘how much impact’ (think recall)

• rel_amt behaves a bit like ‘how concentrated’ (think precision)

• dividing by a weak function of level biases toward simpler segments

So the representative is usually a segment that is impactful, unusually high relative to its baseline and not needlessly specific.

What volume looks like in practice

In real operation, DoorDash typically sees anomalies in several thousand segments per day. Clustering reduces that to 20 to 60 anomalous clusters per day across consumer and Dasher fraud areas, which is a volume the operations team can realistically investigate.

Step 5: Turning clusters into investigations and containment

Detection is not the finish line, it is just the trigger.

The representative anomalous segments, along with all other segments in the cluster and example events (deliveries and Dasher assignments), are accessible in a workflow tool for ops investigation.

Ops agents review example deliveries or assignments within the representative segment, looking for trends or patterns that may represent a new fraud trend.

Sometimes the pattern is non-fraudulent, like a new promotion causing a spike in refunds. Other times it is fraudulent.

When a trend is deemed fraudulent:

• it is root-caused in partnership with engineering and product teams so the root cause can be addressed

• a separate containment team runs queries to identify and stop fraudsters matching the trend pattern until product fixes land

So the system is not just detection. It’s detection wired into investigation, containment and longer-term remediation.

Results

DoorDash now uses the anomaly detection platform as its primary early-warning source for new fraudulent trends.

Key results reported by the team:

• More than 60% of all new fraud trends today are found through anomaly detection, and that share is growing as coverage expands.

• Average time-to-detect new fraud trends dropped from more than 100 days to less than three days over the past year.

• The platform saves tens of millions of dollars per year by flagging small but growing fraud trends before they get out of control.

The full scoop

To learn more about this, check DoorDash's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Following on from previous posts talking to people in the field, today I will be talking with who is a Senior Data Science Manager at Skyscanner and writer of the Senior Data Science Lead newsletter.

Senior Data Science Lead

Helping managers build world-class teams, data professionals master storytelling and guiding those looking to break into Data Science. I have built teams from scratch and lead 50+ data scientists. Now, I share my experience with you.

By Jose Parreño Garcia

We talked about his rise from data analyst to Senior DS Manager at Skyscanner, what “production-ready” really means and why the real intelligence in data science lives before and after the model.

So without further ado, let’s get into it!

Read more

Today we will look at how Dropbox does eval for its conversational AI

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Dropbox’s AI evaluation!

TL;DR

Read more

Today we will look at how Netflix uses neural networks to automate the process of flagging pixel errors

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to automation of video quality control by Netflix

Read more

Today we will look at how Uber uses agentic AI to answer financial questions.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to an actual use case of agentic AI by Uber

Read more

Today we will look at how Target used GenAI to recommend better products to users.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s see how Target leverages GenAI for better shopping experience.

TL;DR

Read more

Today we will look at how Doordash used LLM to show better results to users.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, Let’s see how Doordash leverages LLM for better search results

TL;DR

Situation

DoorDash users search with ultra-specific, complex queries (e.g. low-carb spicy chicken wrap with gluten-free tortilla), expecting accurate matches. Traditional keyword systems couldn’t handle the nuance and pure LLM-based solutions were prone to hallucination.

Task

Build a scalable, accurate and fast search system that could deeply understand both queries and items without sacrificing relevance or blowing up compute.

Action

• DoorDash built a hybrid system using:

  • LLMs (for query segmentation and classification)

  • Knowledge graphs (for structured metadata)

  • Approximate Nearest Neighbor search (for grounding output in known taxonomies)

• Added tight guardrails: constrained vocabularies, structured output prompts, and post-processing validation.

• Plugged structured query outputs into the search rankers and retrained based on improved engagement data.

Result

• ~30% increase in trigger rate for “Popular Dishes” carousel

• 2% lift in whole page relevance (WPR) for intent-heavy queries

• +1.6% WPR boost from retrained rankers due to better training signals

Use Cases

Improved search relevance, query processing at scale, ranking and personalisation

Tech Stack/Framework

LLM, knowledge graph, ANN

Explained further

At DoorDash, users don’t just type “burger” and call it a day. They type “low-carb spicy chicken wrap with gluten-free tortilla” and expect relevant suggestions. These aren’t simple lookups, they’re loaded with layered preferences, dietary restrictions and dish expectations.

So how do you build a search system that can handle that level of complexity without cracking under pressure?

The DoorDash team went with a hybrid approach: combine the brute force of traditional keyword systems with the nuance of LLMs and knowledge graphs. It’s like pairing a calculator with a philosophy major. It works and here is how.

What happens behind the search bar

At a high level, DoorDash's search engine processes two kinds of things:

• Documents - the restaurants or menu/store items

• Queries - what users type in

Each goes through its own journey. Queries get parsed, segmented, labeled and spell-checked. Documents get annotated and enriched with metadata, like vegan, spicy or chicken.

But the real power comes from understanding both deeply enough to match the right taco to the right craving.

Two worlds: documents and queries

Documents at DoorDash aren’t just raw strings, they’re enriched with metadata pulled from custom-built knowledge graphs. These graphs define relationships between things like dish type, cuisine, ingredients and dietary preferences.

Take Non-Dairy Milk & Cookies Vanilla Frozen Dessert - 8 oz as an example. It gets tagged with:

• Dietary_Preference: "Dairy-Free"

• Flavor: "Vanilla"

• Product_Category: "Ice Cream"

• Quantity: "8 oz"

On the query side, things get chunked up and mapped to similar concepts. So “small no-milk vanilla ice cream” gets split into:

[“small”, “no-milk”, “vanilla ice cream”]

Without this kind of breakdown, the search engine would either over-match (ice cream that isn’t dairy-free) or under-match (only exact string matches). With it, search becomes smarter.

But understanding concepts is only half the battle. How do you break down messy, real-world queries in the first place? That’s where things get messy and where traditional methods start to hit their limits.

Using LLMs without letting them run wild

Traditionally, query segmentation has leaned on methods like pointwise mutual information (which checks how often words co-occur compared to random chance) and n-gram analysis (which looks at fixed word sequences) to figure out which terms belong together. That works fine for clean, simple queries. But once you throw in overlapping entities or any real ambiguity, those old-school tricks start to break down fast.

Take a query like “turkey sandwich with cranberry sauce.” Is the cranberry sauce a separate item or part of the sandwich? Without context, traditional methods have no clue. They just see word chunks, not relationships.

LLMs, however, can understand the context if given the right guardrails. They will usually segment things in a way that actually makes sense, picking up on how words relate to each other across different use cases.

But here’s the catch: LLMs are prone to hallucination. So you couldn’t just throw queries at them and hope for the best. Doordash needed a controlled vocabulary to keep things grounded, something that ensured the segments were not only real but useful for the retrieval system.

Fortunately, they already had a knowledge graph in place with a solid ontology and a bunch of taxonomies to work with. So instead of letting the model come up with its own segmentations, it’s prompted to identify meaningful chunks and tag each one with a specific category from Doordash taxonomy.

For food, They have got taxonomies for cuisine types, dish types, dietary tags and more. For retail, it’s things like brand, dietary preference, product category, etc.

Back to our earlier query: “small no-milk vanilla ice cream.”

If we just asked for segments, we might get:

[“small”, “no-milk”, “vanilla ice cream”]

Not bad, but vague. Instead, the team asks the model to return a structured output mapping that relates to one of the Doordash taxonomy categories:

{

Quantity: "small",

Dietary_Preference: "no-milk",

Flavor: "vanilla",

Product_Category: "ice cream"

}

This setup gives the model extra context to work with and it leads to better results. it doesn’t just segment, it classifies. But classification isn’t the end of the road. Once you’ve got the chunks, you need to actually understand what they mean.

Entity linking: turning words into meaning

Once a query is segmented, the next step is to map each segment to a concept in the knowledge graph. For example, “no-milk” isn’t just a string match. It should link to the “dairy-free” concept even if the item description never says “no-milk” explicitly anywhere.

LLMs are handy here too but just like in segmentation, they can still hallucinate. To avoid that problem, the team constrained the model’s output using only concepts from their controlled vocabularies (i.e., taxonomy terms from the knowledge graph).

The guardrails come from a curated candidate list, pulled using approximate nearest neighbor (ANN) retrieval. ANN is a fast way to find the most similar entries in a big dataset without doing a full exhaustive search. This keeps the LLM grounded by forcing it to choose from concepts that actually exist in the system.

Take the earlier example: the segment “no-milk.” Instead of letting the LLM invent a new category like “milk-averse”, the ANN system first retrieves candidates like “dairy-free” or “vegan.” The model then just has to pick the best match based on context, keeping the output both accurate and mappable.

The method behind this is a classic retrieval-augmented generation (RAG) setup:

1. For every query and knowledge graph concept (a.k.a. candidate label), embeddings are generated either from closed-source models, pre-trained ones or DoorDash’s own.

2. Using ANN retrieval, the top 100 closest taxonomy concepts are pulled for each query. This trims the fat from the prompt and avoids blowing up the model’s context window.

3. The LLM is then prompted to link query segments to taxonomy concepts across domains like dish types, dietary tags, cuisines and more.

The retrieval payoff

Let’s go back to “small no-milk vanilla ice cream.” After segmentation and entity linking, the final structured output might look like this:

{

Dietary_Preference: "Dairy-Free",

Flavor: "Vanilla",

Product_Category: "Ice Cream"

}

Now retrieval becomes precise, it can make Dietary Preference a MUST condition (non-negotiable) and treat Flavor as a SHOULD (nice to have). It’s like giving the system a conscience, it knows what matters more.

But smart retrieval isn’t enough. You also need to double-check what the model spits out before someone eats the wrong thing.

Evaluations: catching hallucinations before users do

Accuracy is especially critical for filters like dietary restrictions. One mistake and someone could end up eating something they’re allergic to.

To keep hallucinations in check and make sure both segmentation and entity linking are solid, the team added post-processing steps to validate the model’s output. Once that’s done, they run manual audits on every batch of processed queries to spot any issues before they hit production.

Annotators go through a statistically significant sample of results and check two things:

1. Are the segments correct?

2. Are they mapped to the right concepts in the knowledge graph?

This manual review helps surface any systematic slip-ups, fine-tune prompts and improve the overall pipeline. Still, even a well-reviewed system comes with trade-offs, especially when scale and change are constants.

Memorisation vs. generalisation: the trade-off

LLMs are a solid tool for query understanding but like most things in ML, there’s a trade-off. In this case, it’s memorisation versus generalisation.

If you run LLMs in batch mode on a fixed set of queries, you’ll usually get high accuracy. That works great when the query space is small and predictable. But the further you drift into the long tail (which, let’s be honest, is where a lot of real-world queries live), things start to get tricky.

Relying purely on memorisation comes with baggage:

• Scalability: New queries show up constantly in a system like DoorDash. Preprocessing every single one just isn’t realistic.

• Maintenance overhead: Any changes to the knowledge graph or user behavior mean re-running batches, tweaking prompts and revalidating outputs.

• Stale features: Segments or links that were fine last month might be outdated now.

That’s where more generalisable approaches shine. Things like embedding-based retrieval, traditional statistical models or even smart rule-based systems can handle new queries on the fly without any preprocessing.

These methods bring a few key advantages:

• Scalability: They work on any query, even ones the system hasn’t seen before.

• Flexibility: They keep up with evolving language like new dishes, slang, weird typos

• Real-time readiness: No batch processing delays; they can act immediately.

The downside? They don’t have the same deep contextual smarts as LLMs. That can hurt precision, especially when subtle relationships between terms matter.

So DoorDash takes a hybrid approach. Instead of picking one side, they combine the strengths of both:

• LLMs for deep understanding when you can afford it

• Lightweight retrieval methods when you need scale and speed

The result: better precision where it matters and better coverage everywhere else.

But theory only gets you so far. The real test? Whether it plays nicely with the rest of the stack.

Making it all work in production

Of course, even the best query understanding system is only useful if the rest of the stack knows what to do with it. For DoorDash, that means making sure the new signals actually integrate cleanly into the search pipeline, especially with the rankers.

Rankers are the part of the system responsible for taking a big pile of candidate items or stores and deciding what order they should show up in. So once the team introduced these new, structured query signals, the rankers had to be updated to recognise and use them.

As the rankers adjusted to the new inputs and to the new patterns of user behavior that came with better retrieval, the impact showed up fast. Online metrics went up, both on the relevance side and on key business KPIs.

One feature in particular showed just how well things were clicking.

Real world example: açaí bowl, anyone?

One clear beneficiary of DoorDash’s improved query understanding system is the “Popular Dishes” carousel shown below. This component shows users a ranked list of food items when they search with clear dish intent (like “açaí bowl”).

When someone types in “açaí bowl,” they’re not looking to browse menus. They want a specific thing. The carousel lets them quickly scan options across stores, compare prices and pick what looks good. It’s fast, it’s focused and thanks to the new pipeline, it’s pulling in way more relevant results.

After rolling out the updated query understanding and retrieval system, DoorDash saw a ~30% jump in the carousel trigger rate. That means more queries are now returning results that qualify for the Popular Dishes treatment. It’s a strong signal that the system is getting closer to what users actually want.

And it’s not just about quantity, quality improved too. With better segmentation and entity linking, the system can retrieve a broader, more accurate set of items tied to the user’s intent. This showed up in their whole page relevance (WPR) metric, which measures how relevant the entire result page is from the user’s point of view. For dish-intent queries, WPR jumped by over 2%.

And because users were interacting with a wider variety of results, DoorDash was able to retrain its rankers on a more diverse engagement dataset. That led to a newer, smarter ranker version and another 1.6% bump in WPR. And because users interact more with the right stuff, the system gets better training data for the ranker, which then improves things further and becomes a virtuous cycle.

Lessons learned

• LLMs are powerful but more so with guardrails. Left to their own devices, they hallucinate. But paired with taxonomies and controlled vocabularies, they become sharp tools for segmentation and classification.

• Structured queries unlock smart retrieval. Turning messy user input into clean, labeled chunks (like "Flavor: Vanilla") makes it easier to match intent, not just strings.

• Hybrid > pure LLM. DoorDash got the best results by mixing deep LLM understanding with fast, generalizable methods like embedding retrieval.

• Better understanding improves ranking. Once structured query signals flowed into rankers, DoorDash saw real-world lifts: 30% more carousel triggers and measurable boosts to conversion and relevance.

The full scoop

To learn more about this, check Doordash's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

This article was originally published on around two weeks ago. who writes the newsletter suggested doing a guest post and I was more than happy to do it.

You should check out his newsletter if you need to stay up to date on all the latest developments in AI. He is a machine in terms of keeping track of the latest news and publishing it quickly for his subscribers. So if staying updated on all things AI is your jam, give it a go!

AI Disruption

A Beijing-based AI engineer with a decade of experience shares at least 2 daily articles on AI industry trends and technical insights.

By Meng Li

Now, let’s get to today’s deep dive on how Uber used GenAI in their invoice processing workflow.

Read more

Today we will look at how Reddit handles NSFW content at scale using deep learning

But before that, I wanted to thank those who shared Data Tinkerer with others and share an example of what you can unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Reddit’s automated content moderation

Read more

Today we will look at how Walmart autotuned its time series forecasting

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Walmart’s autotuning framework!

Read more

Today we will look at how an LLM thinks based on the research done by Anthropic.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 3 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to the inner workings of Claude’s mind

Read more

TL;DR

Read more

TL;DR

Situation

Pinterest's existing content retrieval system relied on traditional methods that couldn't fully capture the complex relationships between users and the vast array of content, leading to less personalized recommendations.

Task

Develop a scalable, embedding-based retrieval system capable of learning and representing the nuanced interactions between users and content, effectively processing Pinterest's extensive dataset.

Action

1. System Design: They designed an internal embedding-based retrieval system for organic content, utilizing advanced machine learning techniques to generate embeddings that position users and content within a shared vector space.

2. Data Processing: To train the model effectively, they processed large-scale data, extracting meaningful features from user interactions and content metadata, handling billions of data points to ensure accurate embeddings..

3. Model Training and Deployment: The model was trained on this extensive dataset, optimized for performance and relevance, and seamlessly integrated into Pinterest's infrastructure without disrupting user experience.

Result

Implementing the embedding-based retrieval system improved content relevance and user engagement on Pinterest, leading to more personalized recommendations and increased interaction rates.

Use Cases

Personalized Recommendation, Search Functionality

Tech Stack/Framework

Two-Tower Model, Approximate Nearest Neighbor, Auto Retraining

Explained Further

Content Discovery with Advanced Retrieval System

Read more

TL;DR

Read more

TL;DR

Situation

Grammarly’s 30M+ daily users rely on paid ads, but traditional attribution failed to measure YouTube’s true impact, especially with third-party cookies disappearing.

Task

Find a privacy-first way to determine if YouTube ads drive new users or just capture organic sign-ups using geo experimentation.

Action

• Geo Experimentation: Stopped ads in select regions while maintaining them in others to measure impact.

• Data Modeling: Used Google’s TBR package with a BSTS model to estimate incremental user acquisition.

• Geo Split Selection: Created balanced test/control groups using clustering and randomization.

• Power Analysis: Optimized experiment duration and sample size for accuracy and minimal business disruption.

• KPI Measurement: Focused on new active users instead of revenue, aligning with Grammarly’s freemium model.

Result

YouTube ads were most effective before peak seasons but had less impact during them. When ads were paused, new sign-ups dropped immediately, showing YouTube’s stronger mid-funnel role than expected.

Use Cases

Marketing Attribution Improvement, Media Spend Optimization, Incrementality Measurement

Tech Stack/Framework

Google TBR (Time-Based Regression), Bayesian Structural Time Series (BSTS) Model, Clustering, Randomization

Explained Further

Understanding the Attribution Challenge

Measuring marketing effectiveness is challenging, especially for ads that don’t involve direct clicks. It’s easy to track whether someone clicks on a Google search ad, but what about an ad they only view on YouTube? Traditional attribution models often struggle to assign proper credit to top-of-funnel marketing, leading to an undervaluation of video ads and TV campaigns.

Read more

TL;DR

Read more

TL;DR

Read more

TL;DR

Read more