Today we will look at how Lyft cut translation latency from days to minutes using AI

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to how Lyft turned localisation into a 30-minute AI workflow!

TL;DR

Situation

Lyft’s human-led localisation was high quality but too slow for Québec’s Bill 96 launch and European expansion.

Task

The team needed translations in minutes across 11 locales and 150+ services without losing terminology, tone, compliance or placeholders.

Action

Lyft built an LLM Drafter/Evaluator loop with retries, context injection, deterministic guardrails, prompt versioning and locale-specific tuning.

Result

Translation latency dropped from days to minutes, with a 30-minute SLA target for 95% of batch translations and 95% needing no major human edits.

Use Cases

translation quality evaluation, batch translation, AI-assisted localisation

Tech Stack/Framework

LLMs, Translation Management Systems, Pydantic, prompt engineering

Explained further

Context

For years, Lyft’s localisation setup was built around human translation. Human linguists usually meant strong quality, consistent terminology and fewer awkward mistakes in the app. When Lyft only supported a small number of languages like Spanish, Portuguese and French, multi-day turnaround times and translation costs that grew with every new language were manageable.

Then the business need changed.

Lyft’s Québec launch created a different level of pressure. Bill 96 required French-first user experiences, which meant localisation could not sit in a multi-day queue while product teams waited around. At the same time, Lyft Urban Solutions, the Bikes & Scooters division, wanted to expand into European markets and needed six new languages.

The old workflow was still good at quality, but it was too slow for the pace of the business. So Lyft rebuilt its translation pipeline around AI, while keeping linguists in the loop for review and final quality control. The goal was to reduce translation latency from days to minutes without turning product copy into awkward machine-translated text.

This article breaks down how Lyft did it: the architecture, the Drafter/Evaluator loop, context injection, deterministic guardrails, prompt versioning and locale-specific tuning.

Inside Lyft’s translation pipeline

Before getting into the LLM workflow, it helps to understand the scale of Lyft’s localisation platform.

Lyft’s platform now serves 11 locales across more than 150 services. That means localisation is not just a small content operation sitting next to the app. It is part of the product infrastructure. Translated strings have to flow reliably across many teams, services and surfaces.

The batch translation pipeline handles 99% of Lyft’s app and web content and targets a 30-minute SLA for 95% of translations. Lyft also supports real-time translation, such as ride chat, though that uses a different architecture and is not covered here.

At the centre of the batch pipeline is a dual-path design.

When requesters submit source strings, the pipeline sends them to Smartling, Lyft’s Translation Management System (TMS), for human oversight. At the same time, the same strings go to LLM workers for rapid draft generation.

That split is important because the LLM path gives product teams translated strings quickly enough to unblock launches but the TMS path keeps human review as the quality control layer and system of record. AI translations can ship early, then linguist-reviewed versions replace them once approved.

The batch pipeline has three main phases:

1- Drafting: where translation starts

Requesters submit source strings with context. This includes where the text appears in the UI, what the text is trying to communicate and what tone is expected. This context matters because short UI strings are often ambiguous.

‘Start ride’ is not just two words. It could be a button. It could be an instruction. It could refer to a driver, a rider, a scooter or a bike. Without context, the translation system is guessing.

The Drafter uses that context to generate translation candidates.

2- Unblocking launches: shipping the best draft fast

The Evaluator reviews the candidates and chooses the best one, or rejects all of them.

The selected translation can then be shipped within minutes, which allows product teams to keep moving instead of waiting days for a human translation cycle.

This early-release version is not treated as the final truth. It is a fast, quality-checked draft.

3- Finalisation: human approval and final quality control

Professional linguists review the translations inside the Translation Management System, or TMS. Once approved, the linguist-reviewed version replaces the early-release version and becomes the system of record. If a translation is flagged, it is corrected before distribution.

This dual-path setup is the key design choice.

The LLM path gives speed. The TMS path gives quality control and governance. The AI output can unblock launches, but the human-reviewed version still wins in the end.

Why Lyft chose LLMs over traditional translation?

Lyft started by evaluating traditional Neural Machine Translation providers. These systems were fast, but they often struggled with Lyft-specific terminology and product context.

That is a real problem in localisation because generic correctness is rarely enough. A translation can be grammatically fine while still being wrong for the product. It might ignore official terminology, mishandle regional phrasing, change the tone or translate something that should have stayed in English.

The Lyft team moved toward LLMs because they are better at using context. Traditional NMT services often process each string in isolation. LLMs can take in extra metadata, such as where the string appears, what the intended tone is, what glossary terms apply and which placeholders need to survive untouched.

That extra context is especially useful for product UI.

Take this string:

"Your {vehicle_type} is arriving in {eta_minutes} minutes"

A human immediately sees a few things going on. The tone should probably be clear and calm. It likely appears in a time-sensitive user experience. It needs to sound natural but not too casual. And it may need region-specific wording.

LLMs can work with that additional context.

They can also evaluate translations, not just generate them. That matters because translation is rarely a single-answer problem. Multiple versions can be valid, but one may be clearer, more natural or more aligned with Lyft’s terminology.

This led to Lyft’s core architecture: one model drafts, another model evaluates, and the drafter refines based on feedback when needed.

Building a translation review loop

A simple machine translation pipeline would make one API call: send source text, receive translated text, ship it. But this would be an issue at scale for several reasons:

First, it has no nuance. The output may be technically correct, but not necessarily right for the brand, region or UI.

Second, it has no quality signal. There is no built-in way to know whether the translation is good enough before users see it.

Third, it has no recovery path. If the translation breaks a placeholder, ignores a glossary term or changes the meaning, the system has no structured way to repair it.

Lyft needed something closer to a human review loop: generate a few options, critique them, refine the result and only release a candidate that passes checks.

That became the Drafter and Evaluator pattern.

The “Drafter”: generating translation options

The Drafter’s job is to produce translation candidates. Lyft configures it to produce three distinct candidates for each source string. That is a simple but important design choice.

One translation often converges on the safest or most likely wording. That can be fine, but it may not be the best fit for the specific product context. By asking for three candidates, the system increases the chance that one version handles tone, terminology and regional phrasing better than the others.

For the Drafter, Lyft uses a fast, non-reasoning model. Translation generation is mostly a generative task, and standard models already perform well enough for this part of the workflow.

Speed matters because the system may need to retry. Cost matters because localisation touches a lot of strings. Using a frontier reasoning model for every draft would be overkill.

A simplified Drafter prompt looks like this:

DRAFTER_PROMPT = """
You are a professional translator for Lyft.
Translate into {language} for {country}.
Give {num_translations} translations of the following text.

GLOSSARY: {glossary}
PLACEHOLDERS (preserve exactly): {placeholders}

Text: {source_text}
"""

For example:

source_text = "Your {vehicle_type} is arriving in {eta_minutes} minutes"
language = "French"
country = "Canada"

The Drafter might return:

DrafterOutput(
    candidates=[
        TranslationCandidate(text="Votre {vehicle_type} arrive dans {eta_minutes} minutes"),
        TranslationCandidate(text="Votre {vehicle_type} sera là dans {eta_minutes} minutes"),
        TranslationCandidate(text="Votre {vehicle_type} arrivera d'ici {eta_minutes} minutes"),
    ]
)

That gives the pipeline clear contracts between components. The Drafter returns a known object shape. The Evaluator receives candidates in a predictable format. Parsing is less brittle.

The “Evaluator”: choosing the best translation

The Evaluator acts as a quality gate. It receives all candidates from the Drafter, checks them against a rubric and then either selects the best candidate or rejects the batch.

Lyft uses a reasoning-focused model for this step because evaluation requires comparison and judgment. The model has to check whether the source meaning is preserved, whether glossary terms are followed, whether the tone fits and whether placeholders survived the trip.

The Evaluator grades each candidate across four dimensions.

Accuracy and clarity: Does the translation preserve the full meaning of the source? Is it clear?

Fluency & adaptation: Does it read naturally to a native speaker? Is it appropriate for the target region?

Brand alignment: Does it use official Lyft terminology? Are brand names, proper nouns and airport codes preserved correctly?

Technical correctness: Are spelling, grammar, placeholders and required terms handled properly?

Each candidate receives a grade: pass or revise. If any candidate passes, the Evaluator picks the best one. If all candidates fail, it explains why.

Example output:

EvaluatorOutput(
    evaluations=[
        CandidateEvaluation(
            candidate_index=0,
            grade=Grade.PASS, 
            explanation="Accurate, natural phrasing."
        ),
        CandidateEvaluation(
            candidate_index=1,
            grade=Grade.PASS,
            explanation="Natural and conversational."
        ),
        CandidateEvaluation(
            candidate_index=2,
            grade=Grade.REVISE,
            explanation="'d'ici' implies uncertainty, inappropriate for ETA."
        ),
    ],
    best_candidate_index=0,
)

That last point is a good example of why evaluation matters. The third candidate may look reasonable at first glance, but the phrase introduces uncertainty that does not belong in an ETA message. It is the sort of subtle issue a simple string-by-string translation system can miss.

Why Separate Drafter and Evaluator?

Separating the Drafter and Evaluator gives Lyft several advantages.

The first is practical: spotting errors is often easier than producing the ideal translation on the first attempt. The Evaluator does not need to be the world’s greatest translator. It needs to be a strong reviewer with a clear rubric.

The second is bias reduction. A model judging its own work can be too forgiving. It already has the context of why it chose a phrase, which can make it less likely to challenge the result.

The third is cost control. Lyft can use a faster model for drafting and a more capable reasoning model for evaluation. This keeps the expensive judgment step focused where it matters.

How failed translations get a second (or third) chance

When the Evaluator rejects every candidate, the pipeline does not immediately give up. It captures the critique and feeds it into the Drafter’s next attempt.

Lyft limits this loop to three attempts. The team found that most of the gains come from the first one or two cycles, so three attempts create a useful balance between quality, latency and cost.

A retry critique might look like this:

critique_for_retry = """
All candidates failed glossary compliance. Key issues:
- "Ride" must be translated as "trajet" per Lyft Quebec glossary
- Do not use "course" which is European French
"""

The next Drafter prompt explicitly tells the model what failed and what needs to be fixed: glossary usage, placeholders, tone or formatting.

This feedback loop produces a success rate of more than 95% across most languages. For cases where the Drafter and Evaluator still cannot agree after three tries, the string falls back to the human workflow. Product teams can wait for linguist review in the TMS or request an expedited translation for urgent launches.

Giving the model Lyft’s translation rules

LLMs do not automatically know Lyft’s internal terminology rules. They do not know that {driver_name} is a code variable that must be preserved. They do not know that a certain term should use Lyft’s official translation in Québec French. They do not know which product names should stay in English.

So Lyft injects context into the prompts. The pipeline uses glossary terms, do-not-translate lists and reference data from the TMS.

Glossary terms include official translations for individual words and phrases. For example, “Driver” may need to be translated as “Chauffeur” in French, depending on Lyft’s approved terminology.

Do-not-translate lists cover brand names, product names and proper nouns that should remain in English.

These assets are maintained by linguists and periodically pulled into the Translation Pipeline. It gives both the Drafter and Evaluator the production-specific information they need.

Where rules matter more than prompts

Prompts can reduce errors but they cannot fully prevent them.

LLMs are probabilistic. They may translate placeholders, remove URLs, alter formatting or corrupt structured text. For production localisation, that is a serious problem.

Lyft observed several recurring issues. Placeholder translation was the most common failure mode. For example, {driver_name} could become {nom_du_conducteur}.

Lyft’s guardrail system works in two phases: pre-translation extraction and post-translation validation.

Pre-Translation: masking what must not change

Before content reaches the LLM, the pipeline extracts elements that must be preserved exactly. This includes variables and format strings such as {driver_name}, {eta}, %s, %1$s and %@.

It also includes URLs, email addresses, region codes, HTML tags and escape sequences such as \n, \t and \r. These elements are replaced with numbered tokens like __PH_0__, __PH_1__ and __PH_2__.

Example:

Original:
"Hey {first_name}! Your Lyft arrives at {eta}.\nTrack: https://lyft.com/r/abc"

Masked:
"Hey __PH_0__! Your Lyft arrives at __PH_1__.__PH_2__Track: __PH_3__"

Mapping:
    __PH_0__ → {first_name}
    __PH_1__ → {eta}
    __PH_2__ → \n
    __PH_3__ → https://lyft.com/r/abc

The mapping is included in the prompt so the LLM understands what each token represents and what needs to be preserved.

The numbering also handles reordering. Different languages may place variables in different parts of the sentence, and the token mapping allows that movement without losing the original placeholder.

Post-Translation: validating before release

After the LLM returns a translation, the pipeline validates it before accepting the result. Validation checks that every expected token appears exactly once, no unexpected tokens were introduced and structured content such as HTML tags remains balanced.

When validation fails, the exact error is passed into a retry prompt. For example, the model might be told that __PH_2__ is missing and that it represents \n. That gives the next attempt a concrete repair instruction rather than a vague ‘please do better’, which, to be fair, is a management style but not a validation strategy.

Once validation passes, the tokens are restored:

Translated:
"Salut __PH_0__! Votre Lyft arrive à __PH_1__.__PH_2__Suivre: __PH_3__"

Restored:
"Salut {first_name}! Votre Lyft arrive à {eta}.\nSuivre: https://lyft.com/r/abc"

Treating prompts like production code

In early development, Lyft treated prompts like configuration. The team tweaked wording, tested manually and deployed changes with limited review.

That created a problem. A small prompt change in the Evaluator could increase false rejections, and without proper testing it was difficult to detect the regression or roll back safely.

Lyft now treats prompts as production code.

Prompt templates live alongside the Translation Pipeline code. They go through review. Each prompt version includes a changelog that explains what changed and why. Prompt and model changes are tested against an evaluation suite before receiving production traffic.

The team evaluates changes against ground truth translations in the TMS. Divergences from linguist-approved versions are flagged and reviewed before rollout.

This is a useful lesson for LLM systems generally. Prompts influence system behaviour, so they deserve the same discipline as code: version control, review, testing and rollback.

Testing models without locking into one provider

Lyft does not commit the pipeline to a single model provider. The LLM layer is abstracted, and experimentation happens through Lyft’s configuration infrastructure.

That setup supports traffic splitting, so new models can receive 5%, 20% or 50% of requests. It supports shadow mode, where a candidate model runs in parallel without affecting production. It also supports per-locale overrides and instant rollback without a deploy.

Example config:

drafter:
  model: <fast-generation-model>
  fallback: <fallback-model>
  prompt: <prompt-id>
  shadow:
    model: <candidate-model>
    prompt: <prompt-id>
    traffic_percent: 10

evaluator:
  model: <reasoning-model>
  fallback: <fallback-model>
  prompt: <prompt-id>
  reasoning_effort: medium

locale_overrides:
  <locale_code>:
    evaluator:
      reasoning_effort: high

Through experimentation, Lyft found that faster and cheaper models perform comparably to frontier models for initial generation, while reasoning-focused models do much better at catching subtle evaluation errors.

Tuning prompts to each locale

Different locales need different handling.

Lyft saw this clearly with English variants such as en-GB and en-CA. These variants often require spelling, punctuation and vocabulary adjustments rather than full translation.

For example:

color → colour
trunk → boot

Without clear constraints, the model can overdo it. A simple ‘Try again’ button can become ‘Have another go’. That may be valid British English, but it changes the product voice. It may feel too playful or inconsistent with Lyft’s product voice.

To prevent this, Lyft uses locale-specific overrides that narrow the scope of transformation:

LOCALE_OVERRIDE_EN_GB = """
You are adapting American English text for British English speakers.

IMPORTANT: This is an ORTHOGRAPHIC adaptation, not a full translation.

Only change:
- Spelling (color → colour, center → centre, organize → organise)
- Punctuation conventions where required

DO NOT change:
- Tone or voice
- Sentence structure  
- Casual vs. formal register
- Idioms or expressions (unless they are specifically American and would confuse UK readers)

The goal is that a British reader sees familiar spelling, not that the text "sounds British."
"""

This reduced over-adaptation while keeping Lyft’s brand voice consistent across locales.

What Lyft’s localisation pipeline gets right

Lyft’s localisation pipeline shows how LLMs can be useful in production when the system is designed around their strengths and weaknesses.

The models generate candidates quickly. A separate evaluator reviews them against a rubric. Retry loops repair failures. Context injection keeps terminology aligned with Lyft’s glossary. Deterministic guardrails protect placeholders, URLs, formatting and structured content. Human linguists still review the final output inside the TMS.

That combination reduced translation latency from days to minutes while keeping quality control in place.

The most interesting part is the balance. Lyft uses AI to remove the slowest parts of the workflow, while still relying on deterministic checks and human review where they matter most. The system does not ask the model to be perfect. It gives the model a well-defined job, checks the output and escalates hard cases.

Through this setup, Lyft sees 95% of translations needing no significant changes after linguist review. The remaining 5% tend to be the hard cases: regional idioms, legal language and brand voice decisions.

That feels like the right division of labour. Let the machine handle the repeatable work at speed. Let deterministic systems catch mechanical failures. Let humans focus on the judgment calls where context, nuance and risk actually matter.

Also, keep the placeholders alive. Somewhere out there, {driver_name} is depending on it.

The full scoop

To learn more about this, check Lyft's Engineering Blog post on this topic.

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Pinterest used multimodal AI to organise products into relevant shopping collections

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to PinLanding: Pinterest’s AI system for shopping collections

Situation

Traditional shopping collections built from search logs and manual curation were not enough to cover the growing mix of detailed, compositional and AI-native shopping queries.

Task

Build a scalable system that could generate precise, searchable shopping collections directly from product content while still aligning with real user shopping intent.

Action

Pinterest built PinLanding, a multimodal pipeline that used search behavior to identify intent gaps, vision-language models to extract product attributes, clustering and LLM-as-judge to clean and validate a shopping vocabulary, a CLIP-style model to assign curated attributes at scale and Ray plus Spark to run large-scale inference and feed construction.

Result

The system generated 4.2 million shopping landing pages, expanded unique topics by 4x, improved average Precision@10 from 0.84 to 0.96 and delivered a 35% lift in search performance.

Use Cases

Search relevance improvement, attribute tagging, catalog enrichment, cold-start collection generation

Tech Stack/Framework

Apache Spark, Ray, PyArrow, CLIP, VLM

Explained further

Context

Online retailers and social platforms now manage catalogs with billions of items. Pinterest is one example, but the underlying problem is much broader: how do you organize massive product inventories into shopping collections that people can actually browse?

Pinterest has been dealing with exactly that. Historically, shopping collections were mostly shaped by user search history and manual curation. That worked well enough when search behavior was more predictable and the long tail was smaller. It works a lot less well when people start searching in full sentences, asking for vibes, aesthetics and combinations of constraints all at once.

That shift is what makes PinLanding interesting.

Instead of waiting for search logs and human curation to define the collection space, Pinterest’s team flips the process around. The system starts from the product content itself, then builds shopping collections from that foundation while still staying grounded in how people actually search. In other words, it is content-first, but not content-only.

The system is built around four components: understanding user search patterns, building and validating a shopping collection vocabulary using multimodal LLMs and LLM-as-judge, constructing feeds from attributes and evaluating the system while adapting to AI-native search behavior.

1. Mapping what shoppers actually want

The system starts with something fairly practical: understanding what users are actually trying to do.

Pinterest aggregates signals from search history, autocomplete interactions, filter usage and browse paths to estimate the distribution of shopping intents across the catalog. The goal here is not to build a giant list of every possible topic. It is to get a grounded picture of current demand, current coverage and where the system is falling short.

Two clear patterns show up.

The first is the head of the distribution: high-volume, well-formed shopping queries like ‘black cocktail dress’ or ‘white linen pants.’ These are the kinds of searches traditional collection systems already handle reasonably well. They are structured, frequent and easy to tie back to existing ranking and merchandising logic.

The second is where things get more interesting. There is a growing long tail of conversational and compositional queries, especially as users get more comfortable interacting with AI systems. These are requests that blend style, occasion, constraints and subjective framing like ‘what to wear for Italian summer vacation’ or ‘long red satin dress with lace trim under $200’. They are describing a scenario or aesthetic.

That distinction matters because it reveals the limitations of a purely query-driven collection pipeline. The head is manageable. The tail is messy, sparse and expanding.

Pinterest uses these behavioral signals in three ways. First, they highlight product spaces where user demand is strong but collection coverage is still thin. Second, they reveal which attribute dimensions users care about, including things like color, occasion, style, fit, price and brand, across 20 categories. Third, they create the baseline against which the rest of the pipeline is evaluated.

That last point is important. PinLanding is not trying to throw away query understanding. It is trying to improve topical coverage and precision relative to the old query-driven baseline. Search behavior still matters. The difference is that the system no longer depends on search logs alone to decide what collections can exist.

2. From raw product content to searchable shopping topics

Once Pinterest understands the behavior surface, the next step is to describe each product in a structured way that is useful for collection generation.

Each product is modeled as a multimodal tuple made up of an image plus metadata like title, description, merchant tags and price. A vision-language model (VLM) is then used to generate candidate attributes for that product. Rather than producing free-form descriptions, the model is prompted to return normalized key-value pairs. That design choice makes downstream processing far easier because it turns messy visual and textual information into something closer to structured data.

The initial output is broad and useful in one sense because it has high recall. It captures lots of possible attributes. The problem is that it is too messy to use directly.

The raw VLM output tends to overproduce highly specific descriptors like ‘black insoles’ or ‘lace-trim hem with side slit.’ It also produces near-duplicate variants such as ‘boho,’ ‘bohemian’ and ‘boho-chic’ for what is basically the same shopping concept. That creates a sparse attribute space where too many labels apply to too few products. Once that happens, collection quality starts to fall apart. You end up with lots of tiny fragments instead of reusable shopping topics.

Pinterest addresses this with a curation pipeline designed to build a compact and reusable attribute vocabulary.

The first step is frequency filtering. Attributes that appear only on a tiny number of products are rarely useful as collection keys, so they are removed. This strips out many ultra-specific descriptors while keeping the attributes that are more likely to reflect reusable shopping concepts.

The second step is embedding-based clustering. Dense text embeddings are generated for the remaining attributes and highly similar attributes are merged. When multiple variants express the same idea, the more frequent surface form becomes the canonical one.

The final step uses LLM-as-judge to score candidate topic-query pairs. Given an attribute tuple and its generated query, a second LLM evaluates whether the pair is semantically coherent, whether it sounds like a plausible shopping intent and whether it matches typical search phrasing. This helps rank and filter candidate topics not just for internal consistency but for searchability.

That last part is the key difference between extracting attributes and building shopping topics. Attributes by themselves are too granular and too raw. The curation pipeline turns them into a vocabulary that is compact enough to scale and natural enough to match how people actually search.

2.1 Scaling attribute assignment with a CLIP-style model

Once the curated attribute vocabulary exists, the next question is how to assign those attributes across the full product catalog.

Running the original vision-language model on every product at production scale would be expensive and brittle. So Pinterest trains a dual-encoder model inspired by CLIP instead.

One encoder takes product image and text as input and produces a product embedding. The other encoder takes an attribute phrase and produces an attribute embedding in the same vector space. During training, product-attribute pairs generated from the VLM are treated as positive examples, while non-matching pairs act as negatives.

The training objective is bidirectional contrastive loss. Matching product-attribute pairs are pulled closer together in embedding space and mismatched pairs are pushed apart.

At inference time, the system embeds all products and all attributes once. An attribute is assigned to a product when the similarity between the two embeddings crosses a calibrated threshold. That threshold can also be adjusted with frequency-based weighting to counter long-tail smoothing.

This architecture solves a few problems at once.

It is cheaper than running the VLM everywhere. It creates a denser and more consistent attribute graph. And it reduces the explosion of distinct attributes that came from raw model outputs.

Interestingly, they note that the resulting system ends up with fewer distinct attributes overall, while increasing the average number of attributes per product. That is a useful tradeoff. Fewer labels does not mean less information here. It means the information is compressed into a vocabulary that is actually reusable.

That dense attribute graph becomes the foundation for everything that follows.

3. Building shopping feeds at catalog scale

Once products have been tagged with curated attributes, Pinterest needs to construct feeds at catalog scale. That means dealing with millions of Pins and millions of possible topics.

That is where Ray and large-scale batch inference come in.

Attribute inference runs as a Ray streaming job with three main stages.

The first stage handles data loading and preprocessing. Product images and metadata are downloaded, tokenized and serialized into PyArrow tables, then sharded across a CPU cluster.

The second stage handles ML inference. Ray schedules preprocessed batches onto a GPU pool where the CLIP-based classifier performs forward passes and produces attribute scores.

Because execution is streamed, data loading, preprocessing and inference can overlap instead of waiting on one another. The use of heterogeneous clusters also allows CPU-heavy preprocessing and GPU-heavy inference to scale independently. The training and inference pipeline for the classifier completes in roughly 12 hours on 8 NVIDIA A100 GPUs at an estimated cost of about $500 per training run. For a system operating at this scale, that is pretty reasonable.

Once attribute assignments are available, feed construction moves to a matching layer built around ANN-style and strict attribute matching via Spark.

Each shopping topic is defined as an attribute tuple. For example: category equals dress, color equals yellow, season equals summer, occasion equals party.

Spark is then used to compute relevance scores between topics and products. The scoring aggregates shared attributes between topic and product, weighted by attribute-level confidence.

To keep this tractable, candidate joins are pruned through attribute-based partitioning and minimum-overlap prefilters.

This part is easy to underestimate. Generating a topic vocabulary is one thing. Turning that vocabulary into feeds across millions of products is where many systems fall apart. Pinterest’s design works because the modeling and infrastructure choices line up. The attribute space is structured enough to match efficiently and the batch inference stack is built to operate at the right scale.

4. Measuring whether the system actually works

Pinterest evaluates the system at both the attribute level and the collection level.

For attribute quality, the CLIP-based model is tested on Fashion200K, a standard benchmark for fashion attribute prediction. The reported result is 99.7 percent Recall@10, which substantially exceeds prior methods that sit in the 50 percent range on the same metric. That suggests the model has learned a strong mapping between product imagery and fashion attributes.

Pinterest also looks at the distribution of generated attributes. Compared with raw GPT-4-V outputs, the CLIP-based system produces a much more usable attribute space. There are fewer extremely rare attributes and more attributes that consistently apply across many products.

That is important because quality here is not only about benchmark performance. It is also about whether the resulting label space can support downstream shopping tasks without collapsing into noise.

For collection quality, human raters compare feeds produced by the content-first pipeline against a traditional search-log-derived baseline. The evaluation uses Precision@10, measured as the fraction of the top ten products in a collection that match the collection’s title attributes.

Across attribute families including color, main material, fit and stretch, shape, style, season and festival, occasion and brand, the new system improves average Precision@10 from 0.84 to 0.96. Several categories, including style and brand, reach 1.00.

That is a meaningful jump. It suggests the collections are not just broader in coverage, but also more precise in the items they surface.

In production, the pipeline generates 4.2 million shopping landing pages. That represents a fourfold increase in unique topics relative to the previous search-log-based approach and leads to a 35% improvement in search performance.

That combination is what makes PinLanding interesting because it is not only producing cleaner labels in an offline benchmark but also expanding coverage and improving relevance in production.

Summary

PinLanding is a strong example of what multimodal AI looks like when it is pushed into production with actual constraints.

The system does not rely on a single model to do everything. Instead, Pinterest breaks the problem into stages: measure intent, generate structured attributes, compress them into a usable vocabulary, scale assignment with a dual encoder and then construct collections with distributed matching.

That decomposition is probably the most useful lesson here.

There are plenty of teams experimenting with multimodal models for product understanding. The harder part is turning those outputs into something dense, normalized and operationally stable enough to support search and discovery at web scale. Pinterest’s team shows a fairly clear path for doing that.

The broader shift is also worth noting. Search-log-derived collections are not going away, but they are no longer enough on their own. As people search in more conversational, compositional and trend-driven ways, systems need to infer collections from the content itself while still staying anchored to user intent.

That is the balance PinLanding is aiming for.

And for platforms sitting on billions of products, that balance is probably becoming less of a nice-to-have and more of a survival requirement.

The full scoop

To learn more about this, check Pinterest's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Today we will look at how Shopify scales its product categorisation using agentic AI

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Shopify’s multi-agent taxonomy

TL;DR

Situation

Shopify’s product classification system makes tens of millions of predictions daily, across a taxonomy with 10,000+ categories and 2,000+ attributes. Commerce changes fast, the taxonomy has to keep up or the whole stack starts drifting.

Task

Keep the taxonomy current at scale without relying on slow, reactive, manual curation. Fix volume, expertise and consistency problems before they hit merchants, customers and model quality.

Action

Built an AI multi-agent system: structural analysis + product-driven analysis, then intelligent synthesis. Added equivalence detection (category = broader category + attribute filters) plus automated QA via domain-specific AI judges.

Result

Taxonomy branches can be analyzed in parallel: hundreds of categories instead of a few per day. Quality improved via grounded merchant data + structural consistency, with judges filtering proposals (example: “MagSafe compatible” approved at 93% confidence).

Use Cases

Category discovery, attribute gap detection, taxonomy maintenance, search and filtering improvement

Tech Stack/Framework

AI agent, equivalence detection, multi-agent system

Explained further

Context

Last year, over 875 million people bought items from Shopify merchants. Shopify already runs a product classification system that makes tens of millions of predictions daily with a high degree of accuracy.

But classification is the easy part compared to the thing underneath it: taxonomy. Because the model doesn’t just need to be right, it also needs a clean, consistent set of labels to be right about.

That’s the challenge for Shopify: once you have 10,000+ categories and 2,000+ attributes, the taxonomy becomes its own product with its own failure modes. It can get stale. It can get inconsistent. It can drift away from how merchants actually describe products. And when that happens, the classifier quality takes the blame for what is basically a taxonomy debt problem.

So this post is about what Shopify did next: they built an AI multi-agent system that doesn’t just classify products, it actively improves the taxonomy labels themselves so the system stays agile as commerce changes.

The challenge: scaling taxonomy without losing accuracy

A taxonomy is a contract between three groups that rarely agree:

• Merchants describing products the way they think about them

• Customers searching and filtering with their own mental model

• Platform systems trying to enforce structure so everything stays queryable and comparable

Now add the reality that commerce never sits still. New products appear. Old categories split. Entire verticals get reshaped by trends, tech and regulation. The taxonomy has to keep up or the platform drifts away from how people actually shop and sell.

Shopify frames the challenge as three problems.

The volume problem: manual updates can’t keep up

A global product taxonomy needs constant attention. Every new product type, emerging technology category and seasonal trend potentially triggers taxonomy updates.

Manual curation becomes a bottleneck because taxonomy work is not one change. It is usually a bundle: a category addition, a hierarchy decision, a set of attributes, naming alignment and a check for duplicates or conflicts.

For example, consider the emergence of categories like smart home devices or remote work equipment. Each category represents not just new categories but also entirely new attribute sets.

Smart home devices for instance need connectivity types, power requirements and compatibility. Those are specs that did not exist in the taxonomy before.

So the work isn’t a one-off. It’s continuous expansion and adjustment across a giant tree of concepts.

The expertise problem: every vertical has its own rules

Good taxonomy design is domain-heavy. You do not get it right by being generally smart. You get it right by knowing what matters in that product world. For example, there are nuanced differences between types of guitar pickups or appropriate attributes for skincare products.

A taxonomy team can’t realistically maintain deep expertise across every vertical that merchants sell into. But if the taxonomy is inconsistent or poorly structured, merchants pay for it through reduced discoverability, suboptimal search results and ineffective filters for customers.

The consistency problem: one concept, five different labels

As the taxonomy grows organically, inconsistencies creep in:

• similar concepts represented differently across categories

• naming conventions inconsistent

• discrepancies between merchant categorization and customer expectations

Those inconsistencies compound. Merchants get confused when listing. Customers get frustrated when filtering and comparing. And the classifier quality drops because labels stop being reliably meaningful across the tree.

This is the part most teams underestimate. In a taxonomy, small inconsistencies behave like small data quality issues: they don’t stay small.

From manual taxonomy work to agent-led evolution

Shopify’s taxonomy management evolved from a manual workflow into an AI-driven system.

The old way: Expert review, slow throughput

The traditional pattern is familiar:

1. domain experts analyze product data

2. identify gaps or inconsistencies

3. propose changes

4. implement changes via careful review

It ensures quality but it also creates bottlenecks.

The biggest problem was the reactive nature of it: Shopify would only recognize the need for new categories or attributes after merchants began listing products that didn’t fit. By then, the system had already missed chances to give merchants and customers a better experience.

So even when you do great manual work, you’re always late.

The breakthrough: Two lenses, one system

Advanced language models opened a door: not to replace human experts, but to augment them with scale and consistency.

The key insight was that taxonomy improvement comes from two different angles:

• structural analysis: the logical structure of the taxonomy, gaps in hierarchies, missing relationships

• product-driven analysis: what real product data says merchants actually sell and how they describe it

Each angle catches different issues. Shopify’s breakthrough was combining them into a system that can continuously propose improvements then filter them through quality checks before human review.

Inside the system: How the agents work

The new architecture rests on three principles:

• specialized analysis

• intelligent coordination

• quality assurance

And the intent is clear: continuous evolution, not one-time taxonomy construction.

What’s different: continuous evolution, not one-time creation

AI’s been used for product categorisation and one-off taxonomy builds for a while. The difference here is instead of building it once and hoping it holds, Shopify uses specialised AI agents to keep the taxonomy evolving continuously. There are 3 core components to this approach:

1- Real product grounding: The system integrates actual merchant product data so proposals reflect how merchants describe and categorize products. This keeps decisions grounded in commerce reality rather than only theory.

In other words: if merchants are consistently describing a differentiator, it probably belongs in the taxonomy, even if it offends someone’s idea of a “pure” category tree.

2- Multi-agent specialization: Multiple specialized agents run different analyses. One focuses on structural consistency. Another focuses on product-driven insights. Then those outputs are synthesized. The claim here is that the combination finds improvements that neither agent would find alone.

That makes sense structurally. Taxonomy is both a graph problem and a language problem.

3- Sophisticated equivalence discovery: This is the most interesting component. detecting equivalence relationships where a specific category equals a broader category filtered by attribute values.

This matters because merchants should be able to organize their catalogs however they want, while the platform still understands what products ‘mean’ underneath the merchant’s choices.

So instead of forcing everyone into one rigid structure, Shopify tries to learn mappings that preserve flexibility and still support search, recommendations, and analytics.

Architecture flow

The AI agent workflow works like this:

• enable agents to explore the taxonomy

• run multi-stage analysis (structural + product-driven)

• synthesize and resolve conflicts

• detect equivalences

• run automated QA using judges

• send refined proposals to humans

• update the taxonomy in production

Enabling agent-taxonomy interaction

Before agents can improve anything, they need to ‘read’ the taxonomy like a human would.

Shopify implemented a system that allows agents to:

• search for related categories

• examine hierarchical relationships

• verify whether proposed changes conflict with existing elements

A good example: an agent analyzing guitar-related categories can explore the full musical instruments hierarchy, inspect related attributes across instruments and look for patterns that suggest better structure.

In other words, the agent doesn’t just look at one node. It roams the neighborhood.

The pipeline: specialised agents, staged decisions

For the AI Agent to be work properly, different specialised agents are at work to provide specific insights:

Structural analysis: This agent looks at the taxonomy itself for logical consistency, completeness, gaps in category hierarchies, naming convention inconsistencies and opportunities to reorganize related concepts.

It operates purely on the taxonomy structure and aims to keep the whole thing coherent.

Product-driven analysis: This agent integrates real merchant data and examines how products are described and categorized on the platform.

Specifically, it looks at patterns in product titles, product descriptions and merchant-defined categories. The goal is to find gaps between how merchants think about products and how the taxonomy represents them.

This is an important distinction. A taxonomy can be structurally perfect and still be useless if it doesn’t match merchant reality.

Intelligent synthesis: Now we have two streams of recommendations:

• structure-driven improvements

• product-driven improvements

They can conflict. They can overlap. They can propose redundant changes.

The synthesis step merges insights, resolves conflicts, and eliminates redundancies. And sometimes the best answer is not pick one, it’s combine both.

Equivalence detection: This agent solves a practical commerce problem: merchants want flexibility but platform systems need consistency.

Consider golf shoes:

• Merchant A uses a specific ‘Golf Shoes’ category

• Merchant B uses ‘Athletic Shoes’ with an ‘Activity Type = Golf attribute

Both are valid for the merchant. But search, recommendations and analytics benefit from understanding these represent the same product set.

So the system detects attribute-based equivalences of the form:

specific category = broader category + one or more attribute filters

This lets merchants organize however makes sense for their business while keeping platform intelligence consistent across different catalog structures.

If you’ve ever tried to do cross-merchant analytics at scale, you can probably feel why Shopify cared enough to build an entire agent for this.

Automated QA: judges before humans

After proposals are generated, Shopify adds automated QA through specialized AI judges.

These judges evaluate proposed changes using reasoning capabilities and taxonomy design principles to filter and refine suggestions before human review.

The important detail is that evaluation differs by change type:

• adding new attributes

• creating category hierarchies

• modifying existing structures

Different changes require different criteria, so one generic ‘judge prompt’ would be weak. So instead, they use domain-specific judges.

An electronics-focused judge applies electronics expertise. A musical instruments judge applies that domain’s patterns and rules. The goal is consistent domain-aware evaluation across verticals.

Results

The system can analyze taxonomy branches in parallel, identifying improvement opportunities that used to take weeks of manual work.

Where experts might analyze a few categories per day, the system can evaluate hundreds of categories, checking both:

• structural consistency

• alignment with real product data

This matters most for emerging product categories. When new product types become popular on the platform, the system can quickly identify taxonomy gaps and propose comprehensive solutions, instead of reactive patches that build up debt.

Quality improvements

The multi-agent design improves consistency and comprehensiveness because it combines two lenses:

• structural analysis keeps hierarchy organization logical and consistent

• product-driven analysis keeps categories and attributes aligned with merchant reality

The automated QA layer reduces iteration cycles by catching issues before human review and applying domain expertise consistently.

Example: mobile phone accessories and MagSafe compatibility

Product analysis identified that merchants frequently advertise “MagSafe support” for accessories such as chargers, cases and wallets.

So the agent proposed adding a boolean attribute: ‘MagSafe compatible.’

A specialized electronics judge evaluated the proposal and checked:

• no duplicate attribute already exists

• boolean type is appropriate

• while brand-specific, MagSafe is treated as a legitimate technical standard similar to Bluetooth or Qi

The judge approved the attribute with 93% confidence, noting it would improve customer filtering for MagSafe-ready products.

This example matters because it demonstrates the full loop:

• merchant reality creates a signal

• the agent proposes a structured change

• a domain judge validates it with rule checks and domain framing

• humans get a higher quality proposal to review

Scaling development: from reactive fixes to proactive evolution

The biggest shift is strategic: taxonomy development becomes proactive, not reactive.

Instead of waiting for a merchant pain point or a platform limitation to trigger a change, the system can identify and address gaps earlier.

The system can also reason over the entire taxonomy structure, which supports cross-category consistency. That helps avoid the fragmentation you get when teams fix issues in isolation.

To validate the approach, they applied it to a specific area: Electronics > Communications > Telephony (called “Telephony AI” in their analysis) and compared it against their previous manual expansion method.

As you can see from the chart, the AI-assisted method can compress years of work into weeks for the taxonomy area if the agents are applied across all verticals.

The full scoop

To learn more about this, check Shopify's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Following on from previous posts talking to people in the field, today I will be talking with Ahsaas Bajaj who is a Senior Machine Learning Engineer at Instacart. He works on large-scale recommendation systems that serves millions of customers.

We talked about his rise from software engineering to machine learning at Instacart, how does he decide between rules based vs ML approaches and how he approaches the work now as a more senior stakeholder.

So without further ado, let’s get into it!

Can you tell us a bit about your role?

I’m a Senior ML Engineer at Instacart, working across customer and shopper experiences on large-scale recommendation systems that make millions of decisions each day. For the past three years, I’ve led the technical strategy for the Product Substitutions ML system, focused on solving the out-of-stock problem.

The goal is simple: when an item isn’t available, suggest a replacement that preserves customer intent and keeps the order intact. My role spans system design, modeling and evaluation, balancing customer satisfaction, shopper efficiency and business impact at scale.

How did you get into machine learning?

My path into ML wasn’t a straight line. I started as a software engineer at Samsung Research on the on-device search team, which pushed me deep into information retrieval and search system design. That work sparked an interest in research and led me to pursue a graduate degree in computer science.

It shaped how I approach ML today: less focus on models in isolation, more on how systems behave in production. I wanted that work to have real user impact, which took me to Walmart Labs and eventually to Instacart.

What does a ‘typical’ week look like for you?

As I’ve moved into a more senior role, the balance has shifted from pure coding to a mix of execution and direction. My week usually breaks down into three buckets:

Alignment (30%): The glue work. I spend time with product, backend engineering, and leadership aligning on roadmaps. The focus isn’t just what we’re building, but why, making sure ML work ties directly to business goals.

Deep work (30%): Hands-on modeling, coding and system design. Staying close to the code is non-negotiable for me, even at a senior level.

Analysis and “the why” (40%): This is where I spend the most time. I dig into model errors, read raw customer complaints about failed substitutions and sanity-check improvement ideas. This is also where I write proposal docs. In my view, the highest-leverage work a senior MLE does is deciding what problems to solve next, not just executing on what’s assigned.

How do you decide when a problem actually needs ML or if rules-based is good enough?

I think about it in terms of complexity versus value.

If a problem can be solved deterministically with clear rules and those rules are stable and understandable, that’s often the right solution. Machine learning becomes useful when the space of behaviors is too large, nuanced, or context-dependent for rules to scale.

Good data is also a prerequisite. Without reliable signals and feedback loops, even the most sophisticated model won’t perform well in production.

You have written about your work on a recommendation model at Instacart. Can you share a summary of what you have done?

I’ve spent the past three years leading the technical development of Instacart’s Product Substitutions system, which handles millions of replacement decisions daily. The core challenge is deceptively simple: when a customer’s requested item is out of stock, what should we suggest instead?

What makes this interesting from an ML perspective is that it’s fundamentally a relevance problem, not a search problem. We’re not just matching product attributes—we’re trying to understand what the customer actually wanted and find alternatives that preserve that intent. This required rethinking how we model the relationship between items, how we define “good” substitutions, and how we evaluate success in a way that maps to real customer satisfaction.

The system has evolved significantly over time, moving from simpler heuristics to more sophisticated learned representations. But the north star has always been the same: keep orders complete while respecting what customers actually care about.

And what has been the impact on the business?

Substitutions sit at a critical junction in the order lifecycle. When done well, they’re invisible - customers get what they need and the order stays intact. When done poorly, they create friction everywhere: customers reject items or request refunds, shoppers waste time on unsuccessful suggestions, and order values drop.

Our work has meaningfully moved the needle on the metrics that matter: replacement acceptance rates, refund frequency, and what we call “perfect order fill rate”—the percentage of orders where every item was either found or successfully replaced. These improvements compound across millions of weekly orders.

Beyond the immediate transactional metrics, we’ve also seen positive signals in repeat ordering behavior and customer satisfaction scores, particularly for orders that required multiple substitutions. Instacart has referenced this system publicly when discussing operational improvements at scale.

For me, the real validation is when customers don’t notice the algorithm at all - they just notice their groceries arrived complete.

What does the tech stack look like for ML at Instacart?

Instacart’s ML stack is built around an internal platform called Griffin, which standardizes the end-to-end ML lifecycle, from feature engineering and training to deployment and real-time inference. A core piece of this is a shared Feature Marketplace, where teams define, version and reuse batch and streaming features with strong offline-to-online consistency.

Workflows are orchestrated with Apache Airflow and model training runs through a unified abstraction that supports multiple compute backends and common ML frameworks. With Griffin 2.0, the platform moved to a Kubernetes-based setup and added distributed training with Ray, which significantly improved scalability and iteration speed.

Griffin also includes a centralized model registry and metadata store, making experiments easier to track and reproduce. In production, models are deployed as standardized services that handle feature loading and low-latency inference across both customer and shopper experiences.

The main benefit is focus: teams spend less time on infrastructure and more time on modeling, evaluation and trade-offs.

How do you use AI in your day-to-day work and where do you find it genuinely valuable?

I’ve integrated GenAI primarily to shift my focus from execution to decision-making. It’s useful for routine tasks like scaffolding data pipelines or optimizing SQL queries, but I find the highest leverage comes from qualitative analysis.

I routinely feed thousands of customer comments and shopper notes about bad substitutions into LLM-driven pipelines that cluster feedback into coherent themes. What used to be unstructured noise becomes a prioritized list of failure modes. This allows me to spend less time parsing data and more time solving the specific problems that actually impact customer trust.

How has your perspective changed moving to a more senior role?

The biggest shift is realizing that Judgment > Code. Early in my career, I obsessed over the how - the architecture, the libraries, the latency. Now, I obsess over the what and the why. The real work is filtering ideas. In a sea of seemingly good ideas, my job is to find the most bullish one - the one with the highest ROI - and kill the others.

I’ve also learned that Writing is Engineering. You cannot build big things alone. To get buy-in from leadership and cross-functional teams, you must be able to write crisp, narrative-driven proposals that explain why this mathematical solution solves a human problem.

What’s one thing you wish you’d known earlier about machine learning?

The value of error analysis. It’s easy to celebrate aggregate metrics like accuracy or F1 but the real breakthroughs come from studying the “horror cases,” where the model is confidently wrong. Those examples are uncomfortable to look at but they’re where the most useful ideas come from. You can’t fix what you don’t deeply understand.

If you enjoyed reading this, check out Ahsaas’s original article about his work at Instacart

Was there a question that you would like to ask?

Let me know your thoughts by replying to the email or leaving a comment below!

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep reading

Today we will look at how DoorDash uses anomaly detection to save millions of dollars by flagging fraud trends early.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to DoorDash’s fraud detection!

TL;DR

Situation

Fraud trends at DoorDash often blended into normal delivery noise and went unnoticed for weeks, causing avoidable losses. Existing detection was reactive and too slow.

Task

Detect emerging fraud trends early across millions of users and segments, before they materially impact top-line metrics.

Action

Build a daily anomaly detection platform that segments key fraud metrics across millions of overlapping dimensions, applies time-series z-score detection, clusters related anomalies and routes them into an ops investigation workflow.

Result

Cut average fraud detection time from 100+ days to under 3 days, surfaced 60%+ of new fraud trends early, and saved tens of millions annually.

Use Cases

Anomaly detection, fraud detection, payment monitoring, policy change impact monitoring

Tech Stack/Framework

Apache Airflow, DuckDB, Apache Spark, Python

Explained further

Fraud trend detection before it becomes a headline

Fraud doesn’t always kick the door down. Sometimes it slips in through the side window and blends into the noise of millions of legitimate deliveries.

A small spike in refund claims. A pattern in high-risk charges linked to a specific bank. A subtle shift in behavior that looks like randomness until it isn’t. Left alone, those early signals can snowball into a large trend with real top-line impact.

DoorDash’s fraud team wanted to flip the script. Instead of reacting after a new fraud trend has had weeks to grow unchecked, how could they spot it as early as possible, before significant damage is done?

This post shares how the DoorDash team built an anomaly detection platform that scans for emerging patterns across millions of user segments and surfaces the ones that matter before they spiral into major losses.

Terminology

‘Anomaly detection’ is a broad term. Even within fraud, people can mean very different things by it. For this system, DoorDash defined two categories up front:

Anomalous trend detection

Looking for anomalous behavior in a collection of users that may represent a new fraud or false-positive trend.

Here, no single datapoint needs to be weird. The anomaly is the time-series pattern that emerges from many points together, like a growing fraud segment over time.

Anomalous outlier detection

Looking for individual outliers, like a specific user or transaction that is rare or deviates sharply from normal behavior.

In this case, the datapoint is the anomaly. It might be part of a broader trend, or it might be a one-off.

This post focuses how DoorDash built a system to detect anomalous trends.

Here are some terms used within the article and their definitions and examples to make them easier to understand.

Designing the system around real fraud failures

The DoorDash team started the way you’d hope a fraud platform starts: by talking to the people who have to use it.

They met with frontline fraud teams responsible for tracking and fighting new fraud trends and asked for concrete historical examples of trends that simmered longer than ideal before being discovered and mitigated. These became the positive test cases.

Next, the teams were asked for:

• Their most useful early-warning indicator metrics

• The dimensions they commonly use to slice data when investigating a new fraud trend

That produced a working set of:

• Positive examples (historical missed or late-found fraud trends)

• A set of metrics that act as early-warning signals

• A set of dimensions that represent how investigators naturally segment the world

Then the DoorDash team built the system and backtested it. Tuning came next, but the tuning goal was very specific:

1- Maintain 100% recall on the test trends
2- Minimise the number of non-fraudulent anomalies per day

One observation stood out from this phase. The system was fairly insensitive to exact tuning values. What mattered more was upstream: choosing thoughtful metrics and dimensions that can actually capture fraud trends in the first place.

In other words: the math is important but the slices you choose decide what you can even see.

Architecture overview

The anomaly detection platform runs as a daily job coordinated by Airflow. It looks for fraud trends growing on a day-to-week timescale.

DoorDash currently runs anomaly detection jobs for both consumer fraud and Dasher fraud, with plans to expand to more applications over time.

The platform has five steps:

1. Preparing daily fraud snapshots

2. Metric aggregation on multi-dimensional segments

3. Time-series anomaly detection

4. Hierarchical clustering on anomalous segments

5. Turning clusters into investigations and containment

Step 1: Preparing daily fraud snapshots

DoorDash chose a daily batch job for the initial implementation because the fraud trends they historically missed developed over a few days to a few weeks.

An Airflow DAG prepares a dataset for each anomaly detection job containing the day’s data snapshot in a wide-table format.

If the trends you historically missed unfold across days and weeks, you do not need sub-second streaming to get meaningful wins. You need consistency, coverage and a reliable cadence.

Step 2: Metric aggregation on multi-dimensional segments

This is the scale step. Once the daily snapshot is ready, DoorDash loads the single date’s data into a Python environment via Spark, then computes metric aggregates across segments.

For each metric, they track both:

• Absolute value of the metric

  • Example: dollar value of credit and refund claims

• Relative (normalized) value of the metric

  • Example: credit and refund claims divided by dollar value of orders

Why both? because absolute values catch ‘this is costing real money’ and relative values catch ‘this is spiking compared to what is normal for this slice’.

Then comes segmentation. Segments are formed from single, double and triple product combinations of all dimensions. That quickly becomes huge and can run into 100s of millions of segments at Doordash scale and compute becomes important

DuckDB for aggregation

DoorDash computes metric aggregates using DuckDB, an in-memory Python database optimised for fast OLAP-style operations.

They chose DuckDB because it was:

• Much faster (less than 10 minutes)

• More memory efficient than Pandas

The system also excludes dimensional products with cardinality greater than 10^7 to reduce the total number of segments to a manageable size.

Finally, storage format.

The day’s metrics aggregated across hundreds of millions of segments are stored in the data warehouse in sparse tall table format.

In plain English: if a segment has a metric value of zero, DoorDash drops it. That cuts storage and keeps both DuckDB and the downstream warehouse from filling up with rows that say ‘nothing happened here.’

Step 3: Time-series anomaly detection

After Step 2, DoorDash has daily metric aggregates by segment. They keep the previous 28 days of data in the data warehouse, so the platform now has several hundred million metric time series, each of length 28.

DoorDash chose a simple moving-window z-score approach, because it performed well in testing and detected all historical fraudulent trends they used as positive examples.

Baseline and test setup

• First 21 days form the baseline

• The 28th day is the test day

• There is a 7-day gap between the baseline and the test day

That gap exists for a very specific reason. The team noticed many historical fraud trends had a noisy phase when they first started scaling. By leaving a gap, the baseline variance better reflects ‘normal before the trend’ which reduces missed trends.

What counts as an anomaly

A segment’s time series is flagged as anomalous if it meets both:

1. Statistical significance: The 28th-day relative metric is greater than X standard deviations above the mean of the 21-day baseline. DoorDash found 6 standard deviations worked well empirically.

2. Business significance: The 28th-day absolute metric exceeds the 21-day baseline by a dollar value and/or count that is meaningful for that metric. Thresholds vary by metric and were chosen with operations partners.

That two-part rule matters. Statistical significance alone finds weirdness. Business significance filters it down to weirdness that’s worth a human’s time.

Step 4: Hierarchical clustering on anomalous segments

Real fraud trends rarely show up as a single clean segment anomaly. A single trend often triggers anomalous increases across many partially overlapping segments. Example:

A spike in credit and refund claims at ‘Retailer One’ could cause anomalies in segments like:

• {business_name='Retailer One'}

• {country='US', business_name='Retailer One'}

• {business_vertical='retail', business_name='Retailer One'}

So Step 4 exists to shrink ‘thousands of anomalies’ into ‘a few dozen things to look at’.

Segment graph structure

Dimensional segments have a natural structure that can be represented as a three-layer graph:

• Top layer: singlets

  • {business_name='Retailer One'}

• Middle layer: pairs

  • {business_name='Retailer One', country='US'}

• Bottom layer: triplets

  • {business_name='Retailer One', country='US', checkout_platform='iOS'}

DoorDash further partitions the graph by METRIC_NAME so clustering happens within a metric type.

Clustering rules

To connect anomalies within the same metric type:

1. Connect parent anomalies with child anomalies

   • {business_name='Retailer One'} is parent of {country='US', business_name='Retailer One'}

   • {country='US', business_name='Retailer One'} is parent of {business_name='Retailer One', country='US', checkout_platform='iOS'}

2. Connect sibling anomaly triplets if they share 2/3 of their keys and values

   • {business_name='Retailer One', country='US', checkout_platform='iOS'}
     connects with
     {business_name='Retailer One', country='US', business_vertical='retail'}

Then DoorDash runs a graph partition algorithm to find connected anomaly clusters.

Picking a representative segment

Ops teams review a cluster starting from a single representative segment chosen using a fitness function:

fitness = abs_anom_amt * rel_amt / level^1.2

Where:

• abs_anom_amt = 28th-day metric minus the 21-day baseline

• rel_amt = relative (normalized) 28th-day metric within the segment

• level = 0 for singlets, 1 for pairs, 2 for triplets

The intuition:

• abs_anom_amt behaves a bit like ‘how much impact’ (think recall)

• rel_amt behaves a bit like ‘how concentrated’ (think precision)

• dividing by a weak function of level biases toward simpler segments

So the representative is usually a segment that is impactful, unusually high relative to its baseline and not needlessly specific.

What volume looks like in practice

In real operation, DoorDash typically sees anomalies in several thousand segments per day. Clustering reduces that to 20 to 60 anomalous clusters per day across consumer and Dasher fraud areas, which is a volume the operations team can realistically investigate.

Step 5: Turning clusters into investigations and containment

Detection is not the finish line, it is just the trigger.

The representative anomalous segments, along with all other segments in the cluster and example events (deliveries and Dasher assignments), are accessible in a workflow tool for ops investigation.

Ops agents review example deliveries or assignments within the representative segment, looking for trends or patterns that may represent a new fraud trend.

Sometimes the pattern is non-fraudulent, like a new promotion causing a spike in refunds. Other times it is fraudulent.

When a trend is deemed fraudulent:

• it is root-caused in partnership with engineering and product teams so the root cause can be addressed

• a separate containment team runs queries to identify and stop fraudsters matching the trend pattern until product fixes land

So the system is not just detection. It’s detection wired into investigation, containment and longer-term remediation.

Results

DoorDash now uses the anomaly detection platform as its primary early-warning source for new fraudulent trends.

Key results reported by the team:

• More than 60% of all new fraud trends today are found through anomaly detection, and that share is growing as coverage expands.

• Average time-to-detect new fraud trends dropped from more than 100 days to less than three days over the past year.

• The platform saves tens of millions of dollars per year by flagging small but growing fraud trends before they get out of control.

The full scoop

To learn more about this, check DoorDash's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

Following on from previous posts talking to people in the field, today I will be talking with who is a Senior Data Science Manager at Skyscanner and writer of the Senior Data Science Lead newsletter.

Senior Data Science Lead

Helping managers build world-class teams, data professionals master storytelling and guiding those looking to break into Data Science. I have built teams from scratch and lead 50+ data scientists. Now, I share my experience with you.

By Jose Parreño Garcia

We talked about his rise from data analyst to Senior DS Manager at Skyscanner, what “production-ready” really means and why the real intelligence in data science lives before and after the model.

So without further ado, let’s get into it!

Read more

Today we will look at how Dropbox does eval for its conversational AI

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Dropbox’s AI evaluation!

TL;DR

Read more

Today we will look at how Netflix uses neural networks to automate the process of flagging pixel errors

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to automation of video quality control by Netflix

Read more

Today we will look at how Uber uses agentic AI to answer financial questions.

But before that, I wanted to share with you what you could unlock if you share Data Tinkerer with just 1 more person.

There are 100+ resources to learn all things data (science, engineering, analysis). It includes videos, courses, projects and can be filtered by tech stack (Python, SQL, Spark and etc), skill level (Beginner, Intermediate and so on) provider name or free/paid. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to an actual use case of agentic AI by Uber

Read more

Today we will look at how Target used GenAI to recommend better products to users.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s see how Target leverages GenAI for better shopping experience.

TL;DR

Read more

Today we will look at how Doordash used LLM to show better results to users.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, Let’s see how Doordash leverages LLM for better search results

TL;DR

Situation

DoorDash users search with ultra-specific, complex queries (e.g. low-carb spicy chicken wrap with gluten-free tortilla), expecting accurate matches. Traditional keyword systems couldn’t handle the nuance and pure LLM-based solutions were prone to hallucination.

Task

Build a scalable, accurate and fast search system that could deeply understand both queries and items without sacrificing relevance or blowing up compute.

Action

• DoorDash built a hybrid system using:

  • LLMs (for query segmentation and classification)

  • Knowledge graphs (for structured metadata)

  • Approximate Nearest Neighbor search (for grounding output in known taxonomies)

• Added tight guardrails: constrained vocabularies, structured output prompts, and post-processing validation.

• Plugged structured query outputs into the search rankers and retrained based on improved engagement data.

Result

• ~30% increase in trigger rate for “Popular Dishes” carousel

• 2% lift in whole page relevance (WPR) for intent-heavy queries

• +1.6% WPR boost from retrained rankers due to better training signals

Use Cases

Improved search relevance, query processing at scale, ranking and personalisation

Tech Stack/Framework

LLM, knowledge graph, ANN

Explained further

At DoorDash, users don’t just type “burger” and call it a day. They type “low-carb spicy chicken wrap with gluten-free tortilla” and expect relevant suggestions. These aren’t simple lookups, they’re loaded with layered preferences, dietary restrictions and dish expectations.

So how do you build a search system that can handle that level of complexity without cracking under pressure?

The DoorDash team went with a hybrid approach: combine the brute force of traditional keyword systems with the nuance of LLMs and knowledge graphs. It’s like pairing a calculator with a philosophy major. It works and here is how.

What happens behind the search bar

At a high level, DoorDash's search engine processes two kinds of things:

• Documents - the restaurants or menu/store items

• Queries - what users type in

Each goes through its own journey. Queries get parsed, segmented, labeled and spell-checked. Documents get annotated and enriched with metadata, like vegan, spicy or chicken.

But the real power comes from understanding both deeply enough to match the right taco to the right craving.

Two worlds: documents and queries

Documents at DoorDash aren’t just raw strings, they’re enriched with metadata pulled from custom-built knowledge graphs. These graphs define relationships between things like dish type, cuisine, ingredients and dietary preferences.

Take Non-Dairy Milk & Cookies Vanilla Frozen Dessert - 8 oz as an example. It gets tagged with:

• Dietary_Preference: "Dairy-Free"

• Flavor: "Vanilla"

• Product_Category: "Ice Cream"

• Quantity: "8 oz"

On the query side, things get chunked up and mapped to similar concepts. So “small no-milk vanilla ice cream” gets split into:

[“small”, “no-milk”, “vanilla ice cream”]

Without this kind of breakdown, the search engine would either over-match (ice cream that isn’t dairy-free) or under-match (only exact string matches). With it, search becomes smarter.

But understanding concepts is only half the battle. How do you break down messy, real-world queries in the first place? That’s where things get messy and where traditional methods start to hit their limits.

Using LLMs without letting them run wild

Traditionally, query segmentation has leaned on methods like pointwise mutual information (which checks how often words co-occur compared to random chance) and n-gram analysis (which looks at fixed word sequences) to figure out which terms belong together. That works fine for clean, simple queries. But once you throw in overlapping entities or any real ambiguity, those old-school tricks start to break down fast.

Take a query like “turkey sandwich with cranberry sauce.” Is the cranberry sauce a separate item or part of the sandwich? Without context, traditional methods have no clue. They just see word chunks, not relationships.

LLMs, however, can understand the context if given the right guardrails. They will usually segment things in a way that actually makes sense, picking up on how words relate to each other across different use cases.

But here’s the catch: LLMs are prone to hallucination. So you couldn’t just throw queries at them and hope for the best. Doordash needed a controlled vocabulary to keep things grounded, something that ensured the segments were not only real but useful for the retrieval system.

Fortunately, they already had a knowledge graph in place with a solid ontology and a bunch of taxonomies to work with. So instead of letting the model come up with its own segmentations, it’s prompted to identify meaningful chunks and tag each one with a specific category from Doordash taxonomy.

For food, They have got taxonomies for cuisine types, dish types, dietary tags and more. For retail, it’s things like brand, dietary preference, product category, etc.

Back to our earlier query: “small no-milk vanilla ice cream.”

If we just asked for segments, we might get:

[“small”, “no-milk”, “vanilla ice cream”]

Not bad, but vague. Instead, the team asks the model to return a structured output mapping that relates to one of the Doordash taxonomy categories:

{

Quantity: "small",

Dietary_Preference: "no-milk",

Flavor: "vanilla",

Product_Category: "ice cream"

}

This setup gives the model extra context to work with and it leads to better results. it doesn’t just segment, it classifies. But classification isn’t the end of the road. Once you’ve got the chunks, you need to actually understand what they mean.

Entity linking: turning words into meaning

Once a query is segmented, the next step is to map each segment to a concept in the knowledge graph. For example, “no-milk” isn’t just a string match. It should link to the “dairy-free” concept even if the item description never says “no-milk” explicitly anywhere.

LLMs are handy here too but just like in segmentation, they can still hallucinate. To avoid that problem, the team constrained the model’s output using only concepts from their controlled vocabularies (i.e., taxonomy terms from the knowledge graph).

The guardrails come from a curated candidate list, pulled using approximate nearest neighbor (ANN) retrieval. ANN is a fast way to find the most similar entries in a big dataset without doing a full exhaustive search. This keeps the LLM grounded by forcing it to choose from concepts that actually exist in the system.

Take the earlier example: the segment “no-milk.” Instead of letting the LLM invent a new category like “milk-averse”, the ANN system first retrieves candidates like “dairy-free” or “vegan.” The model then just has to pick the best match based on context, keeping the output both accurate and mappable.

The method behind this is a classic retrieval-augmented generation (RAG) setup:

1. For every query and knowledge graph concept (a.k.a. candidate label), embeddings are generated either from closed-source models, pre-trained ones or DoorDash’s own.

2. Using ANN retrieval, the top 100 closest taxonomy concepts are pulled for each query. This trims the fat from the prompt and avoids blowing up the model’s context window.

3. The LLM is then prompted to link query segments to taxonomy concepts across domains like dish types, dietary tags, cuisines and more.

The retrieval payoff

Let’s go back to “small no-milk vanilla ice cream.” After segmentation and entity linking, the final structured output might look like this:

{

Dietary_Preference: "Dairy-Free",

Flavor: "Vanilla",

Product_Category: "Ice Cream"

}

Now retrieval becomes precise, it can make Dietary Preference a MUST condition (non-negotiable) and treat Flavor as a SHOULD (nice to have). It’s like giving the system a conscience, it knows what matters more.

But smart retrieval isn’t enough. You also need to double-check what the model spits out before someone eats the wrong thing.

Evaluations: catching hallucinations before users do

Accuracy is especially critical for filters like dietary restrictions. One mistake and someone could end up eating something they’re allergic to.

To keep hallucinations in check and make sure both segmentation and entity linking are solid, the team added post-processing steps to validate the model’s output. Once that’s done, they run manual audits on every batch of processed queries to spot any issues before they hit production.

Annotators go through a statistically significant sample of results and check two things:

1. Are the segments correct?

2. Are they mapped to the right concepts in the knowledge graph?

This manual review helps surface any systematic slip-ups, fine-tune prompts and improve the overall pipeline. Still, even a well-reviewed system comes with trade-offs, especially when scale and change are constants.

Memorisation vs. generalisation: the trade-off

LLMs are a solid tool for query understanding but like most things in ML, there’s a trade-off. In this case, it’s memorisation versus generalisation.

If you run LLMs in batch mode on a fixed set of queries, you’ll usually get high accuracy. That works great when the query space is small and predictable. But the further you drift into the long tail (which, let’s be honest, is where a lot of real-world queries live), things start to get tricky.

Relying purely on memorisation comes with baggage:

• Scalability: New queries show up constantly in a system like DoorDash. Preprocessing every single one just isn’t realistic.

• Maintenance overhead: Any changes to the knowledge graph or user behavior mean re-running batches, tweaking prompts and revalidating outputs.

• Stale features: Segments or links that were fine last month might be outdated now.

That’s where more generalisable approaches shine. Things like embedding-based retrieval, traditional statistical models or even smart rule-based systems can handle new queries on the fly without any preprocessing.

These methods bring a few key advantages:

• Scalability: They work on any query, even ones the system hasn’t seen before.

• Flexibility: They keep up with evolving language like new dishes, slang, weird typos

• Real-time readiness: No batch processing delays; they can act immediately.

The downside? They don’t have the same deep contextual smarts as LLMs. That can hurt precision, especially when subtle relationships between terms matter.

So DoorDash takes a hybrid approach. Instead of picking one side, they combine the strengths of both:

• LLMs for deep understanding when you can afford it

• Lightweight retrieval methods when you need scale and speed

The result: better precision where it matters and better coverage everywhere else.

But theory only gets you so far. The real test? Whether it plays nicely with the rest of the stack.

Making it all work in production

Of course, even the best query understanding system is only useful if the rest of the stack knows what to do with it. For DoorDash, that means making sure the new signals actually integrate cleanly into the search pipeline, especially with the rankers.

Rankers are the part of the system responsible for taking a big pile of candidate items or stores and deciding what order they should show up in. So once the team introduced these new, structured query signals, the rankers had to be updated to recognise and use them.

As the rankers adjusted to the new inputs and to the new patterns of user behavior that came with better retrieval, the impact showed up fast. Online metrics went up, both on the relevance side and on key business KPIs.

One feature in particular showed just how well things were clicking.

Real world example: açaí bowl, anyone?

One clear beneficiary of DoorDash’s improved query understanding system is the “Popular Dishes” carousel shown below. This component shows users a ranked list of food items when they search with clear dish intent (like “açaí bowl”).

When someone types in “açaí bowl,” they’re not looking to browse menus. They want a specific thing. The carousel lets them quickly scan options across stores, compare prices and pick what looks good. It’s fast, it’s focused and thanks to the new pipeline, it’s pulling in way more relevant results.

After rolling out the updated query understanding and retrieval system, DoorDash saw a ~30% jump in the carousel trigger rate. That means more queries are now returning results that qualify for the Popular Dishes treatment. It’s a strong signal that the system is getting closer to what users actually want.

And it’s not just about quantity, quality improved too. With better segmentation and entity linking, the system can retrieve a broader, more accurate set of items tied to the user’s intent. This showed up in their whole page relevance (WPR) metric, which measures how relevant the entire result page is from the user’s point of view. For dish-intent queries, WPR jumped by over 2%.

And because users were interacting with a wider variety of results, DoorDash was able to retrain its rankers on a more diverse engagement dataset. That led to a newer, smarter ranker version and another 1.6% bump in WPR. And because users interact more with the right stuff, the system gets better training data for the ranker, which then improves things further and becomes a virtuous cycle.

Lessons learned

• LLMs are powerful but more so with guardrails. Left to their own devices, they hallucinate. But paired with taxonomies and controlled vocabularies, they become sharp tools for segmentation and classification.

• Structured queries unlock smart retrieval. Turning messy user input into clean, labeled chunks (like "Flavor: Vanilla") makes it easier to match intent, not just strings.

• Hybrid > pure LLM. DoorDash got the best results by mixing deep LLM understanding with fast, generalizable methods like embedding retrieval.

• Better understanding improves ranking. Once structured query signals flowed into rankers, DoorDash saw real-world lifts: 30% more carousel triggers and measurable boosts to conversion and relevance.

The full scoop

To learn more about this, check Doordash's Engineering Blog post on this topic

If you are already subscribed and enjoyed the article, please give it a like and/or share it others, really appreciate it 🙏

Share

Keep learning

This article was originally published on around two weeks ago. who writes the newsletter suggested doing a guest post and I was more than happy to do it.

You should check out his newsletter if you need to stay up to date on all the latest developments in AI. He is a machine in terms of keeping track of the latest news and publishing it quickly for his subscribers. So if staying updated on all things AI is your jam, give it a go!

AI Disruption

A Beijing-based AI engineer with a decade of experience shares at least 2 daily articles on AI industry trends and technical insights.

By Meng Li

Now, let’s get to today’s deep dive on how Uber used GenAI in their invoice processing workflow.

Read more

Today we will look at how Reddit handles NSFW content at scale using deep learning

But before that, I wanted to thank those who shared Data Tinkerer with others and share an example of what you can unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Reddit’s automated content moderation

Read more

Today we will look at how Walmart autotuned its time series forecasting

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 2 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to Walmart’s autotuning framework!

Read more

Today we will look at how an LLM thinks based on the research done by Anthropic.

But before that, I wanted to share an example of what you could unlock if you share Data Tinkerer with just 3 other people.

There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!

Refer a friend

Now, with that out of the way, let’s get to the inner workings of Claude’s mind

Read more

TL;DR

Read more

TL;DR

Situation

Pinterest's existing content retrieval system relied on traditional methods that couldn't fully capture the complex relationships between users and the vast array of content, leading to less personalized recommendations.

Task

Develop a scalable, embedding-based retrieval system capable of learning and representing the nuanced interactions between users and content, effectively processing Pinterest's extensive dataset.

Action

1. System Design: They designed an internal embedding-based retrieval system for organic content, utilizing advanced machine learning techniques to generate embeddings that position users and content within a shared vector space.

2. Data Processing: To train the model effectively, they processed large-scale data, extracting meaningful features from user interactions and content metadata, handling billions of data points to ensure accurate embeddings..

3. Model Training and Deployment: The model was trained on this extensive dataset, optimized for performance and relevance, and seamlessly integrated into Pinterest's infrastructure without disrupting user experience.

Result

Implementing the embedding-based retrieval system improved content relevance and user engagement on Pinterest, leading to more personalized recommendations and increased interaction rates.

Use Cases

Personalized Recommendation, Search Functionality

Tech Stack/Framework

Two-Tower Model, Approximate Nearest Neighbor, Auto Retraining

Explained Further

Content Discovery with Advanced Retrieval System

Read more

TL;DR

Read more

TL;DR

Situation

Grammarly’s 30M+ daily users rely on paid ads, but traditional attribution failed to measure YouTube’s true impact, especially with third-party cookies disappearing.

Task

Find a privacy-first way to determine if YouTube ads drive new users or just capture organic sign-ups using geo experimentation.

Action

• Geo Experimentation: Stopped ads in select regions while maintaining them in others to measure impact.

• Data Modeling: Used Google’s TBR package with a BSTS model to estimate incremental user acquisition.

• Geo Split Selection: Created balanced test/control groups using clustering and randomization.

• Power Analysis: Optimized experiment duration and sample size for accuracy and minimal business disruption.

• KPI Measurement: Focused on new active users instead of revenue, aligning with Grammarly’s freemium model.

Result

YouTube ads were most effective before peak seasons but had less impact during them. When ads were paused, new sign-ups dropped immediately, showing YouTube’s stronger mid-funnel role than expected.

Use Cases

Marketing Attribution Improvement, Media Spend Optimization, Incrementality Measurement

Tech Stack/Framework

Google TBR (Time-Based Regression), Bayesian Structural Time Series (BSTS) Model, Clustering, Randomization

Explained Further

Understanding the Attribution Challenge

Measuring marketing effectiveness is challenging, especially for ads that don’t involve direct clicks. It’s easy to track whether someone clicks on a Google search ad, but what about an ad they only view on YouTube? Traditional attribution models often struggle to assign proper credit to top-of-funnel marketing, leading to an undervaluation of video ads and TV campaigns.

Read more

TL;DR

Read more