Data Tinkerer

Data Tinkerer

Share this post

Data Tinkerer
Data Tinkerer
Boosting A/B Testing Efficiency for 1M+ Users
Data Science

Boosting A/B Testing Efficiency for 1M+ Users

Explore how Stripe rolled out A/B testing at scale

Data Tinkerer's avatar
Data Tinkerer
Dec 29, 2024
∙ Paid
2

Share this post

Data Tinkerer
Data Tinkerer
Boosting A/B Testing Efficiency for 1M+ Users
1
Share
(Source: Stripe)

TL;DR


Situation

Stripe users wanted to test how new payment methods impacted conversions and revenue but faced complex, resource-intensive A/B testing processes

Task

Stripe set out to build a no-code A/B testing tool, enabling users to experiment with payment methods and analyze outcomes effortlessly

Action

The team took three actions into account:

  1. Statistical Significance: Introduced a time window feature to let the same customer appear in both control and treatment groups over time.

  2. Avoiding Dilution: Filtered transactions to ensure only relevant data was analyzed for accurate results.

  3. Event Linking: Built a pipeline to connect payment "render" and "confirm" events for precise metric tracking

Result

Indiegogo (Stripe customer) saw a 2% increase in conversion

Use Cases

Payment Method Optimization, Checkout Experience Improvement

Tech Stack/Framework

A/B Testing


Explained Further


Improving Statistical Significance

Stripe addressed the challenge of long A/B testing durations for smaller customer bases by increasing the number of data points without artificially boosting transaction volumes. Traditionally, each checkout session counts as a data point, but to accelerate statistical significance, Stripe introduced a time-window component. This allowed the same customer to appear in both control and treatment groups over time. Within a specific time interval, a customer would see one set of payment methods, and after the interval elapsed, they could encounter a different set during their next purchase. This method effectively increased sample sizes and shortened experiment durations.

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 Data Tinkerer
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share