Boosting A/B Testing Efficiency for 1M+ Users
Explore how Stripe rolled out A/B testing at scale
TL;DR
Situation
Stripe users wanted to test how new payment methods impacted conversions and revenue but faced complex, resource-intensive A/B testing processes
Task
Stripe set out to build a no-code A/B testing tool, enabling users to experiment with payment methods and analyze outcomes effortlessly
Action
The team took three actions into account:
Statistical Significance: Introduced a time window feature to let the same customer appear in both control and treatment groups over time.
Avoiding Dilution: Filtered transactions to ensure only relevant data was analyzed for accurate results.
Event Linking: Built a pipeline to connect payment "render" and "confirm" events for precise metric tracking
Result
Indiegogo (Stripe customer) saw a 2% increase in conversion
Use Cases
Payment Method Optimization, Checkout Experience Improvement
Tech Stack/Framework
A/B Testing
Explained Further
Improving Statistical Significance
Stripe addressed the challenge of long A/B testing durations for smaller customer bases by increasing the number of data points without artificially boosting transaction volumes. Traditionally, each checkout session counts as a data point, but to accelerate statistical significance, Stripe introduced a time-window component. This allowed the same customer to appear in both control and treatment groups over time. Within a specific time interval, a customer would see one set of payment methods, and after the interval elapsed, they could encounter a different set during their next purchase. This method effectively increased sample sizes and shortened experiment durations.