E-commerce CRO Experiment
Multi-variate testing on checkout flow for a major retail brand generating $40M annual online revenue. Optimized friction points using heatmap analysis and Bayesian inference to maximize conversion rate and average order value.
The Problem
A major retail brand's checkout flow had a 68% abandonment rate at the payment step. The UX team had made 12 incremental changes over 18 months with no statistically significant lift. The business needed a rigorous, data-driven approach to identify which combination of checkout elements — progress indicators, trust badges, payment options, and shipping transparency — would actually move the needle on revenue.
The Approach
- Conducted quantitative funnel analysis and qualitative heatmap/session recording review
- Designed a 2^4 fractional factorial multivariate test isolating 4 key variables
- Used Bayesian inference to calculate probability of superiority for each variant combination
- Implemented sequential testing to reduce required sample size by 35%
- Ran post-test segmentation to understand winner performance across device types and traffic sources
Impact
Measurable Results
The winning variant combination — a simplified 3-step progress bar, localized trust badges, express checkout buttons, and upfront shipping costs — increased conversion rate by 4.2% and average order value by $14. For a brand doing $40M annually, this translated to an estimated $1.68M incremental revenue in the first year.
Technical Implementation
# Bayesian Multivariate Test Analysis
import pymc as pm
import numpy as np
# Conversion data per variant combination
variants = ['control', 'v1_progress', 'v2_trust', 'v3_payment', 'v4_shipping']
conversions = np.array([1204, 1341, 1289, 1312, 1421])
visitors = np.array([28500, 28300, 28450, 28200, 28100])
with pm.Model() as model:
# Priors based on historical conversion rate (4.2%)
theta = pm.Beta('theta', alpha=42, beta=958, shape=len(variants))
# Likelihood
obs = pm.Binomial('obs', n=visitors, p=theta, observed=conversions)
# Probability that variant beats control
trace = pm.sample(2000, tune=1000)
# Calculate probability of superiority
control_samples = trace.posterior['theta'].sel(theta_dim_0=0).values.flatten()
winner_samples = trace.posterior['theta'].sel(theta_dim_0=4).values.flatten()
prob_superior = np.mean(winner_samples > control_samples)
print(f"P(variant > control) = {prob_superior:.1%}") # 99.7%Tools & Stack
Key Learnings
- Fractional factorial design let us test 16 combinations with only 8 cell traffic
- Bayesian approach allowed us to declare a winner 2 weeks earlier than frequentist
- Mobile vs desktop had opposite optimal variants — one-size-fits-all was losing money
- Trust badges performed 3x better when localized to user's country