What is A/B Testing?
A/B testing randomly splits traffic between control and treatment (e.g., new ranker, different facets) and compares outcomes like CTR, add-to-cart, conversion, revenue.
How It Works (quick)
- Bucketing: Assign users/sessions deterministically.
- Metrics: Define primary/guardrail metrics (latency, zero-results).
- Run: Until power thresholds; monitor anomalies.
- Decide: Ship, iterate, or roll back.
Why It Matters in E-commerce
- Replaces guesswork with evidence.
- Catches regressions (e.g., faster but less relevant SERPs).
Best Practices
- Bucket at session/user level to avoid cross-exposure.
- Pre-register hypotheses and duration; avoid peeking.
- Segment by category/locale; compute CIs.
- Pair with interleaving for faster early reads.
Summary
Use A/B tests to prove impact before wide rollout—clean bucketing, clear metrics, and disciplined decisions.
