A/B Testing (Split Testing)

Product Strategy & Growth • Design Systems & Engineering

A/B testing is a controlled experiment that randomly assigns eligible participants to different versions of a product experience and compares predefined outcomes. A well-run test helps estimate the effect of a change for the audience and conditions tested.

How do you plan a UX A/B test?

Suppose new accounts draft an invoice but hesitate to send it because they cannot see what their client will receive. The team tests a preview that shows the invoice as the client would see it.

Decision Example
Hypothesis A preview helps more new accounts send a first real invoice by showing what the client will receive.
Control and treatment Group A sees the current experience. Group B gets an option to preview the invoice as the client would see it.
Unit of randomization Assign each eligible new account at signup so teammates use the same version.
Primary metric Percentage of assigned accounts that send a first real invoice within seven days.
Guardrails Accidental sends and invoice-related support requests per assigned account.

Before launch, agree on the smallest effect worth acting on, the sample and duration needed, and the analysis and stopping rules. Make sure each account has the full outcome window before the final analysis. In this example, the denominator includes all eligible assigned accounts, including accounts that never open the preview.

What should you check before reading the result?

Verify that assignment and events work in both variants. Compare the observed group sizes with the planned allocation. A statistically significant difference between the observed and planned group proportions is a sample ratio mismatch. It can indicate problems with assignment, eligibility, or tracking. Diagnose it before interpreting the treatment effect.

How do you interpret an inconclusive test?

Review the estimated difference and its uncertainty alongside guardrails. Suppose the estimated change is +2 percentage points, with a 95% confidence interval from −1 to +5 points. The interval spans both a decrease and an improvement. If those outcomes would lead to different decisions, the test has left the decision uncertain.

Agree on the next step based on what is still unknown. Verify the implementation if data quality is in doubt. Revisit the hypothesis if users did not understand the preview. If you need a more precise estimate, plan a follow-up with enough traffic and a suitable analysis rule.

Sources