Metric
Sample ratio mismatch
Before reading an experiment, check that its groups have the sizes they were meant to have.
For a 50/50 test with n users, chi_square = 2 × (control − n/2)² / (n/2); above 3.84, the split is very unlikely to be
chance. A mismatch means the assignment or the logging is broken, and no result of the experiment can be trusted. Check
it for every experiment, then per segment (device, country, week).
WITH split(experiment, control, treatment) AS (VALUES
('pricing_page_v2', 22026, 21881), ('checkout_v3', 10400, 9600))
SELECT experiment, control, treatment,
ROUND(2 * POWER(control - (control + treatment) / 2.0, 2) / ((control + treatment) / 2.0), 2) AS chi_square
FROM split
ORDER BY experiment;
| experiment | control | treatment | chi_square |
|---|---|---|---|
| checkout_v3 | 10400 | 9600 | 32.0 |
| pricing_page_v2 | 22026 | 21881 | 0.48 |