Data-quality checks
These gates come from Section 5 of the analysis plan and run before any outcome is compared across groups. A failed integrity gate or a sample-ratio p-value below 0.001 halts the analysis.
1. Integrity
All gates pass · 64,000 rows (documented: 64,000), 12 columns.
| Gate | Violations | Result |
|---|---|---|
| Row count equals the documented population size | 0 | Pass |
| No missing values in any column | 0 | Pass |
| Spend is never negative | 0 | Pass |
| Positive spend implies a conversion | 0 | Pass |
| A conversion implies a visit | 0 | Pass |
| segment: only documented levels | 0 | Pass |
| history_segment: only documented levels | 0 | Pass |
| zip_code: only documented levels | 0 | Pass |
| channel: only documented levels | 0 | Pass |
| mens: only documented levels | 0 | Pass |
| womens: only documented levels | 0 | Pass |
| newbie: only documented levels | 0 | Pass |
| visit: only documented levels | 0 | Pass |
| conversion: only documented levels | 0 | Pass |
2. Sample ratio mismatch
No mismatch · Chi-square test of group sizes against an equal split: χ²(2) = 0.203, p = 0.904. Halt threshold p < 0.001.
| Group | Observed | Expected | Share |
|---|---|---|---|
| Mens E-Mail | 21,307 | 21,333.3 | 33.29% |
| Womens E-Mail | 21,387 | 21,333.3 | 33.42% |
| No E-Mail | 21,306 | 21,333.3 | 33.29% |
3. Covariate balance
Standardized mean difference (SMD) of each pre-period covariate, each email group vs No E-Mail. Rows with |SMD| > 0.1 are flagged and highlighted. 0 of 36 comparisons flagged; largest |SMD| = 0.0164.
| Covariate | Mens E-Mail | Womens E-Mail |
|---|---|---|
| Months since last purchase | +0.0068 | +0.0052 |
| Prior-year spend ($) | +0.0076 | +0.0065 |
| Bought men's merchandise | −0.0046 | −0.0086 |
| Bought women's merchandise | +0.0076 | +0.0049 |
| New customer | −0.0009 | +0.0026 |
| Prior-year spend band: 1) $0 - $100 | +0.0109 | −0.0007 |
| Prior-year spend band: 2) $100 - $200 | −0.0164 | −0.0143 |
| Prior-year spend band: 3) $200 - $350 | +0.0055 | +0.0114 |
| Prior-year spend band: 4) $350 - $500 | −0.0043 | +0.0087 |
| Prior-year spend band: 5) $500 - $750 | −0.0097 | +0.0006 |
| Prior-year spend band: 6) $750 - $1,000 | +0.0061 | −0.0088 |
| Prior-year spend band: 7) $1,000 + | +0.0158 | +0.0035 |
| Zip code class: Rural | +0.0137 | +0.0040 |
| Zip code class: Suburban* | −0.0117 | −0.0011 |
| Zip code class: Urban | +0.0020 | −0.0018 |
| Purchase channel: Multichannel | −0.0042 | −0.0053 |
| Purchase channel: Phone | −0.0083 | +0.0086 |
| Purchase channel: Web | +0.0110 | −0.0051 |
* Shown as “Suburban”. The source file spells this level “Surburban”; analysis code and data exports keep the source spelling (see Deviations in the analysis plan). Method: SMD = (mean_treated - mean_control) / sqrt((v_treated + v_control) / 2); continuous covariates use sample variances (ddof=1), binary covariates and one-hot categoricals use p(1-p).
Note: prior-year spend (history) and the prior-year spend band (history_segment) overlap by construction, because the band is derived from the dollar amount. Both are listed because the plan names both as pre-period covariates, but they are not independent balance checks.
4. Planning power (minimum detectable effects)
Conversion rate · MDE vs No E-Mail
| Assumed baseline | Mens E-Mail | Womens E-Mail |
|---|---|---|
| 0.5% | 0.21 pp42% relative | 0.21 pp42% relative |
| 1.0% | 0.30 pp30% relative | 0.30 pp30% relative |
| 2.0% | 0.42 pp21% relative | 0.42 pp21% relative |
Visit rate · MDE vs No E-Mail
| Assumed baseline | Mens E-Mail | Womens E-Mail |
|---|---|---|
| 10.0% | 0.90 pp9% relative | 0.90 pp9% relative |
| 15.0% | 1.07 pp7% relative | 1.07 pp7% relative |
| 20.0% | 1.19 pp6% relative | 1.19 pp6% relative |
Revenue per customer · MDE per customer vs No E-Mail
| Assumed SD of spend | Mens E-Mail | Womens E-Mail |
|---|---|---|
| $5 | $0.15 | $0.15 |
| $10 | $0.30 | $0.30 |
| $15 | $0.45 | $0.45 |
| $20 | $0.60 | $0.60 |
| $30 | $0.90 | $0.90 |
Method: Normal approximation; MDE = (z_{1-alpha/2} + z_power) * sd * sqrt(1/n_t + 1/n_c); proportions use sd = sqrt(p(1-p)) at the assumed baseline. These are normal-approximation planning values; exact small-sample power, the skew of spend and multiple-testing adjustments beyond the per-test α are not modelled.