Email Experiment Readout

Effect estimates

Every estimate specified in Sections 6–8 of the pre-registered analysis plan, with its interval. Results are reported whether or not they are statistically significant.

Estimates only. The cost-based decision rule and targeting analysis are applied in Sprint 3.

1. Revenue per customer (primary)

Mean spend over all customers in the arm, including zeros. H1 and H2 form the Holm family at family-wise α = 0.05; H3 is tested the same way but outside the family.

No E-Mail
$0.65
95% CI $0.50 to $0.81 · n = 21,306
Mens E-Mail
$1.42
95% CI $1.18 to $1.66 · n = 21,307
Womens E-Mail
$1.08
95% CI $0.87 to $1.28 · n = 21,387
  • Welch analytic 95% CI
  • Bootstrap percentile 95% CI (10,000 resamples)
H1 · Mens E-Mail vs No E-MailHolm p = 2.3 × 10⁻⁷
+$0.77 per customer · analytic +$0.49 to +$1.05 · bootstrap +$0.49 to +$1.06
Relative lift (secondary): +118% (95% CI +54% to +181%)
H1, Mens E-Mail vs No E-Mail: +$0.77 per customer; analytic CI +$0.49 to +$1.05; bootstrap CI +$0.49 to +$1.06
H2 · Womens E-Mail vs No E-MailHolm p = 0.001
+$0.42 per customer · analytic +$0.17 to +$0.68 · bootstrap +$0.17 to +$0.68
Relative lift (secondary): +65% (95% CI +15% to +115%)
H2, Womens E-Mail vs No E-Mail: +$0.42 per customer; analytic CI +$0.17 to +$0.68; bootstrap CI +$0.17 to +$0.68
H3 · Mens E-Mail vs Womens E-Mailp = 0.030 (outside Holm family)
+$0.35 per customer · analytic +$0.03 to +$0.66 · bootstrap +$0.04 to +$0.67
Relative lift (secondary): +32% (95% CI −1% to +65%)
H3, Mens E-Mail vs Womens E-Mail: +$0.35 per customer; analytic CI +$0.03 to +$0.66; bootstrap CI +$0.04 to +$0.67
Difference in revenue per customer (treatment − comparison), dollars

For H3, the dollar difference and the relative lift are different estimands: the relative-lift interval is wider because it also carries the uncertainty in the comparison group's (Womens E-Mail) mean.

Test statistics and the pre-registered agreement check (bootstrap width ÷ analytic width must lie in 0.80–1.25, same sign pattern).
Welch t (df)pWidth ratioAgree
H15.30 (36,671)1.2 × 10⁻⁷0.994Yes
H23.26 (40,065)0.0011.000Yes
H32.16 (41,585)0.0301.006Yes

2. Visit and conversion rates (secondary)

Difference in proportions with the Newcombe (1998) method 10 hybrid score interval (95%); two-proportion z-test, pooled standard error. Holm correction within each metric across H1 and H2.

Visit rate

No E-Mail: 10.62% (10.21%–11.04%) · Mens E-Mail: 18.28% (17.76%–18.80%) · Womens E-Mail: 15.14% (14.67%–15.63%)

ContrastDifference (95% CI)Holm p
H1Mens E-Mail vs No E-Mail+7.66 pp+7.00 pp to +8.32 pp1.1 × 10⁻¹¹¹raw 5.7 × 10⁻¹¹²
H2Womens E-Mail vs No E-Mail+4.52 pp+3.89 pp to +5.16 pp3.2 × 10⁻⁴⁴raw 3.2 × 10⁻⁴⁴

Conversion rate

No E-Mail: 0.57% (0.48%–0.68%) · Mens E-Mail: 1.25% (1.11%–1.41%) · Womens E-Mail: 0.88% (0.77%–1.02%)

ContrastDifference (95% CI)Holm p
H1Mens E-Mail vs No E-Mail+0.68 pp+0.50 pp to +0.86 pp3.0 × 10⁻¹³raw 1.5 × 10⁻¹³
H2Womens E-Mail vs No E-Mail+0.31 pp+0.15 pp to +0.47 pp1.6 × 10⁻⁴raw 1.6 × 10⁻⁴

Spend among converters · descriptive only

Spend among converters conditions on a post-treatment outcome (conversion), so the groups being compared are no longer randomized; any difference mixes the email's effect with who was induced to convert. Reported for context only; never used for inference or the recommendation.

  • No E-Mail: $114.00 mean spend across 122 converters
  • Mens E-Mail: $113.53 mean spend across 267 converters
  • Womens E-Mail: $121.89 mean spend across 189 converters

3. Variance reduction with CUPED (secondary estimate)

Revenue per customer adjusted for prior-year spend (history), θ = 0.00128 estimated on pooled data. The unadjusted estimate remains the primary result. The adjustment reduced within-arm variance by at most 0.060%.

CUPED produced negligible variance reduction because prior-year spend is only weakly correlated with two-week spend: pooled correlation r = 0.022.

Variance reduction by arm: No E-Mail 0.004% · Mens E-Mail 0.060% · Womens E-Mail 0.054%

  • Unadjusted (primary), 95% CI
  • CUPED-adjusted, 95% CI
H1 · Mens E-Mail vs No E-MailSE ratio 0.9998
Unadjusted +$0.77 (+$0.49 to +$1.05) · adjusted +$0.77 (+$0.48 to +$1.05) · signs agree
H1: unadjusted +$0.77, adjusted +$0.77
H2 · Womens E-Mail vs No E-MailSE ratio 0.9998
Unadjusted +$0.42 (+$0.17 to +$0.68) · adjusted +$0.42 (+$0.17 to +$0.68) · signs agree
H2: unadjusted +$0.42, adjusted +$0.42
Difference in revenue per customer vs No E-Mail, dollars

4. Effects by customer segment (heterogeneity)

Pre-specified dimensions only. For each email and dimension, a joint Wald test asks whether the effect differs across segments; Holm correction across all 8 tests. 0 of 8 tests reject at family-wise α = 0.05. Segment estimates are shown with 95% CIs whether or not the test rejects, in a fixed order, and are not ranked. All panels share one axis.

Prior merchandise purchase (mens x womens)

Mens E-Mail vs No E-Mail
Wald χ²(2) = 2.95, p = 0.228, Holm p = 1.000
Bought bothn = 2,181 / 2,149
+$1.82 (+$0.49 to +$3.16)
Bought both: +$1.82 per customer, CI +$0.49 to +$3.16; n = 2,181 emailed, 2,149 control
Bought men's onlyn = 9,558 / 9,638
+$0.68 (+$0.25 to +$1.12)
Bought men's only: +$0.68 per customer, CI +$0.25 to +$1.12; n = 9,558 emailed, 9,638 control
Bought women's onlyn = 9,568 / 9,519
+$0.62 (+$0.27 to +$0.96)
Bought women's only: +$0.62 per customer, CI +$0.27 to +$0.96; n = 9,568 emailed, 9,519 control
Effect on revenue per customer, dollars
Womens E-Mail vs No E-Mail
Wald χ²(2) = 2.14, p = 0.343, Holm p = 1.000
Bought bothn = 2,118 / 2,149
+$0.16 (−$0.81 to +$1.14)
Bought both: +$0.16 per customer, CI −$0.81 to +$1.14; n = 2,118 emailed, 2,149 control
Bought men's onlyn = 9,622 / 9,638
+$0.28 (−$0.12 to +$0.68)
Bought men's only: +$0.28 per customer, CI −$0.12 to +$0.68; n = 9,622 emailed, 9,638 control
Bought women's onlyn = 9,647 / 9,519
+$0.63 (+$0.30 to +$0.97)
Bought women's only: +$0.63 per customer, CI +$0.30 to +$0.97; n = 9,647 emailed, 9,519 control
Effect on revenue per customer, dollars

New customer

Mens E-Mail vs No E-Mail
Wald χ²(1) = 1.73, p = 0.189, Holm p = 1.000
Existing customern = 10,621 / 10,611
+$0.58 (+$0.14 to +$1.01)
Existing customer: +$0.58 per customer, CI +$0.14 to +$1.01; n = 10,621 emailed, 10,611 control
New customern = 10,686 / 10,695
+$0.96 (+$0.59 to +$1.33)
New customer: +$0.96 per customer, CI +$0.59 to +$1.33; n = 10,686 emailed, 10,695 control
Effect on revenue per customer, dollars
Womens E-Mail vs No E-Mail
Wald χ²(1) = 5.62, p = 0.018, Holm p = 0.142
Existing customern = 10,624 / 10,611
+$0.11 (−$0.28 to +$0.50)
Existing customer: +$0.11 per customer, CI −$0.28 to +$0.50; n = 10,624 emailed, 10,611 control
New customern = 10,763 / 10,695
+$0.73 (+$0.40 to +$1.06)
New customer: +$0.73 per customer, CI +$0.40 to +$1.06; n = 10,763 emailed, 10,695 control
Effect on revenue per customer, dollars

Purchase channel

Mens E-Mail vs No E-Mail
Wald χ²(2) = 2.22, p = 0.330, Holm p = 1.000
Multichanneln = 2,577 / 2,606
+$1.21 (+$0.39 to +$2.03)
Multichannel: +$1.21 per customer, CI +$0.39 to +$2.03; n = 2,577 emailed, 2,606 control
Phonen = 9,240 / 9,327
+$0.56 (+$0.16 to +$0.97)
Phone: +$0.56 per customer, CI +$0.16 to +$0.97; n = 9,240 emailed, 9,327 control
Webn = 9,490 / 9,373
+$0.85 (+$0.40 to +$1.30)
Web: +$0.85 per customer, CI +$0.40 to +$1.30; n = 9,490 emailed, 9,373 control
Effect on revenue per customer, dollars
Womens E-Mail vs No E-Mail
Wald χ²(2) = 3.58, p = 0.167, Holm p = 1.000
Multichanneln = 2,579 / 2,606
+$1.16 (+$0.26 to +$2.06)
Multichannel: +$1.16 per customer, CI +$0.26 to +$2.06; n = 2,579 emailed, 2,606 control
Phonen = 9,454 / 9,327
+$0.23 (−$0.12 to +$0.58)
Phone: +$0.23 per customer, CI −$0.12 to +$0.58; n = 9,454 emailed, 9,327 control
Webn = 9,354 / 9,373
+$0.42 (+$0.02 to +$0.81)
Web: +$0.42 per customer, CI +$0.02 to +$0.81; n = 9,354 emailed, 9,373 control
Effect on revenue per customer, dollars

Zip code class

Mens E-Mail vs No E-Mail
Wald χ²(2) = 0.05, p = 0.977, Holm p = 1.000
Ruraln = 3,243 / 3,139
+$0.76 (+$0.05 to +$1.46)
Rural: +$0.76 per customer, CI +$0.05 to +$1.46; n = 3,243 emailed, 3,139 control
Suburban*n = 9,501 / 9,625
+$0.80 (+$0.35 to +$1.25)
Suburban: +$0.80 per customer, CI +$0.35 to +$1.25; n = 9,501 emailed, 9,625 control
Urbann = 8,563 / 8,542
+$0.74 (+$0.31 to +$1.16)
Urban: +$0.74 per customer, CI +$0.31 to +$1.16; n = 8,563 emailed, 8,542 control
Effect on revenue per customer, dollars
Womens E-Mail vs No E-Mail
Wald χ²(2) = 3.08, p = 0.215, Holm p = 1.000
Ruraln = 3,181 / 3,139
+$0.51 (−$0.12 to +$1.15)
Rural: +$0.51 per customer, CI −$0.12 to +$1.15; n = 3,181 emailed, 3,139 control
Suburban*n = 9,650 / 9,625
+$0.18 (−$0.19 to +$0.55)
Suburban: +$0.18 per customer, CI −$0.19 to +$0.55; n = 9,650 emailed, 9,625 control
Urbann = 8,556 / 8,542
+$0.67 (+$0.25 to +$1.09)
Urban: +$0.67 per customer, CI +$0.25 to +$1.09; n = 8,556 emailed, 8,542 control
Effect on revenue per customer, dollars

* Shown as “Suburban”. The source file spells this level “Surburban”; analysis code and data exports keep the source spelling (see Deviations in the analysis plan).

n = emailed / control customers in the segment. Model: OLS spend ~ T + segment dummies + T x segment dummies, fitted on one email arm plus No E-Mail; HC3 standard errors; T + T x segment linear combination; 95% normal CI from the full HC3 covariance.