Should the retailer send these emails again?

1. Recommendation

Send the Mens E-Mail to every customer: in the full experiment it added $0.77 of revenue per customer (95% CI $0.49 to $1.05), against an assumed send cost of $0.10 per email; on the untouched holdout, no targeting policy beat sending to everyone.

Pre-registered rule (Section 10): “send” if the lower bound of the 95% interval exceeds the cost per email; “promising — test again” if only the point estimate does. The cost per email is an assumption, not a property of the data, and revenue is not profit: see when it stops paying, below.

2. Why

Both emails increased revenue per customer over the two weeks after the send, compared with customers who got no email: each interval lies above zero. All estimates →

Sources: effects_primary.json: contrasts[H1, H2].estimate, .ci_analytic, .p_holm

3. Does targeting help?

Not on this data. On a holdout of customers set aside before any model was trained, no targeted policy beat sending the best email to everyone. Targeting analysis →

The targeting rules chosen on the training split (P3, P4a) turned out to assign every customer the same action as P1. The best targeted policy that did differ (P5) came out −$0.05 per customer against P1 (95% CI −$0.40 to +$0.26). The uplift models ranked customers no better than chance: holdout Qini Mens E-Mail −$0.04, Womens E-Mail −$0.01 per customer.

On the 19,200 holdout customers, sending everyone the Mens E-Mail added +$0.47 per customer (95% CI −$0.12 to +$1.03). That interval overlaps the full-experiment estimate above and is wider, because it uses only the holdout and a policy-value estimator. The holdout exists to compare targeting against blanket sending, not to re-confirm the email effect, which rests on the pre-registered full-sample test in section 2.

Sources: targeting.json: winner, policies[].net_value_minus_best_blanket(_ci), qini_holdout[].coefficient; decision.json: recommendation.holdout.incremental_revenue_vs_p0(_ci)

4. When it stops paying

Net revenue per customer at each assumed cost per email, from the full-experiment estimates. Solid lines use the point estimate; dashed lines use the lower bound of the 95% interval. Sending stops paying where a line crosses zero.

• Mens E-Mail: breaks even at a cost of $0.77 per email at the point estimate, $0.49 at the lower bound.
• Womens E-Mail: breaks even at a cost of $0.42 per email at the point estimate, $0.17 at the lower bound.

Supplementary, not pre-registered: at $0.10 per email, sending pays if gross margin is at least 13.0% (at the point estimate) or 20.6% (at the lower bound of the 95% CI). Presentation aid derived from pre-registered quantities; the data contains no margin.

5. What we saw in the funnel

Both emails lifted visits and purchases: every interval lies above zero. For both emails the point estimate of the relative lift is larger for purchases than for visits, but the purchase intervals are wide and overlap the visit intervals, so that ordering is not statistically established.

• Mens E-Mail: visits +7.66 pp (95% CI +7.00 pp to +8.32 pp), a relative lift of +72% (+64% to +80%); purchases +0.68 pp (+0.50 pp to +0.86 pp), a relative lift of +119% (+72% to +166%).
• Womens E-Mail: visits +4.52 pp (95% CI +3.89 pp to +5.16 pp), a relative lift of +43% (+35% to +50%); purchases +0.31 pp (+0.15 pp to +0.47 pp), a relative lift of +54% (+19% to +89%).

Descriptive only: among customers who visited, the share who bought was 5.4% (No E-Mail), 6.9% (Mens E-Mail), 5.8% (Womens E-Mail). Purchase rate among visitors conditions on a post-treatment outcome (visiting): the email changes who visits, so emailed and non-emailed visitors are no longer randomized, comparable groups. Reported for context only; never used for inference or the recommendation.

Relative lift of each rate (treatment rate / comparison rate - 1) with a delta-method 95% interval. Supplementary presentation quantity added in v1.0.1; not pre-registered and not Holm-corrected.

Sources: effects_secondary.json: metrics.visit_rate / conversion_rate .contrasts[].estimate, .ci_newcombe, .relative_lift; effects_secondary.json: purchase_rate_among_visitors (descriptive_only)

6. Limitations

• Outcomes cover only the two weeks after the send; longer-term effects are unknown.
• One historical send, in 2008. Customers, channels and costs have changed since.
• No guardrail data: unsubscribes, complaints and long-term retention were not recorded.
• No margin data: revenue is not profit, and the cost per email is an assumption.
• Weak pre-period features: prior-year spend correlates with two-week spend at only r = 0.022, which limits both variance reduction and targeting.

Sources: cuped.json: correlation_spend_history.pooled; targeting.json: qini_holdout[].coefficient

7. How we know

• The pre-registered plan, committed before any data was loaded (48c63f4).
• Data-quality checks: integrity, sample ratio, balance.
• Effect estimates, every one with an interval.
• Targeting analysis: 19,200 holdout customers, evaluated once.
• How it was built with Claude Code, including the correction log.