Does targeting beat sending to everyone?
Seven policies, each fixed before the holdout was opened, compared on 19,200 customers who were set aside before any model was trained. Net value = revenue per customer minus an assumed $0.10 per email sent.
1. Policy comparison on the holdout
Winner: P1. P1 (a blanket policy) has the highest holdout net value. A targeted policy could replace the best blanket policy (P1) only if the paired 95% interval of their difference lay entirely above zero.
Assigns every customer the same action as P1.
Assigns every customer the same action as P1.
vs P1: −$0.17 (−$0.79 to +$0.38)
vs P1: −$0.05 (−$0.40 to +$0.26)
Estimator: nominal IPW, V = mean(spend * 1{T = pi(x)} * 3); NV = V - cost * share sent; 2,000 paired bootstrap resamples. Robustness check (Hajek (self-normalized by holdout arm shares); robustness check only): agrees in sign with the primary estimator for every policy.
2. Can the models rank customers by uplift?
A Qini curve shows cumulative incremental revenue when customers are targeted in order of predicted uplift. A model that ranks well bows above the straight line of random targeting; the coefficient is the area between them, per customer.
- Model ranking
- Random targeting
- Model ranking
- Random targeting
Out-of-fold diagnostics use hyperparameters tuned on the same training data, so they are mildly optimistic.
3. Net value by targeting depth (descriptive)
k was chosen on the training split (Mens 100%, Womens 100%). These holdout curves are shown for context only; picking k from them would reuse the holdout.
4. The segment rule (P3), chosen by cross-validation
One candidate rule per pre-specified dimension, each giving every segment the action with the highest training net revenue; the candidate with the best 5-fold cross-validated net value on the training split was selected, with no human choice.
- Prior merchandise purchase: CV net value +$1.37 per customerselected
- New customer: CV net value +$1.37 per customer
- Purchase channel: CV net value +$1.32 per customer
- Zip code class: CV net value +$1.34 per customer
The top candidates tied exactly because their rules made identical assignments; the pre-registered tie-break (dimension order) selected the first.
Selected rule (Prior merchandise purchase): Bought both → Mens E-Mail; Bought men's only → Mens E-Mail; Bought women's only → Mens E-Mail.
Training split: 44,800 customers. Hyperparameters chosen by 5-fold CV: Mens E-Mail depth 3, rate 0.05, 100 trees, leaf 200; Womens E-Mail depth 3, rate 0.05, 100 trees, leaf 200; No E-Mail depth 3, rate 0.05, 100 trees, leaf 200. Uplift-assignment policy on training (out-of-fold): Mens E-Mail 64.0%, Womens E-Mail 26.5%, No E-Mail 9.5%.