Email Experiment Readout

How it's built

A case study in using an AI coding agent for analysis that can be checked. The order in which things happened is the evidence, so the timeline below is read from the repository's git history rather than written by hand.

Timeline from git

  1. Pre-registration: analysis plan committed
    48c63f4 · 2026-09-23T22:45:22Z
    pre-registration: analysis plan and project conventions (before data access)
  2. First data access: data loader committed
    6dfc24f · 2026-09-23T22:57:40Z
    Step 3: data loader with provenance and canonical cross-check
  3. Plan Deviations entry
    dbc384b · 2026-09-23T23:05:21Z
    Step 4: data-quality gates (integrity, SRM, balance) and planning power grid
  4. Outcome unlock (Sections 6–8)
    cf44832 · 2026-09-23T23:41:58Z
    outcome unlock: sections 6-8 (sprint 2)
  5. Plan Deviations entry
    7a3e092 · 2026-09-23T23:47:27Z
    plan deviation: heterogeneity dimension 1 has three levels; sections 6-8 method clarifications
  6. Plan Deviations entry
    363e98b · 2026-09-24T00:54:16Z
    plan deviation: sections 9-10 targeting and decision rules (before any split)
  7. Holdout sealed (hash committed before training)
    3731c5a · 2026-09-24T00:58:42Z
    split and seal holdout (before any model training)
  8. Holdout evaluated (single use)
    c989424 · 2026-09-24T01:14:55Z
    holdout evaluation (single use)
  9. Plan Deviations entry
    e06cca2 · 2026-09-24T03:49:47Z
    plan deviation: project renamed from LiftLab to Email Experiment Readout (plan content unchanged)

Division of labor

  1. Planning in chat

    The business question, hypotheses, metrics, gates, decision rule and every sprint brief were worked out in conversation with Claude and committed as files. Decisions the plan left open were settled by the human owner, in writing, before the data that could influence them was visible.

  2. Execution in Claude Code

    Claude Code carried out each sprint brief inside the repository under binding rules: no hand-typed numbers, outcomes unlocked section by section, targeting only on the training split, a manifest on every export. It stopped and asked whenever the plan was ambiguous or a check failed, rather than working around it.

  3. Verification by tests and review

    Every published number is checked against an independent computation: DuckDB SQL, scipy and statsmodels, published worked examples, synthetic data with known answers. Guard tests enforce the locks. Screenshot review and human review caught what tests could not, and every catch is logged.

The correction log, by the numbers

17 errors were caught and recorded, each committed together with its fix. The counts below are parsed from ai-workflow/correction-log.md by code, not typed.

Where the error came from

  • Claude Code12
  • Claude Chat5

How it was caught

  • Human review5
  • Claude Code self-review4
  • Claude Code pre-check or plan review (before results)3
  • Screenshot review3
  • Automated test or guard2
  • Sprint 1 · CLAUDE.md specified a Python runtime that was not installed (Claude Chat · Claude Code pre-check or plan review (before results))
  • Sprint 1 · Editable install line in requirements.txt resolved against the wrong directory (Claude Code · Claude Code self-review)
  • Sprint 1 · Unverified explanation of the dataset hash difference (Claude Code · Claude Code self-review)
  • Sprint 1 · Hand-typed numbers in website copy (rule 1) (Claude Code · Claude Code self-review)
  • Sprint 1 · Display defects on /checks (Claude Code · Screenshot review)
  • Sprint 2 · Balance SMD used sample variance for binary covariates (Claude Code · Human review)
  • Sprint 2 · Tables clipped at true phone width on /checks (Claude Code · Screenshot review)
  • Sprint 2 · Plan specified a four-cell mens × womens crossing that the population cannot contain (Claude Chat · Claude Code pre-check or plan review (before results))
  • Sprint 2 · Chart axis labels misstated tick values on /results (Claude Code · Screenshot review)
  • Sprint 3 · Planned winner selection would have chosen k on the same holdout used to judge it (Claude Chat · Claude Code pre-check or plan review (before results))
  • Sprint 3 · First split module exposed holdout indices around the guard (Claude Code · Automated test or guard)
  • Sprint 3 · Qini test asserted a guessed threshold instead of a derived value (Claude Code · Automated test or guard)
  • Sprint 3 · Rule-9 guard test is lexical and flagged a non-targeting module (guard-design limitation) (Claude Code · Human review)
  • Sprint 3 · Commit bbeb180 was made with a failing test because the gate checked the wrong exit code (Claude Code · Human review)
  • v1.0.1 · Funnel interpretation on the readout was contradicted by the data (Claude Chat · Human review)
  • v1.0.1 · Correction-log parser silently dropped entries not labelled "Sprint N" (Claude Code · Claude Code self-review)
  • v1.0.2 · Project name collided with an existing company in the same domain (Claude Chat · Human review)

Classification: Claude Chat if the 'What was produced' field names Claude Chat, otherwise Claude Code; how-caught categories are keyword rules exported alongside the counts.

Read the record