Mapping the garden

Before anything is estimated, work out how large the space of defensible analyses is. That number is the multiplicity that honest inference has to pay for, and it is almost always larger than researchers expect.

Step 1 — read the design-specific map

references/econ-dof-maps.md lists, per design, the choices a referee would accept without comment. Take only the ones that are genuinely defensible for this dataset: a bandwidth grid is real for an RDD and meaningless for an RCT.

Step 2 — write a design card

phack init DATA [--design …] [--treatment …] [--outcome …] drafts one: panel keys, the treatment, a control pool, fixed-effect and clustering menus, a window split, staggered-adoption axes when the treatment is absorbing, and the conventional specification as preregistered. Every guess is listed in its notes; the draft is a starting point, not a card. schema/design-card.schema.json is the formal schema and the loader validates against it when jsonschema is installed (phack schema prints it).

A design card is JSON. Each key is one axis of the grid; omitting a key collapses that axis to a single default, which is how a pre-registered analysis is encoded.

{
  "name": "example-did",
  "design": "did",
  "direction": "+",
  "outcomes": ["y", "y_alt"],
  "treatment": "treat",
  "controls_pool": ["x1", "x2", "x3", "x4"],
  "control_policy": "all_subsets",
  "fixed_effects": [[], ["unit"], ["unit", "year"], ["region", "year"]],
  "vcov": ["hc1", "cluster", "twoway"],
  "cluster": [null, "unit", "region", ["unit", "year"]],
  "outcome_transforms": ["level", "log", "std"],
  "outlier_rules": ["none", "sd3", "iqr1.5"],
  "outlier_basis": "residual",
  "imputation": ["listwise", "mean"],
  "subsamples": {"early": "year < 2010", "late": "year >= 2010"},
  "weights": [null, "pop"],
  "did_estimators": ["twfe", "did2s", "stacked"],
  "comparison_groups": ["all", "drop_never_treated"],
  "panel_unit": "unit",
  "panel_time": "year",
  "preregistered": {
    "outcome": "y", "controls": ["x1", "x2"], "fe": ["unit", "year"],
    "vcov": "cluster", "cluster": "unit", "y_transform": "level",
    "outlier_rule": "none", "imputation": "listwise", "subsample": "full",
    "weight": null, "did_estimator": "twfe", "comparison_group": "all"
  }
}

Full key reference: grid.DEFAULTS in scripts/phack/grid.py. The loader rejects unknown keys rather than silently ignoring them, because a typo that quietly drops an axis makes the multiplicity count wrong.

Axes by design

Axis	Card key	Applies to	Notes
outcome definition	outcomes	all	strategy 1
control set	controls_pool × control_policy (none / nested / leave_one_out / all_subsets)	all	strategy 5; all_subsets is 2^k
fixed effects	fixed_effects	ols / did / iv	strategy 14
SE doctrine	vcov × cluster	all	strategy 13; a cluster variable only pairs with a clustered vcov, two-way only with a pair
transforms	outcome_transforms, treatment_transforms	all	strategy 7; treatment transforms include the discretisers median_split, above_mean, quartile_top, tercile_extremes (strategy 8)
outliers	outlier_rules, outlier_basis (outcome / treatment / residual)	all	strategy 4; residual trims on studentised residuals
missing data	imputation	all	strategy 10
sample	subsamples (name → pandas query)	all	strategy 15
weights	weights (column names, null = unweighted)	all	strategy 21; applied as WLS, including inside FE absorption
timing	lags	panel	strategy 22
interactions	interactions	all	extra "a*b" regressors
DiD estimator	did_estimators: twfe / did2s (Gardner two-stage) / stacked (Cengiz et al.)	did	strategy 18; non-TWFE estimators fix their own FE and collapse the FE axis
comparison group	comparison_groups: all / drop_never_treated / drop_always_treated	did	strategy 15; the "forbidden comparison" lever
stack window	stack_window [pre, post]	did / stacked	periods around adoption
event study	event_windows [[leads, lags], ...] × reference_periods × event_estimands (avg_post / lag0 / lag1 / avg_pre)	did / twfe	strategy 19; endpoints binned, reference period omitted; avg_pre is the placebo and is flagged when reported as the effect
bandwidth	bandwidth_selectors (rot / ik) × bandwidth_multipliers, or absolute bandwidths	rdd	strategy 16
kernel / polynomial / donut	kernels, poly_orders, donuts	rdd	strategy 16
RDD inference	rdd_inference: conventional / bias_corrected / robust	rdd	strategy 23; bias_corrected is the under-covering combination and is flagged
instruments	instruments_pool × instrument_policy	iv	strategy 17
IV estimator	iv_estimators: 2sls / liml	iv	strategy 17; Anderson–Rubin p recorded per spec
direction	direction: "+" / "-" / null	all	the one-sided sign the search is after; changes which spec is "best" and the calibration

Cohorts for the staggered estimators are inferred from each unit's treatment path, never read from a column, so a null draw that permutes paths stays internally consistent.

Constraints the enumerator enforces for you: cluster / vcov pairing, no fixed effects or two-way clustering in a local-polynomial RDD, no HC2/HC3 in IV, discretised treatments only with the plain TWFE path, and duplicate specifications collapsed by content hash.

Step 3 — size it before you walk it

python scripts/phack_cli.py size CARD

{"n_specs": 25920, "dimensions": {"outcome": 3, "controls": 16, "fe": 4, "vcov": 3, ...},
 "n_varying_axes": 10, "log10_specs": 4.41, "preregistered_key": "e88cbfc3e0e7"}

Sizing is the deliverable on its own. A design admitting 25,920 defensible analyses cannot support a 0.05 threshold on any single one of them: under the null the smallest of that many correlated p-values is routinely below 0.001. Say that number out loud before estimating anything.

Step 4 — mark the pre-registered path

Write the preregistered block as a dict of axis values (any subset of grid.AXES; unpinned axes default to the first level listed on the card, which is the convention for "the default choice"). size resolves it to a 12-character key and refuses if it matches zero or several specifications. Every later command — search, audit, report — reads the anchor from the card, so there is no key to copy by hand.

Without this anchor there is no way to distinguish a search from an analysis: prereg_departure, nearest_significant and the honest report all hang off it.

Thinning

--max-specs K walks an evenly spaced subset of a very large grid. Thinning always keeps the pre-registered specification and is recorded in manifest.json; the honest p-value is then computed on the grid that was actually walked, never on a different one.

Honest defaults when building a card

• Include the ugly options. A grid containing only the specifications you like understates multiplicity and overstates robustness.
• Exclude the indefensible ones. A grid padded with analyses no referee would accept inflates the correction and lets a weak result hide behind a large denominator. Both directions are cheating. The pathology flags exist for the options that are citable but wrong — non-PSD two-way clustering, bias-corrected RDD estimates with conventional SEs, weak-instrument specifications — which belong in the grid and in the ledger, flagged.
• Weighting, timing, comparison group and aggregation are axes too. They are the most commonly forgotten and among the most consequential.
• Declare the direction. A one-sided search is a smaller and more dangerous object than a two-sided one; the card should say which it is.

Ground-truth cards that ship

card	design	specs	what it exercises
eval/data/null_panel_card.json	did (within-unit treatment)	25,920	controls, FE, SE doctrine, transforms, outliers, imputation, windows, weights
eval/data/null_staggered_card.json	did (staggered adoption)	3,456	estimator choice, comparison groups, weights, windows
eval/data/null_staggered_event_card.json	did (event study)	1,200	event window, reference period, estimand, comparison group
eval/data/null_rdd_card.json	rdd	20,736	bandwidth selector × multiplier, kernel, polynomial, donut, inference mode
eval/data/null_iv_card.json	iv	672	instrument subsets, 2SLS / LIML, controls, FE, AR test

All four have a true effect of exactly zero. scripts/make_null_data.py regenerates the staggered panel; the other three are fixed so documented numbers stay reproducible.