Immunising an analysis

Two situations, two answers.

A. Before the data are analysed

Pre-specify one specification, and pre-specify the grid around it.

A pre-analysis plan that names a single specification is good. One that also names the set of defensible alternatives is better, because it converts every later robustness check from a discretionary choice into a committed one. Write the design card (02-forking-paths) with a preregistered block and a direction, deposit it, and let phack size print its key and its multiplicity. The card is the pre-analysis plan for the analytical part.

Contents that matter, in order of how often they are omitted:

1. primary outcome, named singly, with its exact construction
2. the estimating equation, including fixed effects, the estimator (under staggered adoption: which one, and why) and the clustering level
3. sample inclusion rules, the exact date window, and the comparison group
4. the outlier rule and the missing-data rule
5. the subgroups to be examined, and that they are secondary
6. the multiplicity correction to be applied across outcomes and subgroups
7. the sign the hypothesis predicts, so a two-sided test cannot be swapped in later
8. what would count as a null result

Split the sample. Where N allows, hold out a confirmation sample and lock it. Explore freely in the training half; the held-out half gets exactly one analysis, the one that exploration selected. This converts a search into a legitimate two-stage design and is by far the strongest available protection. It is also measurable: --procedure split_sample --stage holdout runs it, and the null replay prices it (09-search-procedures). On the null panel an exhaustive search over 400 specifications reports p < .05 on 86% of null datasets when the pilot estimate is reported, 45% when the pilot is pooled into the confirmatory sample, and 10% on the held-out units alone; a greedy walk from the pre-analysis plan with a continuation rule at pilot p < .10 reports on the held-out units at 4.8%. Two conditions, both from Adda, Decker & Ottaviani (2020): the confirmatory sample must be fresh — pooling the pilot in keeps half the inflation, and "we explored on half and confirmed on everything" is optional stopping — and the held-out estimate must be run with a fixed inference doctrine, because a pilot that picks the smallest p picks anti-conservative standard errors, and fresh data does not cure a standard error that under-states uncertainty everywhere (on this panel the held-out estimate of a pilot-chosen hc1 specification rejects 19% of the time; a unit-clustered one, 5.6%). Selective continuation — running the confirmatory stage only after a promising pilot — is fine on its own: it changes which projects a registry sees, not the size of their tests.

Blind the analysis. Analyse with the treatment label permuted, fix every analytical choice, then unblind. Everything decided under blinding is pre-specified by construction. search._draw_null is exactly this permutation and can be used to produce the blinded dataset.

B. After a search has happened

The search is not the problem. Reporting one point from it as a confirmatory test is. Five escalating repairs, all of which phack search performs at once:

1. Report the curve, not the point

Median coefficient, interquartile range, share significant, share flipping sign, and the full ledger in a repository.

2. Test the curve, not the point

The Simonsohn–Simmons–Nelson joint tests (ssn_joint): is the median effect, the share of significant specifications, or the share significant in the predicted direction unusual against the null re-runs of the same curve? These are the inferential statements a multiverse supports.

3. Correct the point for multiplicity, using the dependence

Bonferroni over S specifications is valid but badly conservative, because specifications reuse the same rows. Romano–Wolf stepdown bootstraps the joint distribution of the family and controls FWER while exploiting the correlation. effective_tests (Li & Ji) tells you how much over-correction you avoided.

4. Calibrate the whole search

The strongest repair. Re-run the identical search on data where the null holds by construction, and report the share of null datasets on which the search found something at least as significant. That is min_p_test.honest_p, and it needs no assumption about how the specifications are correlated because it re-runs the procedure rather than modelling it.

If the search was sequential rather than exhaustive — it usually was — replay that procedure (09-search-procedures): procedure_test.honest_p is the calibration for what was actually done, and null_share_reporting_significant is its false-positive rate.

python scripts/phack_cli.py search DATA CARD --direction + \
    --null-draws 500 --null-scheme cluster_permute --n-jobs 8

Pick the null scheme to match the design — cluster_permute for a panel with unit-level or staggered treatment, permute_within_time for within-period assignment. A mis-specified null gives a too-tight reference distribution and an honest p-value that is not honest.

5. Randomisation inference for the reported specification

Where treatment assignment is known, permutation inference on the single reported specification is exact and assumption-free about the error structure. The null draws already contain it: the column of p_null.npy for the reported specification is its randomisation distribution. It does not fix multiplicity — combine it with 3 or 4.

Two diagnostics worth reporting even when nothing is significant

Distance from pre-registration (nearest_significant). How many analytical choices separate the committed specification from the nearest significant one, and which. Distance 1 is a design whose conclusion rests on a single defensible-looking choice; say so.

Attribution (axis_influence). Which axis has the largest spread in the share significant across its levels. If it is the standard-error doctrine or the inference mode, the finding is about inference, not about the world.

The honest report

phack search writes report.md from the audit; phack report RUN_DIR regenerates it. It contains, in order: what was searched and its multiplicity; the curve and the joint tests; the pre-registered specification and where it sits; the most favourable specification, with every correction from "as reported" down to the null-calibrated value; which choices did the work; and a conclusion that follows from the honest p rather than the reported one. The paragraph it produces reads like this:

We estimated the effect under 301 defensible specifications, enumerated in advance and listed in the replication package. The pre-registered specification gives β̂ = −0.050 (SE 0.064, p = 0.44). Across the family the median estimate is −0.028, 1% of specifications are significant at 5% and none in the predicted direction; the Simonsohn joint test on the share significant gives p = 0.56. The most favourable specification gives β̂ = 0.214 (one-sided p = 0.036) and is eight analytical choices away from the pre-registered one; calibrated against 200 re-runs of the identical search on null data, the probability of a result at least this significant is 0.59. We therefore do not treat the result as evidence of a non-zero effect.

That paragraph is publishable and it is honest. The temptation it resists is reporting only the sixth sentence — and because the report is generated from the ledger, the sixth sentence cannot be produced without the others.

What immunisation cannot do

None of this makes a fragile result robust. It makes fragility visible and correctly priced. If the honest p-value is 0.59, the analysis has not found an effect, and the right conclusion is that the design cannot resolve one — which is itself worth reporting, and which pre-registration makes publishable.