Detecting p-hacking in a collection of results

What can and cannot be established

Every test here is a test on the distribution across many results. None of them can establish that any individual paper was p-hacked, and the report says so on every run. Treat a flag as a reason to look, never as a finding about a person.

The theory in one paragraph

Elliott, Kudrin & Wüthrich (2022) show that in the absence of p-hacking the density of p-values across studies is non-increasing on (0, α), and for tests based on asymptotically normal statistics it is also continuous and bounded above. p-hacking breaks these: it moves mass from just above the threshold to just below, producing a hump, a discontinuity, or a non-monotonicity. Each testable implication gives a test.

Running

python scripts/phack_cli.py detect stats.csv --pcol p --zcol z

Input is one row per reported result. Supply p-values, z-statistics, or both; whichever is missing is derived.

The battery

Shape tests — clean nulls, trust these:

Test	Null	Small p-value means
binomial	≥ half of significant p-values fall below α/2	left-skewed p-curve; no evidential value
fisher	p-curve is uniform or right-skewed	left skew
stouffer	as Fisher, normal aggregation	left skew
lcm_monotonicity	density non-increasing on (0, α)	the curve humps; bootstrapped against the least favourable null

Bunching tests — weaker, read with care:

Test	What it compares
discontinuity	mass just below p = .05 against a Poisson log-linear counterfactual fitted outside a donut
caliper	mass just above z = 1.96 against the same kind of counterfactual

Why the counterfactual matters. The classical caliper test asks whether more than half the statistics in a window around 1.96 lie above it. That null is right only when the density is locally flat. When a real effect is present the density of |z| is rising through 1.96, so an honest literature fails the naive test. On simulated honest data with a true effect, the naive caliper reports 65% above and p = .011 — a false positive. The counterfactual version reports p = .11 and correctly declines to flag. Both are printed; naive_share_above is kept for comparability with the published literature, but p_value is the counterfactual one.

Threshold signatures — Adda, Decker & Ottaviani (2020) on 12,621 registered clinical-trial p-values found no spike past z = 1.96 anywhere, but a discontinuity at 1.96 in phase III for small industry sponsors. Those are different things, and two tests separate them:

Test	Counterfactual	Small p-value means
density_jump	local polynomial density of |z| on each side of 1.96 (Cattaneo, Jansson & Ma 2020), bootstrap SE	the density is higher just past the line than just before it
spike	log-linear Poisson fit on [1.96 + 0.2, 1.96 + 0.8] extrapolated over the caliper, with the extrapolation's own variance	mass just past the line is out of line with the density beyond it

Read them together. A spike with or without a jump is results pushed across the line: p-hacking. A jump without a spike is a level shift: the results below the line are missing, not moved — selective reporting, the small-sponsor pattern in the paper. Neither, with a rising share significant between stages of a project, is selection between stages, which no threshold test can see. report() prints this reading as threshold_signature. The classical caliper fits its counterfactual through the threshold and so flags any discontinuity alike; it is kept for comparability.

On simulated ground truth: withholding half the non-significant results gives a density jump with p < 0.001 and a clean spike test; pushing half the results in [1.6, 1.96) to just past 1.96 gives a spike with p < 0.001. Over 30 draws of a half-normal with n = 3,000 and of a null/effect mixture with n = 1,800, the spike test rejects at 5% on 3% and 3% of draws; the jump test on 30 draws of the rising-density literature above rejects on 5%.

pcurve_power estimates the average power behind the significant results. A value near α means the significant findings carry no evidential value at all.

Across stages: selection is not manipulation

The paper's central finding is that the share of significant primary outcomes rises from 45.7% in phase II to 70.6% in phase III for industry sponsors (34.7% to 34.8% for non-industry), with a smooth phase III distribution. Sponsors continue to phase III after promising phase II results; the later stage inherits the selection. That is rational, possibly socially desirable, and invisible to every test above. Three functions handle it:

# stats.csv: one row per result, a column with two stage labels, and on the
# early-stage rows a 0/1 column saying whether the project continued
python scripts/phack_cli.py detect stats.csv --zcol z --stagecol phase --contcol continued

• phase_shift_test(z_early, z_late) — share significant by stage, a two-proportion test, and a one-sided KS test that the later stage stochastically dominates.
• continuation_decomposition(z_early, continued, z_late) — the paper's method: a logit of continuation on the early-stage |z|, the early-stage distribution reweighted by the fitted continuation probabilities, and the later stage's excess split into the part the continuation rule explains and the part it leaves unexplained, with a bootstrap SE on the latter. In the paper the rule explains ≈ 85% of the phase III excess for the ten largest sponsors and ≈ 29% for small ones; the unexplained 18 points for small sponsors is what is left for selective reporting.
• phase_report(...) runs both stages through the threshold tests, the phase shift and the decomposition and prints a signature.

The decomposition's identifying assumption is the paper's: conditional on continuing, the expected later-stage z equals the earlier-stage z. It holds when the sponsor's decision uses information of which the registered z is a noisy reflection, and it fails when continuation is a deterministic function of the registered z itself (the logit separates; the function says so). simulate.continuation_shift generates a population that satisfies it — heterogeneous true effects, a logistic continuation rule on the sponsor's own read of the pilot, an optional probability conceal of withholding a non-significant confirmatory result — and is what the battery is validated on: with no concealment the rule explains 90–100% of the phase III excess and the later stage shows no threshold signature; with half the null confirmatory results withheld, 19 points are left unexplained (p < 0.001) and the later stage shows a density jump with no spike.

python scripts/phack_cli.py simulate --continuation --n-sims 6000 --conceal 0.5

Reading the verdict

The verdict weights shape tests above bunching tests, because only the shape tests have clean nulls:

• strong evidence — two or more shape tests flag
• moderate — one shape test plus bunching
• weak / ambiguous — bunching only — this is the interesting case. It is what a mixed literature looks like: genuine effects dilute the shape tests while threshold-seeking still leaves a bunching signature. In simulation, a literature that is 5/6 null with mild threshold-pushing shows excess mass of 2.3× at the caliper while every shape test comes back clean.
• no distributional evidence — the tests found nothing; this is not proof of absence, particularly with few results

Validation

The battery is calibrated on simulated data with known ground truth:

Data	Shape tests	Bunching	Estimated power	Verdict
honest, real effect	clean	clean	0.80	no evidence
p-hacked, true null	all four flag	both flag	0.05	strong evidence
mixed	clean	both flag	0.30	weak / ambiguous

Reproduce from tests/test_detect.py.

Sample-size guidance

The shape tests need at least ~15 significant results and are underpowered below ~50. The bunching tests need enough mass near the threshold to fit a counterfactual and return a note rather than a p-value when they do not; the density-jump test wants ≥ 30 statistics within its bandwidth on each side, and the decomposition ≥ 30 early-stage projects with variation in continuation. Elliott et al. and the follow-up work on when p-hacking is detectable are clear that none of these tests has much power against mild hacking; a clean report from a small collection means very little.