IEEE S&P Experiments

Use this to build or audit the evidence an S&P paper stands on. The venue's reviewers are professional adversaries; an evaluation that would satisfy a systems or ML PC often leaves an Oakland reviewer's central question — "does this survive a real, adaptive attacker?" — unanswered.

Match the evaluation to the contribution type

Contribution	Evaluation that closes the loop	Fatal gap
Attack	End-to-end demonstration against a realistic, current target	Toy target; unrealistic preconditions
Defense	Adaptive adversary who knows the design; cost/overhead	Only non-adaptive or prior attacks
Measurement	Representative sampling + validation + ethics	Convenience sample presented as population
System	Security property demonstrated and performance	Property asserted, not tested
SoK	Systematic re-analysis under one framework	Cherry-picked coverage

The adaptive-adversary rule dominates defense papers

A defense evaluated only against existing or non-adaptive attacks is the most common S&P defense rejection. The standard:

• Define the adaptive adversary explicitly: knows the mechanism, the parameters, and the deployment.
• Show your defense against attacks designed to break it, not just yesterday's attacks it happens to stop.
• Report the cost of adaptation for the attacker and the overhead for the defender — both are part of the security claim.
• If a class of adaptive attack is out of scope, say so in the threat model and own the boundary; do not leave it for a reviewer to discover.

Measurement validity is an evidence question and an ethics question

For measurement papers, the sampling story and the ethics story are the same paragraph in reviewers' minds:

• State the population, the frame, and the sampling method; quantify coverage and bias.
• Validate a subsample by an independent method where possible.
• Active measurement (scanning, probing) must respect opt-out norms, rate limits, and the ethics record (ieeesp-review-process) — an experiment that harms the systems it measures is a reject regardless of results.
• Human-subjects components need IRB determination before running, not a post-hoc note.

Statistics for attacks and fuzzing

Security evidence is often probabilistic and gets held to a real bar:

Attack-success reporting:
  n trials (state n) · success rate ± dispersion · target set described
  → "worked" without n is an anecdote, not a result

Fuzzing / bug-finding comparison (the field's known pitfalls):
  - equal budgets (CPU-time, not wall-clock)
  - ≥ 5–10 campaigns per configuration; report variance
  - identical seed corpora across compared tools
  - a ground-truth or triage method for "unique" bugs
  → a single-run bug count comparison is not evidence of superiority

Timing / side-channel:
  noise floor stated · machine quiescence (isolated cores, freq pinning)
  · distinguisher's statistical test named

Baselines and ablations Oakland reviewers ask for

• The strongest prior attack/defense, at its best configuration, not a weakened reimplementation.
• An ablation isolating the component you claim is responsible for the security gain.
• A cost baseline: what does the attacker/defender spend, and is it realistic at the claimed scale?
• Negative results where they bound the claim (attack fails against target class Y — state it; it strengthens the scoped claim).

Ethics as an experimental design constraint, not an afterthought

Some experiments are simply not runnable as first imagined:

• Testing an exploit against live third-party systems without authorization is out; build a representative testbed instead.
• Collecting user data beyond what the IRB and the ethics record cover is out.
• Disclosure timing constrains when certain measurements can be published — design the timeline so the evidence and the fix do not collide (ieeesp-reproducibility).

Audit worksheet

For each experiment:
  claim it supports | contribution type | adaptive adversary evaluated? |
  strongest baseline used? | n trials + dispersion | ethics clearance |
  realistic target? | rerunnable? (→ ieeesp-reproducibility)
Flag any row with: non-adaptive-only defense · anecdotal success rate ·
  weakened baseline · unmet ethics precondition

Output format

[Contribution type] attack / defense / measurement / system / SoK
[Loop closed?] <the demonstration/eval that proves the claim> ✓/✗
[Adaptive adversary] evaluated ✓/✗/n-a — scope stated?
[Baselines] strongest prior used ✓/✗ · ablation ✓/✗ · cost baseline ✓/✗
[Statistics] trials+dispersion ✓/✗ · fuzzing pitfalls avoided ✓/✗/n-a
[Ethics preconditions] IRB ✓/✗/n-a · authorization ✓/✗ · disclosure timing ok ✓/✗