Author a coder-eval task

You are writing coder-eval task YAML. The user's request is: $ARGUMENTS

If $ARGUMENTS is empty, ask what the task should test. Do not invent a subject.

Good tasks use simple prompts: state the goal and the expected output, then let the agent work out the approach. A single request can produce several task files — "create tasks for all the registry subcommands" means one task per subcommand.

Step 1 — Understand the request, and check the CLI is there

Run coder-eval --version first. Steps 6 and 7 both shell out to it, and finding that out after writing several task files means the user gets a bare command not found with nothing to act on. Installing this plugin did not install the CLI.

If it is missing, follow ${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md: offer the install, ask before running it, and confirm with coder-eval --version afterwards. Never install unprompted, and do not write any task files if the user declines.

That reference also covers the other half of the version check — whether this project pins a coder-eval version, and what to do when the installed one does not match it.

Then establish:

• What is being tested — which tool, SDK, CLI, skill, or capability?
• How many tasks — one operation, or several?
• Difficulty — smoke, basic, or intermediate?
• Dependencies — network, packages, starter files, external services?

State any assumptions you make rather than silently picking.

Step 2 — Look at what already exists

Find the repository's task tree by following ${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md, and say what you resolved. If a task already covers this ground, say so and offer to modify it instead of adding a near-duplicate.

Repo-local convention beats anything bundled with this plugin — where the two disagree, the repo wins. Before writing, read what the repository declares about task authoring: its own contributor or convention documents, a task template if it ships one, and a few neighbouring tasks. Adopt what you find — naming, tags, thresholds, weights, where files go — and say in your report which conventions you adopted, so the choice is visible rather than implied.

Two limits on that:

• A repository that declares nothing leaves the bundled rubric as the whole answer. Precedence is about deferring to a local rule that exists, not about doing nothing until one does.
• Precedence covers style, not soundness. If a local convention would produce a criterion that cannot fail, the rubric's correctness checks still bite — follow the convention where you can, say plainly where you did not and why.

Step 3 — Design the task

Task ID — lowercase kebab-case, unique, <domain>-<action> (e.g. registry-list-processes).

Initial prompt — minimal. State the goal and the expected output; nothing else.

• Good: "Use the foo CLI to list the available processes and save the result to processes.json."
• Bad: a step-by-step recipe with the exact flags, or a restatement of what the criteria check.

Key rule: prompts instruct, criteria validate. Never leak criteria detail into the prompt. If a criterion checks that the output contains a count field, the prompt must not mention count — otherwise you are testing transcription, not capability.

The subtle version of this, and the easiest to write by accident: a criterion that matches a literal the prompt already dictates. "Use pypdf to read the fields" in the prompt plus a criterion grepping for pypdf is a criterion that cannot fail — the agent was told the answer. Either the constraint is a real requirement (keep it in the prompt, and score what the agent did with it instead) or it is the thing under test (drop it from the prompt). Never both.

(The rubric below carries this same trap as a review-time check, and is the declaration a reviewer applies. The paragraphs above are the authoring-time version: they exist to stop you writing it in the first place.)

Success criteria — read ${CLAUDE_PLUGIN_ROOT}/reference/task-rubric.md before choosing them. It is what this work will be checked against in step 5, and a criterion set designed against it is far cheaper than one repaired after the fact.

Pick by what actually needs verifying:

What to check	Criterion type
File exists, has content, matches a pattern	file_check (prefer over file_exists + file_contains)
JSON structure or specific values	json_check (JSON Schema + JMESPath assertions)
A script runs, tests pass, or a scorer emits a float	run_command
Output resembles a reference solution	reference_comparison
Subjective or open-ended quality	llm_judge
A deep, tool-using verdict on the sandbox	agent_judge (expensive)
The agent used a specific tool	command_executed
Tool-call efficiency against a budget	commands_efficiency
The agent engaged a target skill	skill_triggered (see /coder-eval:check-skill)
A predicted label vs. ground truth	classification_match

Read ${CLAUDE_PLUGIN_ROOT}/reference/criteria.md for each type's exact fields — it is generated from coder-eval's own models, so it is the authoritative field list.

Rules that matter:

• Every task needs at least one criterion that checks output content, not just existence. A suite of file_exists checks passes when the agent writes an empty file.
• Use command_executed sparingly — only when it genuinely matters how the result was produced. Set require_success: true whenever the command's success is what you are grading; the permissive default (false) counts a crashed invocation as evidence the work was done, and survives only for a genuine exception — a probe whose failure is an acceptable outcome.
• When the prompt genuinely must name a literal — a flag like --json, an output filename — a criterion matching that literal is a smoke check, not evidence: it only proves the agent typed back what it was told. Keep it if you like, at a low weight, and put the weight on a criterion that checks the resulting behaviour.
• weight reflects importance: 0.5 nice-to-have, 1.0 standard, 1.5–2.0 critical. weight: 0 makes a criterion informational (reported, but excluded from the score and the pass/fail gate).
• The default pass_threshold: 0.9 is right for most criteria; use 1.0 only for binary checks.
• Omit the agent: block unless the task needs non-default settings. Agent config is resolved from the experiment layer, and hardcoding it in every task defeats experiment-level control such as A/B model comparisons.

Tags — keep them portable: a difficulty tag (smoke, basic, intermediate) plus whatever domain vocabulary the repository's existing tasks already use.

Step 4 — Write the file(s)

One file per task, named after the task ID with underscores (registry-list-processes → registry_list_processes.yaml), in the repository's task directory.

task_id: "<kebab-case-id>"
description: "<one line: what this task tests>"
initial_prompt: |
  <the natural-language request>
tags: ["smoke", "your-domain"]      # a difficulty tag plus the repo's domain vocabulary

sandbox:
  # `tempdir` runs the agent's commands on THIS machine — it isolates the working
  # directory, not the host. For a task that fetches or executes third-party content,
  # use `driver: "docker"` instead; that is the real confinement boundary.
  driver: "tempdir"
  python: {}              # a venv with no extra packages; add env_packages if needed

success_criteria:
  - type: "<criterion_type>"
    description: "<what this checks>"
    # ... type-specific fields
    weight: 1.0

Add template_sources if the task needs starter files (a codebase to modify, a fixture to read).

Step 5 — Could this pass for the wrong reason?

Now re-apply ${CLAUDE_PLUGIN_ROOT}/reference/task-rubric.md to the files you just wrote. Designing against it and checking against it are different acts: the first shapes your choices, the second catches what you actually typed.

Answer the rubric's framing question in writing — what is the cheapest thing an agent could do that scores full marks? — and if that cheapest path does not resemble the work the task claims to test, fix the criteria before going further. Work every section of the rubric, including its fixture-lifecycle section whenever the task touches state outside the sandbox.

Fix what you find here rather than reporting it. Note which checks you applied; step 7 asks for them.

Step 6 — Validate

For each file written, run coder-eval plan <path> and fix everything it reports. It validates through the real Pydantic models, so a mistyped field name or a missing required key surfaces here rather than halfway through a paid run.

Then re-read your own work and check:

• every criterion refers to a file or command the prompt actually leads the agent to produce;
• the prompt leaks no criteria detail;
• at least one criterion inspects content.

A task nobody has ever run is not finished. plan proves the YAML is well formed; it says nothing about whether the criteria can be satisfied, or whether they can be satisfied too easily. Only a run answers that, so once plan exits 0:

State the task count, the agent and model the tasks resolve to, and that a run costs real tokens — then offer to run it and ask. Never run unprompted.

coder-eval run <path>

Then interpret the result rather than reporting it:

• A first run scoring 1.000 is suspicious, not a success. A task written and passed on the first attempt is more often a task that grades something trivial than a task that happened to be perfect. Go back to the framing question in step 5 and re-answer it against the trajectory you now have: what did the agent actually do, and would the cheapest path have scored the same?
• A failing run is a diagnosis, not a prompt edit. Decide first which layer is wrong: something a real user would plausibly have said (fix the prompt), or something the skill or the underlying tool should have supplied (fix that instead, and leave the task failing until it exists). Patching the prompt to route around a missing capability makes the score green and changes nothing for users.
• Never ship a task that cannot pass yet. A task that always fails is noise: it trains everyone reading the suite to ignore a red result. Either withdraw it, or say plainly what has to exist before it is worth scheduling.

If the user declines the run, that is a fine outcome — record it as declined in the report rather than implying the task is validated.

Step 7 — Report

Summarize what you wrote:

File	Task ID	Criteria	Tags	Run verdict

The run verdict is the score from step 6, or an explicit not run with the reason (the user declined, no credentials, a dependency does not exist yet). An empty cell reads as a pass to everyone who sees the table later.

Then:

• Your answer to the framing question — the cheapest path to full marks, and why the criteria do not accept it. One or two sentences, not a restatement of the rubric.
• Which rubric checks you applied, and what any of them changed.
• What the run showed, if it happened — particularly if it scored 1.000 and what you concluded about that.
• Any assumptions you made.
• The command to re-run it: coder-eval run <path> (real tokens, real cost).