reading-a-ci-verdict

SkillDev tools

Use when reading CI results, check runs or reviewer verdicts on a pull request: deciding whether the required jobs actually ran, why a job is skipped or missing, whether a workflow was triggered at all, whether a review bot really reviewed, or before merging anything on green.

Available today. Use it from your connected AI after setup.

No other account needed.

Connect ahel once, and every AI you use reads what you have installed.

Then ask your AI: use the reading-a-ci-verdict skill

What this skill tells your AI

The instructions your AI receives, as published by nextlyhq/nextly in .claude/skills/reading-a-ci-verdict/SKILL.md and read by ahel’s review.

Green means nothing reported a failure

That is a weaker statement than "the checks passed", and the gap between them is where this file lives. the verifying-merged-work skill answers did my code land; this one answers was anything actually run. Both have been verified correctly on the same PR while it was broken, because they are different questions and each reads complete on its own.

Every section below is a way the second question comes back green with no job having reported anything. They are not variations on one mistake: a dropped trigger, a base that filters the workflow out, a dependency that skipped its dependents, and a run bound to a commit that no longer describes reality each produce an identical rollup, and none produces a red anywhere to notice.

The one habit that covers all of them: decide which jobs this commit REQUIRES, then assert success on each of those. Absence of failure is not a verdict — queued, in_progress, skipped and never triggered all satisfy it.

set -o pipefail
gh api "repos/nextlyhq/nextly/commits/$PR_HEAD_SHA/check-runs?per_page=100" \
  --paginate --jq '.check_runs[]|"\(.status)/\(.conclusion // "none")\t\(.name)"' \
  | sort || { echo "check-runs query FAILED — not clean" >&2; exit 2; }

That command REPORTS; it does not gate. sort exits 0 for any successful response, including one whose rows are all queued, and including an empty check_runs array — so piping it into a merge decision reproduces the false clean this file is about. Read it, then assert success on each job the scope decision requires, and exit nonzero otherwise.

Query the PR HEAD sha, not the merge sha. Measured on this PR, commits/<head>/check-runs returned 31 rows and commits/<merge_commit_sha>/check-runs returned 0.

That is about where the check RECORDS are attached, not about what CI executed. The two are easy to run together and they are different: those workflows check out refs/pull/N/merge, so what was BUILT AND TESTED is your branch merged with the base, while the resulting check-runs are attached to the PR HEAD. So the head is where to look them up, and "the head tree passed" is not what a green check means — it means the merge of head and base passed. (Inside a workflow the relationship inverts, but only for pull_request: there github.sha IS the synthetic merge commit. On pull_request_target it is the tip of the BASE branch — main for an ordinary PR, and the PARENT FEATURE BRANCH for a stacked one, so labeler.yml and pr-title.yml see whichever the PR targets rather than the PR's own revision. On push it is the pushed commit. Do not carry a variable named SHA between contexts, and qualify the claim by event AND by base before relying on it.)

A pipeline reports the LAST command's status, so any check read through one is UNREAD. Above, without pipefail the status is sort's, so an authentication failure, a rate limit or a transient 5xx from gh yields an empty list AND a success status — the precise false-clean this file exists to prevent, reproduced by the command recommending against it.

That is one instance of a property with three now, across three mechanisms, so it is worth stating as the property rather than as a note about one command:

status lost to the pipe. A test run piped for readability reported "42 passed" from a run that had exited 1.
status lost to the pipe, and merged. A comment-convention run read through tail -1 printed a summary line while exiting 1. The pass was recorded, the pull request merged, and main was red on that check until someone else's pull request inherited the failure and reported it.
OUTPUT lost to the pipe. A refusal printed below the twelfth line was cut off by head -12 and the run read as clean. That one is in the derived-checks skill, from the output side rather than the status side.

Both halves have to reach the reader, and a pipe can lose either.

set -o pipefail is the obvious remedy and it has a scoping trap, identical to the one the whole-file-writes rule documents for set -o noclobber. The option applies to the shell that executes it. Where each command runs in a FRESH shell — which is every tool invocation for an agent — setting it in one call and running the pipeline in the next protects nothing, because the option is back at its default. Measured: set -o pipefail; false | true reports 1; a false | true in the next invocation reports 0.

So it has to be in the SAME invocation as the pipeline:

set -o pipefail; gh api ... | sort      # ONE command, both parts

Prefer not piping at all when what you want is a verdict. Redirect, capture the status on its own line, then read the file. It has no scoping subtlety to get wrong and it keeps the whole output:

node scripts/check-comment-convention.mjs > out.log 2>&1; echo "EXIT=$?"

An unavailable answer must never read as a passing one, and neither must an answer whose status you did not look at.

Both of those are the same shape one level up — an instrument reporting on something it never examined — and it reaches well past CI: a test fixture that never reaches the mechanism, a search over the wrong set, a printed label that means something adjacent to what you asked. the auditing-an-instrument skill carries the measured instances and the controls that catch them.

A redirect is not safe BECAUSE it is a redirect — it is safe because the status is read on the next line. Redirecting to /dev/null destroys the verdict exactly as a pipe does, and the silence that follows is indistinguishable from success. Measured here: a lane ran pnpm install --frozen-lockfile > /dev/null 2>&1 after a cherry-pick and never read $?. The install FAILED at that moment and said so, into /dev/null.

So the property to hold is not "redirect rather than pipe". It is that both halves of a command's answer — its output and its exit status — reach a reader, and each can be lost independently:

what you wrote	output	status	why it reads as clean
`cmd \| tail -60`	truncated	`tail`'s, always 0	the tail of a failing run looks like a summary
`cmd > /dev/null 2>&1`	discarded	available, unread	nothing is printed, so nothing looks wrong
`cmd > out.log 2>&1`	kept	available, unread	the log is on disk and nobody opens it
`cmd > out.log 2>&1; echo "EXIT=$?"`	kept	READ	—

Only the last row answers. | tail is worth naming separately, because it is written for readability rather than to discard anything and it loses BOTH halves at once: the output a caller needs is cut off, and the status belongs to tail. Measured while writing this section: a unit run reporting 360 failures came back through | tail -60 as exited with code 0, with 31 of its 32 failing files missing from what the reader saw.

Requiring success from a FIXED list is the obvious form and it is wrong here, because some skips are the pipeline working. ci.yml's first job decides what the commit can affect and publishes an inert output, and Lint / Typecheck / Test / Build carries if: needs.changes.outputs.inert != 'true'. A docs-only commit therefore skips it BY DESIGN, and its three dependents with it — so a fixed list makes a correct docs-only PR permanently unverifiable, and whoever hits that learns to wave the rule through.

The expected set is DERIVED, not enumerated. Decide what this commit can affect is the job that knows, so it is the one to require unconditionally:

Decide what this commit can affect — must be success. It fails open (every early exit sets inert=false), so a broken decision costs a full run rather than a silent pass.
inert=true → Lint / Typecheck / Test / Build and its dependents are legitimately skipped.
inert=false → each of them must be success.

Both kinds of skip render identically, which is the whole difficulty: a job skipped because the scope decision excluded it and a job skipped because its dependency failed are both completed/skipped. What separates them is not the row — it is whether the job they depend on succeeded. Read the decision, then read the jobs it implies.

Measured on 59d84ddc0, 39 check-runs: 33 completed/skipped, 3 success, 3 failure. Even with every job now finished, a rollup dominated five-to-one by jobs that declined to run is not the coverage the count suggests — and when that commit was first read, four of those rows were queued/none and none had failed, so a query for conclusion == "failure" returned nothing and read as green.

Re-measure rather than quote a figure from a handoff. The reading above is not the one recorded when the incident was filed: queued rows complete, and the snapshot that proves "nothing had run" stops reproducing within hours. The shape is durable, the numbers are not, and a stale count cited as current is the same error this file is about.

So record the EVENT, not the state. The distinction matters more than timestamping, because a perishable claim does not merely expire — it discredits the durable claim welded to it. "It merged with all eight checks queued, nothing having run" is two statements: it merged while its checks were queued is an event and stays true forever, while nothing ran rots within minutes. Anyone re-checking later sees an ordinary green result, concludes the whole note was misread, and discards the true half with the false one.

Two queued rows are a photograph of something that is by definition about to stop being true. Write down what HAPPENED and the query that would have shown it; leave the counts as illustration, marked as of a moment. Where a durable witness exists, prefer it outright — skipped persists, which is why the dependent-job cascade below makes a better standing example than any count of pending jobs.

Ask whether the RUN exists, not how many checks there are

A workflow that never fired and a workflow waiting for a runner look the same from the checks list, and the obvious discriminator is wrong.

Counting checks does not separate them. A queued workflow contributes fewer checks than a running one, because its jobs become checks only once it starts. Observed on #758: 9 checks and no Lint / Typecheck / Test / Build, which reads exactly like a dropped trigger — while gh run list showed the CI workflow present and queued. Nothing had been dropped; the runners were busy. PR #753 sat for a day at 5 green checks, and that one was real.

Both counts have since risen to 17 and 14 as their runs finished, which is the point: a check COUNT is a reading of one moment and cannot be re-derived later. Nor is there a total to compare against — the matrices expand with the diff, so a change touching templates/base/**, templates/blank/** or templates/blog/** adds scaffold-build.yml's matrix legs on top of everything else. Note which paths those are: templates/plugin/** is deliberately NOT in that trigger list, so a plugin-template PR correctly gets no scaffold run, and expecting one there turns a working filter into a suspected dropped trigger. Do not substitute one fixed number for another; a document whose conclusion is that counts are unreliable should not hand you a range to check counts against. The run-level query is the discriminator.

Ask at the workflow level, scoped to the commit and to the workflow you require:

gh run list --commit "$HEAD_SHA" --workflow ci.yml --limit 5 \
  --json name,status,conclusion,event \
  --jq '.[]|"\(.status)/\(.conclusion // "-")  \(.event)  \(.name)"'

--branch alone is not enough, and it fails in the reassuring direction. It returns the branch's runs from EARLIER commits as well, so a workflow that was dropped for the current head still shows a run — the previous push's — and reads as present.

--commit narrows that and does not close it, because a SHA is not an event. Close-and-reopen — recommended below for a retargeted PR — produces a second run at the SAME head, so if the reopened event is the one that got dropped, the earlier run still answers and the workflow reads as present. Where that matters, compare the run's createdAt against the event you are gating on, or capture the run ID before the event and require a different one after. It also sweeps in unrelated pull_request workflows, so the list looks populated whatever happened to the one you care about. Pin --commit to the head you are gating, and ask per required workflow rather than eyeballing a mixed list.

A busy queue returns a run, queued. An empty result is not yet a finding, because three different things produce it and only one is a defect:

a dropped trigger — the event should have started this workflow and did not. A push re-triggers it.
a path filter excluded the commit — paths or paths-ignore in the workflow's own on: block. Legitimate, and nothing is wrong.
a base filter excluded the PR — branches: [main] against a stacked base, which is the section below.
the PR has a merge conflict — GitHub cannot compute refs/pull/N/merge, so no pull_request workflow runs at all. The remedy is to resolve the conflict; a push that does not resolve it changes nothing, which reads as the trigger still being dropped.

Settle it by reading that workflow's on: block against this commit's diff, not by looking harder at the run list. All four return the same nothing.

Reading the diff LOCALLY is not the same comparison GitHub made, and on a large change it disagrees. Path filters are evaluated against the first 300 files of the generated diff, so a matching path beyond that boundary does not trigger the workflow — while a local git diff sees it and says the run should exist. That reads as a dropped trigger and invites a push that changes nothing. Above 300 files, treat the discriminator as INDETERMINATE and say so, rather than reporting a defect the evidence cannot support.

Measured on this very PR, which is the cheapest available demonstration: integration.yml declares paths-ignore: ["docs/**", "**/*.md"], the diff is one .md file, and the workflow correctly produces no run at all. Reading that absence as a dropped trigger would mean pushing to "fix" a filter doing its job — and, because ci.yml was queued at that moment, the push would have cancelled the one run that mattered.

The remedies are opposite, which is why guessing is expensive. Any push re-triggers a genuinely dropped run. The fix for a path-filtered absence is to do nothing at all. And pushing at a QUEUED run may cancel it — but only where the workflow says so, and that is per workflow rather than a property of pushing:

workflow	concurrency	a push at a queued run
`ci.yml`	group + `cancel-in-progress: true`	cancels and re-queues at the back
`integration.yml`	group + `cancel-in-progress: true`	cancels and re-queues at the back
`preview.yml`	group + `cancel-in-progress: true`	cancels and re-queues at the back
`secret-scan.yml`	none declared	starts a SECOND run; the first survives

So read the workflow's concurrency: block before deciding a push is expensive. Three of the four here cancel; the fourth does not, and treating its queued run as fragile costs a wait for no reason.

A stacked base runs none of the jobs that gate a merge

Four workflows are declared pull_request: branches: [main], so a PR whose base is another FEATURE BRANCH never triggers them. Measured across .github/workflows/:

workflow	trigger	fires on a stacked PR?
`ci.yml`	`pull_request: branches: [main]`	no
`integration.yml`	`pull_request: branches: [main]`	no
`secret-scan.yml`	`pull_request: branches: [main]`	no
`preview.yml`	`pull_request: branches: [main]`	no
`labeler.yml`, `pr-title.yml`	`pull_request_target`	yes
`scaffold-build.yml`, `package-smoke.yml`	`pull_request` + `paths`, no branch filter	yes, if a path matches

This is the most dangerous entry in the file, and the last two rows are why. A stacked PR is not visibly empty of CI. It carries the two pull_request_target checks and, whenever it happens to touch one of the scaffold-eligible template directories, packages/ui/** or the lockfile, path-filtered builds as well — and those carry matrices, so scaffold-build.yml alone contributes six legs. A stacked PR can therefore present a comfortably populated list of green checks, several of them having genuinely compiled something. No bound is given here deliberately: the count comes from whichever matrices the diff triggered, and quoting a ceiling would reintroduce exactly the fixed number this file argues against. There is no red to notice and nothing conspicuously missing.

A stacked PR's CI verdict is therefore UNOBTAINABLE, not pending. Local runs are the only evidence until the base is main.

Retargeting alone does not start CI, which is the trap on the way out. Changing a PR's base emits the pull_request activity edited, and the default activity set is opened, synchronize and reopened — none of these four workflows declares types:, so none subscribes to edited. Retargeting makes the branch filter eligible and emits an event nothing is listening for, leaving a PR that now LOOKS main-targeting with still no substantive run against it.

Follow it with an operation that DEMONSTRABLY moves the head, then confirm a new run exists. A plain git rebase main is not that: when the branch is already a descendant of main it reports the branch up to date and changes nothing, so the push that follows emits no synchronize and the four main-filtered workflows stay absent — having done exactly what the instruction said. Prefer an EMPTY COMMIT, then PUSH it:

git commit --allow-empty -m "chore: retrigger ci after retarget"
git push

The commit moves the local head and nothing else; synchronize is emitted by the PUSH, and until then GitHub has seen no event and started no workflow. The same applies to a rebase — a local history rewrite that is never pushed leaves the pull request exactly as it was, while looking done from the terminal it was run in. An empty commit is preferred because it pushes fast-forward.

git rebase --force-rebase also replays, and it is the worse choice HERE despite doing the job: the rewritten commits need a non-fast-forward push, so GitHub records head_ref_force_pushed, and the verifying-merged-work skill treats any such event as disqualifying — its tail check reports NOT CHECKABLE from then on, permanently, because a force-push can erase a tail and the surviving ref cannot prove otherwise. Taking a remedy from this file that disables the companion procedure is a bad trade for a retarget. Close-and-reopen also starts CI, with the caveat below.

The check that follows depends on which you picked, and conflating them rejects a remedy that worked:

rebase, push, empty commit — the head SHA must DIFFER afterwards, and gh run list --commit <new head> --workflow ci.yml must return a run.
close and reopen — the head cannot differ, by construction. Require a run whose identity is NEW instead: capture the run ID before reopening and require a different one after, or compare createdAt against the reopen.

Either way something must be confirmed to have started. A remedy that is believed to have worked is how a stacked PR sits for a day looking retargeted.

Close-and-reopen carries a cost the other remedies do not, which is the caveat promised above: it leaves the head SHA unchanged, so the diff expands to include the parent stack while every existing review still points at that same SHA — and a coverage check keyed on the head reuses reviews taken when those commits were not in scope. Moving the head is what invalidates them. Gate on the run, never on the base having been changed.

Retargeting changes what a review MEANS, not just what CI runs. A review is evidence about a diff, and the diff is base..head; moving the base moves the diff underneath a head that has not moved. Where a stacked PR is retargeted without a rebase, treat every review predating the retarget as stale regardless of the SHA it names.

A failed job takes its dependents with it, silently

ci.yml fans out from one job:

changes -> ci (Lint / Typecheck / Test / Build) -> e2e            (Browser tests)
                                                -> scaffold-smoke (Scaffold smoke)
                                                -> dev-script-smoke (Dev script ...)

All three declare needs: [ci]. When ci fails they do not run, and a skipped dependent looks nothing like a failure — so three jobs of coverage leave every PR at once, reported as one red job.

Measured on PR #787's head d243c855c, which is what that looks like:

completed/failure   Lint / Typecheck / Test / Build
completed/skipped   Browser tests
completed/skipped   Dev script starts every watcher (${{ matrix.os }})
completed/skipped   Scaffold smoke (${{ matrix.os }})
completed/success   Integration (mysql|postgres|sqlite), gitleaks, ...

Nine green rows, one red, and three jobs that were never evaluated.

The consequence is the trap: fixing ci is the first time those three are evaluated at all. Treating a run as "one failure, now fixed" understates it by three untested jobs, and their first real execution is the push you were expecting to be green.

A completed run does not re-evaluate when `main` moves

A run is bound to the commit and merge ref evaluated when it started. When a broken main is fixed, every PR that went red because of it stays red. No one's checks clear themselves, and a stale red is indistinguishable from a real one on inspection.

A re-run re-fetches the ORIGINAL merge commit, so it does not pick the repair up. Two phases decide this and reading only the second one inverts the answer.

GitHub's documentation says a re-run "will also use the same GITHUB_SHA (commit SHA) and GITHUB_REF (git ref) of the original event that triggered the workflow run". actions/checkout then FETCHES using that pinned commit — getRefSpec in the pinned action, with commit non-empty:

else if (upperRef.startsWith('REFS/PULL/')) {
  const branch = ref.substring('refs/pull/'.length)
  result.push(`+${commit}:refs/remotes/pull/${branch}`)
}

Only afterwards does getCheckoutInfo name refs/remotes/pull/N/merge — a LOCAL ref the fetch above just populated from the pinned commit. Reading that second phase alone suggests the ref is resolved fresh from the server. It is not: the name is reused, the content is pinned.

Shortened here. Read the whole file on GitHub.

Signals

GitHub stars: 57
Forks: 6
Last commit: Sep 2026

Advanced

Catalog kind: skill
Gateway key: reading-a-ci-verdict
Source: github.com/nextlyhq/nextly

github.com/nextlyhq/nextly

More in Dev tools

handoff
Skill · mattpocock triage
Skill · mattpocock prototype
Skill · mattpocock teach
Skill · mattpocock implement
Skill · mattpocock wayfinder
Skill · mattpocock