Health Check

Report-first resource triage. Answers two questions: can we start <target> right now, and what is the safest way to recoup headroom if not. Destructive reclaim is never automatic — only the regenerable build cache is pruned without asking.

This skill spans two scopes, and step 0 decides which one you are in. Docker inventory and every reclaim lever live on the machine running the daemon; memory, swap, disk and CPU can be read on either side but mean different things depending on where you read them. The sandbox has had no host Docker socket since #756 removed it as a host-root escape path, so a sandboxed invocation reports container-scope metrics and hands the Docker half to the orchestrator. Never present container figures as host figures — that is the failure mode this structure exists to prevent, and it is worse than reporting nothing, because a green container disk row reads like permission to start a multi-GB image build.

target is free text naming what you intend to start (a compose path, make dev, a service name). It sizes the verdict — a full docker compose build and a make dev against already-running services have very different footprints, so always pin down which before judging "sufficient."

Performance rules

Default to a fast path: one scope classification, one host snapshot, one Docker summary, one running-container list, and one exited-container list. Do not run verbose or per-container probes until the summary shows a likely binding constraint or the user asks for reclaim candidates.

• Gate expensive commands: docker system df -v, docker stats, docker exec <container> ps ..., du, and nested-Docker inspection are second-pass diagnostics, not baseline checks.
• Run expensive diagnostics once and reuse the captured output. Never call docker system df -v inside a loop over volumes/images/containers.
• Prefer scoped reads over broad scans: explicit cache paths (~/.npm, project .pnpm-store) are acceptable; du -sh /home/* or filesystem-wide find is not.
• Keep reruns delta-oriented: if a prior health-check in the same thread already found the shape of the problem, recheck only the changed metrics unless the target changed.
• Never retry a Docker call that failed for want of a daemon. Step 0 settles reachability once. Nine copies of the same connection error is not diagnosis.

Instructions

0. Classify the scope — always first, before any other command

SNAP=${SNAP:-$(mktemp -d)}
bash .agro/skills/health-check/scripts/scope-preflight.sh | tee "$SNAP/scope.txt"

The script resolves the Docker endpoint (HEALTH_CHECK_DOCKER_SOCK, then DOCKER_HOST, then docker context, then /var/run/docker.sock, then the rootless path) and settles it with at most one round-trip. It always exits 0; a classification step that exits non-zero is the misleading failure this replaces.

It emits SCOPE, DOCKER_CLI, DOCKER_ENDPOINT, DOCKER_TRIAGE and METRICS_SCOPE.

The KEY=VALUE lines select the branch. They are not the report. Surface the single HEALTH-CHECK SCOPE-NOTICE: line to the user and nothing else from this file. Pasting the preflight output as the answer is not a health check.

Branch on DOCKER_TRIAGE:

Value	Meaning	What to do
available	The daemon answered	Run every step below as written.
host-only	No endpoint here, or no CLI to reach one	Skip steps 2, 5, 7 and the reclaim ladder without issuing a Docker call. Report the notice, the scoped metrics, and the host-side procedure.
unreachable	The endpoint exists and the daemon did not answer	Same skip. The one round-trip already spent is the whole diagnosis — do not retry per command. Report the notice; the endpoint may be a dead daemon or a socket this user cannot open.

Do not test for the docker binary to decide this. /usr/bin/docker is installed in the sandbox and proves nothing about reachability — treating its presence as availability is precisely the defect #762 records.

1. Snapshot resources — label the scope you measured

Use one grouped command so the baseline is cheap and internally consistent:

SNAP=${SNAP:-$(mktemp -d)}
{
  echo "=== MEMORY ===" && free -h
  echo "=== SWAP ===" && (swapon --show || echo "no swap")
  echo "=== DISK ===" && df -h /
  echo "=== CPU ===" && nproc && uptime
} | tee "$SNAP/host.txt"

Read for: available memory (not "free" — buff/cache counts), presence of swap (no swap = no cushion), disk % used on /, and load vs core count.

Carry METRICS_SCOPE into every number you report. Under METRICS_SCOPE=container these are the container's cgroup and overlay figures. They answer container-local questions honestly — will uv sync OOM this container, is the workspace filesystem filling — and they do not answer what the Docker host has spare. Where a container's limits are unset, free -h reports the host's physical memory while the container cannot necessarily use it; say so rather than passing the number through.

Under METRICS_SCOPE=host, / is the binding number when Docker's data root is on the root filesystem. Confirm that rather than assuming it — docker system df in step 2 and df -h on the data root settle it. (An earlier version of this skill asserted the root overlay was always the binding number "on this host". That was written for one environment and is false wherever the daemon lives elsewhere or is not visible at all.)

2. Docker resource breakdown — host-only

Requires DOCKER_TRIAGE=available. Under host-only or unreachable, skip this step entirely and point at the host-side procedure. Do not issue these commands to see what happens.

Keep the baseline to three Docker calls:

SNAP=${SNAP:-$(mktemp -d)}
docker system df | tee "$SNAP/docker-df.txt"
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}' | tee "$SNAP/docker-running.txt"
docker ps -a --filter status=exited --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Size}}' | tee "$SNAP/docker-exited.txt"

docker system df splits usage into Images / Containers / Local Volumes / Build Cache with a RECLAIMABLE column — that column is the entire reclaim opportunity at a glance. Only if the summary shows material reclaim or the user asks for a ranked cleanup list, capture verbose detail once:

docker system df -v > "$SNAP/docker-df-v.txt"

Read candidates from $SNAP/docker-df-v.txt; do not rerun docker system df -v per object.

If $SNAP/docker-running.txt shows a Docker-in-Docker container (common names: *dind*, ci-runner-dind), inspect only that nested daemon before concluding the reclaim plan:

docker exec <dind-container> docker system df
docker exec <dind-container> docker ps -a --size --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}\t{{.Image}}'

Nested CI sidecars often hide the best disk win: unused inner CI images and build artifacts inside the sidecar volume. Skip this branch when there is no DIND container. If ongoing cleanup is requested, prefer an idle-aware spindown or watchdog over repeated manual pruning — a lever that runs on a schedule beats one that needs an operator to remember it.

3. Size the target

Match the verdict to what target actually does:

Target shape	Dominant cost	Watch	Answerable under host-only?
docker compose build / fresh stack	Disk — new images + build cache (multi-GB)	Disk %; build cache balloons	No. Image and cache growth land on the Docker host, which is not visible.
make dev / run against live services	Memory + small disk (.venv, node_modules)	Available mem, no-swap spikes during uv sync / npm install	Partly — the container-local memory and workspace disk cost is measurable here.
Pull-only (compose up, no build)	Disk — pulled image layers	Disk %	No. Layers land on the Docker host.

Verify assumptions instead of trusting them, but scope the verification. If the user says "services are already running," confirm against the already-captured container list first (or, under host-only, against the ports themselves — a reachable port is evidence a service is up even when the container list is not available), then check only named/known ports and dependency dirs:

PORTS="<space-separated-ports>"
for p in $PORTS; do (echo > /dev/tcp/127.0.0.1/$p) 2>/dev/null \
  && echo "port $p OPEN" || echo "port $p closed"; done
TARGET_DIR="<target-dir>"
[ -d "$TARGET_DIR" ] && du -sh "$TARGET_DIR/.venv" "$TARGET_DIR/node_modules" 2>/dev/null || echo "deps not yet installed"

A closed port or missing .venv/node_modules is a finding worth surfacing — it changes "ready to start" into "ready to provision, then start." If no ports or target directory are named, skip this probe rather than guessing.

4. Reclaim ladder (safest → most destructive) — host-only

Every lever here acts on the Docker daemon's storage and requires DOCKER_TRIAGE=available. Under host-only or unreachable, propose nothing from this table as something you can do — hand over the host-side procedure instead. There is no in-container substitute; a sandboxed agent cannot free a byte of the host's image store.

Walk the ladder top-down. Run tier 1 freely. Stop and confirm before tiers 2–4.

Tier	Lever	Yield	Risk	Gate
1	docker builder prune -f	Often the biggest single win (regenerable cache)	None — cache rebuilds, only costs future build time	Run freely (skip if --dry-run)
2	Remove exited/abandoned containers (docker rm <id>) + their images	High (stale sandboxes hoard GBs)	Destructive — data in the container is gone	Confirm; never auto-remove a container you didn't create
3	Dangling images (docker image prune -f)	Medium	Low — only untagged layers	Confirm
4	Orphaned volumes (docker volume prune / named docker volume rm)	Usually small, occasionally large	Destructive — may hold DB state	Confirm each named volume individually

Rank candidates by safety × yield, not yield alone. The default action of this skill is tier 1 only; everything below is a proposal.

Identifying tier-2 candidates — use the baseline file for exited containers from torn-down sandboxes, not in the active set:

cat "$SNAP/docker-exited.txt"

Cross-check names against $SNAP/docker-running.txt before proposing removal. An "Exited (255) N days ago" sandbox that isn't one of the live ones is the prototypical safe-to-reclaim target — but still confirm, per the don't-delete-what-you-didn't-create rule. If size/yield is unclear, parse the single verbose snapshot ($SNAP/docker-df-v.txt) rather than issuing new Docker queries in a loop.

5. Per-container RAM reclaim — host-only

Requires DOCKER_TRIAGE=available. This step reads resident memory across containers from the daemon's side; it is not something a container can do for itself. Under host-only or unreachable, skip it. For the RAM inside this container, step 1's figures plus a plain ps -eo rss,args --sort=-rss are the honest local equivalent, and killing a stale process here is still worth proposing.

When memory — not disk — is the binding constraint, the reclaimable RAM may be in-container process accumulation that is invisible to the Docker-object ladder above. docker system df and docker ps --size report layer/diff sizes on disk; they do not predict per-container resident-set memory.

Run docker stats only for memory-bound checks or explicit "system health" requests that ask for RAM/process reclamation:

docker stats --no-stream | tee "$SNAP/docker-stats.txt"

This emits one row per running container showing live MEM USAGE / LIMIT and %MEM. Inspect only the heaviest one or two containers, not every running container:

docker exec <container> ps -eo rss,args --sort=-rss | head -20 | tee "$SNAP/<container>-top-rss.txt"

Typical findings: stale node / python dev servers from a previous session, a hung test runner, or an orphaned build worker that was never cleaned up. Killing the process (not the container) reclaims its RSS immediately and is safer than any tier-2+ ladder action.

Propose the kill to the user with the process name, RSS, and estimated RAM freed — do not auto-kill. This step is memory-only; skip it when disk is the sole binding constraint.

6. Run tier 1 where you can, then report the verdict

Under DOCKER_TRIAGE=available, and unless --dry-run, run the build-cache prune and show before/after with the smallest useful recheck:

df -h / | tail -1
docker builder prune -f
df -h / | tail -1
docker system df

Under host-only or unreachable, there is nothing to prune here. Say so once and move to the verdict.

If the request is a rerun/check-again in the same thread, make the report delta-oriented: call out what changed since the prior health check (disk %, available memory, CPU load, Docker restart/status changes) before repeating the verdict. Do not re-explain the full ladder or rerun verbose diagnostics unless the finding changed; keep it focused on current state plus material deltas.

Then emit a verdict table. Name the scope in the header, and withhold any row you cannot support:

Scope: container (METRICS_SCOPE=container) · Docker triage: host-only

| Resource | Status | Detail |
|----------|--------|--------|
| Disk     | 🟢/🟡/🔴 | <free / %used, headroom vs target> |
| Memory   | 🟢/🟡/🔴 | <available, swap presence, spike risk> |
| CPU      | 🟢/🟡/🔴 | <cores vs load> |
| Docker   | ⚪ N/A | host-only — see the host-side procedure |

Rating guide: 🔴 if starting the target would cross a hard limit (disk → ~90%+, OOM with no swap); 🟡 if it fits but with thin margin (no cushion, transient spikes possible); 🟢 if comfortable. State the binding constraint explicitly — it's usually one resource, not all three.

Withholding rule — the verdict must refuse what the scope cannot support. Under DOCKER_TRIAGE=host-only or unreachable with a build-shaped target (any row in step 3 marked No), the Disk row is not a RAG rating. Render it:

| Disk | ⚪ N/A | host-only — image and build-cache growth land on the Docker host, which is not visible from here |

and state in prose: no build-sizing verdict is possible from inside this container. A labelled green disk row beside a docker compose build target still reads as permission to start the build; a label explains a number, it does not withdraw a conclusion. For container-local targets, keep the real RAG rating — there the container view is the correct view, and refusing to answer would be its own failure.

7. Propose-then-confirm for tiers 2–4 — host-only

Requires DOCKER_TRIAGE=available. Under host-only or unreachable there is nothing here for you to propose: the levers are in the host-side procedure, and the person running it owns the confirm gate.

If tier-1 reclaim already clears the target, say so and present the rest as optional headroom — don't push destructive removal that isn't needed. When you do propose it, quote the concrete artifact (container name, age, size) so the user can judge:

Optional further reclaim:
- <name> (exited <N>d ago, <size>) — removing frees ~<X>G → <new free>/<new %>.

Build this list from the baseline files and, when needed, one cached verbose snapshot. For volume candidates, read the Local Volumes section from $SNAP/docker-df-v.txt; never run docker system df -v once per volume.

Then ask (use AskUserQuestion). Run the removal only on explicit approval. With --dry-run, propose everything and write nothing — including skipping the tier-1 prune.

Host-side Docker triage

The relocated half of this skill. Docker inventory and every reclaim lever run where the daemon runs — they are not something a sandboxed agent can do, and #756 closed that route deliberately.

Who runs this. The orchestrator — the agent session at the host project root. Root AGENTS.md § "Agent work stays inside the sandbox" assigns Docker and sandbox lifecycle to that role. If you are that session, DOCKER_TRIAGE reports available and you run steps 2 through 7 as written; this section is what you hand to somebody else.

The round trip, both directions. A sandboxed agent cannot run this block. Print it, say where it runs, and say what comes back:

1. Ask the operator to run the block below on the machine hosting this sandbox, at the harness project root — the orchestrator session, not this container.
2. Ask them to paste the output back into this session as their next message.
3. Fold that output into steps 2–7 exactly as if you had captured it yourself, then re-issue the verdict with Docker triage: host (operator-supplied).
4. If nobody comes back, finish the report anyway. State Docker headroom as UNKNOWN, keep the container-scope rows, and keep the build-sizing refusal from step 6. Never leave the report silently pending on a paste that may never arrive.

# Run on the Docker host, at the harness project root.
echo "=== SCOPE ===" && bash .agro/skills/health-check/scripts/scope-preflight.sh
echo "=== DISK ===" && df -h /
echo "=== MEMORY ===" && free -h
echo "=== DOCKER SUMMARY ===" && docker system df
echo "=== RUNNING ===" && docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}'
echo "=== EXITED ===" && docker ps -a --filter status=exited \
  --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Size}}'

That is the read-only baseline and is safe to run unprompted. Everything beyond it follows the same gates as in-scope use: tier 1 (docker builder prune -f) is free to run, tiers 2–4 need a per-artifact confirm, and docker stats --no-stream is a second-pass probe for memory-bound checks only. With --dry-run, hand over the baseline and nothing else.

Anti-patterns

• Reporting container figures as host figures. The single worst outcome available here: it looks like an answer, it passes a glance, and it green-lights a build the host cannot fit. Carry METRICS_SCOPE into every number.
• Retrying Docker after step 0 said no. One classification, one answer. Nine identical connection errors is what #762 was filed about.
• Testing for the docker binary instead of the endpoint. The CLI ships in the sandbox image. Its presence means nothing.
• Rating a resource you cannot see. A RAG row implies a measurement. Under host-only, a build-shaped Disk verdict is fabrication with a colour on it — render N/A and say why.
• Repeating expensive diagnostics. docker system df -v, docker stats, docker exec <container> ps, and du are second-pass probes. Capture each once, reuse the output, and never put verbose Docker calls inside per-object loops.
• Auto-removing containers/volumes. Only tier 1 runs without asking. A container or named volume you didn't create gets a confirm gate every time.
• Trusting "it's already running." Closed ports and missing .venv/node_modules are silent until you check — verify, then size.
• Ranking by yield alone. A 5G exited container is not a better lever than a 9G cache prune when the prune is risk-free. Safety × yield.
• Pushing destructive reclaim that isn't needed. If tier 1 clears the target, the rest is optional headroom, framed as such — not a recommendation.
• Handing over the host procedure and then going quiet. The report finishes with or without the paste-back; UNKNOWN is a result, silence is not.