Debugging Ares

You are debugging a running or recent Ares operation. Pick the cheapest source first; only escalate if it doesn't answer the question.

Read this before you do anything

Do not declare an op healthy from process liveness, NATS/Redis ping, or token-rate alone. A wedged Ares op happily presents as status=running, workers active, Redis green, cache hit ≥80%, and tokens climbing — while making zero external progress for hours. This has happened. Don't repeat it.

The only valid "healthy" verdict requires a comparison:

1. Compare this op's objective state now vs. 60s ago — has_domain_admin, domain compromise count, hosts owned, creds, hashes, vulns exploited. If none changed, that's churn, not progress.
2. Compare this op to recent ops' baseline. Pull ares ops list and look at how long prior ops took to hit DA / 2nd domain. If this op is more than ~2× slower to a milestone the last 3 ops hit, treat it as wedged regardless of token rate.

Token churn is the signature of the LLM re-evaluating the same frozen state every tick; high cache-hit rate (>80%) on a slow op is a symptom of the wedge, not evidence of health.

Worker per-role log mtimes are not a signal. In steady state the orchestrator centralizes everything via NATS into /var/log/ares/orchestrator.log; per-role files (recon.log, cracker.log, etc.) stay near-empty. Don't read into stale mtimes.

Before you propose a code fix

Ares timeline events (evt-exploit-fail-* in ares:op:*:timeline) and "Assistance needed" strings the LLM emits ("the tool schema does not accept X", "current toolset lacks Y", "tool requires password but only hash available") are the failing LLM agent's confabulated explanation of its own failure — not a bug report. The agent does not know its own tool schemas or the orchestrator's dispatch layer, and it will invent plausible-sounding gaps that don't exist.

Before recommending a fix from one:

1. Open the tool wrapper in ares-tools/src/**/*.rs — does the tool actually accept the arg the LLM said was missing?
2. Open the LLM-facing schema in ares-llm/src/tool_registry/**/*.rs — does it declare the field?
3. Open the automation dispatcher in ares-cli/src/orchestrator/automation/*.rs — does it inject the credential/state from Redis into the payload?

If all three already do the thing, the LLM was confabulating. The real failure is elsewhere — the tool ran and hit a Kerberos error, dispatch timed out, worker didn't have the credential in state, etc. Grep the orchestrator log for the actual dispatch record + tool stdout/stderr; those are ground truth. Timeline events are not.

Tight-loop / wedge signatures (grep the orchestrator tail for these first)

Run Step 0, then before drawing any conclusion grep the tail of orchestrator.log for each pattern below. If any hit, that's almost certainly your wedge:

Pattern (regex)	Means
clearing dedup for retry	Wrapper-level retry loop; same task being re-dispatched every tick
Dispatching <same_tool> ... <same_target> repeated ≥3×	Automation hot loop with no backoff
KDC_ERR_TGT_REVOKED|KDC_ERR_S_PRINCIPAL_UNKNOWN|KDC_ERR_PREAUTH_FAILED|TGT has been revoked	Kerberos error that will not self-heal; orchestrator may be retrying anyway
tool exited with code Some\(0\) followed by stderr content	Zero-exit-with-error: wrapper treats stderr-on-zero-exit as transient and re-tries
Same task_id shape (e.g. trust_raise_child_<hex>) repeated with distinct hex per tick	Dedup key churning instead of blacklisting
Processing real-time discoveries count=1 ticking every 5s with no other state change	Orchestrator stuck in discovery-replay loop
Waiting for blue team to finish\.\.\. active_investigations=[0-9]+ ticking every 10s	Not a wedge — red is DONE. Op is holding open until blue investigations drain. Check red_completed_at / red_completion_reason in meta (see Step 0).
Loki request error \(retryable\) / Retrying Loki query after transient failure flooding the tail	Blue team's external Loki ($LOKI_URL) is flapping; blue investigations grind to a crawl and starve out post-red op close. Not a red bug.
Tool binary not found \(spawn failed\) — removing from available tools firing across many recon tools (nmap_scan, enumerate_users, enumerate_shares, smb_signing_check, username_as_password) in the first seconds of the op	Tool-pruning cascade — a prior spawn failure poisoned the worker's per-process unavailable_tools HashSet. Only a genuine worker-process restart clears it — not task ec2:restart, which never touches ares@ units (see Step 8). Fix: task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"'. Full mechanism + confirmation queries in Step 3.5.

If you don't see these but the op is slow vs. baseline, escalate to Loki / Tempo for cross-tick LLM latency or tool-call stalls.

What goes where

Source	Latency	Coverage	How to query
task ec2:status	seconds	Worker process state, Redis ping	Bash
task ec2:runtime	seconds	Per-op token/cost/domain banner	Bash
Loki (Grafana)	seconds	Historical /var/log/ares/*.log + syslog/auth	mcp__grafana__query_loki_logs (datasourceUid loki)
Tempo (Grafana)	seconds	OTEL traces of LLM calls + tool dispatch	mcp__grafana__* Tempo proxy tools
SSM task ec2:exec	~5-15s	Anything on the host (redis-cli, journalctl)	Bash, never tail -f
task ec2:logs	streaming	Live tail of one role's log	DO NOT use in Claude — it's an interactive SSM session

Rule: never run task ec2:logs from an agent — it opens an interactive SSM session that won't terminate. Always use Loki (preferred) or task ec2:exec EC2_NAME=kali-ares CMD='tail -n 200 /var/log/ares/<role>.log'.

AWS auth: use whatever ambient AWS profile has SSM access to the box — do not hard-code one. Ownership of the kali-ares instance has moved between profiles/accounts multiple times; a stale prefix (AWS_PROFILE=personal AWS_REGION=us-east-1) will produce No running instance found matching: kali-ares even when the box is up. Verify resolution with task ec2:ops EC2_NAME=kali-ares first; if it fails, try flipping between lab and personal and between us-east-1 and us-west-2. The command examples below run against the ambient profile — set it explicitly only if the ambient one doesn't resolve the box.

Step 0 — mandatory baseline triage (run all in parallel, on every invocation)

Do not skip any of these. Do not respond to the user with a verdict until you've inspected each output. The point of this step is to make it impossible to declare "healthy" without the evidence.

# 0a. Current op id + status
task ec2:ops EC2_NAME=kali-ares LATEST=true

# 0b. Current op objective state + tokens
task ec2:runtime EC2_NAME=kali-ares LATEST=true

# 0c. Process / Redis / NATS health
task ec2:status EC2_NAME=kali-ares

# 0d. The single most important probe — orchestrator tail. Grep it for the wedge signatures listed above.
task ec2:exec EC2_NAME=kali-ares \
  CMD='tail -n 300 /var/log/ares/orchestrator.log'

# 0e. Historical baseline — last several ops, to compare runtime-to-milestone
ares --ec2 kali-ares ops list | head -20

# 0f. Failed tasks for the current op
ares --ec2 kali-ares ops tasks --latest --status failed | head -80

Pull op-YYYYMMDD-HHMMSS from 0a/0b and use that as $OP below. After collecting:

1. Is red already done? Before anything else, check red_completed_at, red_completion_reason, and red_blocked_on_blue in the op's meta. If red_completed_at is set, red is NOT wedged — it ended (either by success, "all forests dominated (post-exploitation complete)", or by hitting "max runtime exceeded"). The op status will still show running because the operation as a whole is holding open for blue investigations to drain; that's the "Waiting for blue team to finish" pattern in the wedge table. Don't misdiagnose an ended-red as wedged. One command:

   task ec2:exec EC2_NAME=kali-ares CMD="sudo redis-cli hmget ares:op:$OP:meta red_completed_at red_completion_reason red_blocked_on_blue has_domain_admin has_golden_ticket"
   

2. Grep the 0d output for each pattern in the "Tight-loop / wedge signatures" table. If any hits ≥3 times, you have your root cause; jump to reporting.

3. Compare 0b's Domains compromised and Vulns exploited against the runtime banner of recent ops in 0e. If the prior 3 ops compromised more domains in less time at this point, the current op is regressed regardless of how healthy 0a/0c look.

4. Read 0f — the failure mode of the first 5-10 failed tasks usually points at the role/tool that's flailing.

Only proceed past Step 0 to deeper probes (Loki, Tempo, SSM journals) if none of the above lands a verdict.

Two footguns in the Step 0 commands themselves — read before you file a "Redis broken" bug:

• ares --ec2 kali-ares ops list (0e/0f) connects to local Redis on the machine you're running from, not to the box's Redis over SSM. From an agent host with no redis-server and no ec2:redis:forward running, it will exit with Failed to connect to Redis: Connection refused. That's not "the box is broken" — it's the CLI wanting a live connection. When you see it, fall back to task ec2:exec EC2_NAME=kali-ares CMD='sudo redis-cli ...' for anything you'd have asked the CLI for.

• There is no ares:op:<op>:creds key. It is :credentials. Any command built on :creds returns an empty/zero result that reads exactly like "no credentials found" — the most expensive false negative in this document's history. Verified against ares-core/src/state/keys.rs and the writer verbs in ares-core/src/state/reader.rs:

  Key	Writer verb	TYPE	Count with	Dump with
  :meta	hset	HASH	HLEN	HGETALL / HMGET
  :credentials	hset_nx	HASH	HLEN	HGETALL
  :hashes	hset	HASH	HLEN	HGETALL
  :vulns	hset_nx	HASH	HLEN	HGETALL
  :completed_tasks	hset	HASH	HLEN	HGETALL
  :hosts	rpush	LIST	LLEN	LRANGE k 0 -1
  :users	rpush	LIST	LLEN	LRANGE k 0 -1
  :timeline	rpush	LIST	LLEN	LRANGE k -50 -1

  Wrong verb → WRONGTYPE, which is loud. Wrong key name → 0, which is silent. When in doubt: redis-cli type <key>.

Step 1 — fast triage (Loki, last hour)

Loki has every ares log line shipped from the EC2 box. Datasource UID is loki. Logs are JSON; the actual line is in the message field, with labels app="ares", deployment="alpha-operator-range-kali-ares", job=<role>.log.

Run these in parallel:

mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |~ "(?i)error|fatal|panic|traceback|RUST_BACKTRACE"'
  limit: 30

mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "WARN|ERROR"'
  limit: 30

Narrow by role when you know the suspect: change job="orchestrator.log" to one of recon.log, credential_access.log, cracker.log, acl.log, privesc.log, lateral.log, coercion.log.

Narrow by op id (substring match on the log line):

logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |= "op-20260630-201500"'

Use query_loki_stats first when you're guessing the selector — it tells you whether the stream has any entries before you waste a query_loki_logs call.

Step 2 — failed tasks (operation-level)

task red:multi:tasks:list LATEST=true STATUS=failed   # K8s
ares --ec2 kali-ares ops tasks --latest --status failed   # EC2

Failed tasks include the worker's error message and the role that failed. Cross-reference against Loki by role + timestamp.

Step 2.5 — attribute a specific tool call to the worker that ran it

Use this when the question is "did tool X actually run for task Y, on which worker, and did it succeed?" The canonical case is verifying cross-role routing (e.g. credential_access-originated password_spray / username_as_password / laps_dump calls must land on a recon worker because netexec lives there — see RECON_ROUTED_TOOLS in orchestrator/tool_dispatcher/mod.rs).

Ground truth is the OTel span line each worker emits at INFO level when it starts a tool:

Executing tool tool=<T> call_id=<T>_<hex> task_id=<origin_role>_<hex>

The span attributes on that same line are what you actually want:

• agent.role = the worker that executed the tool. Cross-routing fired if this differs from the role prefix of task_id.
• attack_operation_id / op.id = the op — scope every grep to this to avoid conflating past ops.
• The follow-up line for the same call_id carries Tool execution failed tool=<T> err=<message> on failure.

The three canonical failure strings and where they come from — memorize these because they distinguish "binary missing" from "tool ran and errored":

String	Source	Meaning
failed to spawn '<binary>' — is it installed?	ares-tools/src/executor.rs:219	ENOENT: the binary isn't on this worker's $PATH
failed to spawn impacket-ntlmrelayx (is it installed?)	ares-tools/src/coercion.rs:586	Same, special-cased (no single quote — do not narrow greps to require one)
Tool '<T>' is not installed on this worker.	worker/tool_executor.rs::unavailable_tool_response	Cached unavailability — a prior call ENOENT'd and future calls return this without re-spawning

Everything else in err=... means the binary ran and the tool logic failed (timeout, KDC error, no creds, etc.).

Query patterns. Prefer Loki when the label narrow is easy; SSM grep -a when you need cross-file correlation on the box.

# Loki: every executor span for tool X in this op
mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |= "Executing tool" |= "tool=<T>" |= "<OP>"'
  limit: 50

# SSM: same thing, plus the failure line for the same call_id
task ec2:exec EC2_NAME=kali-ares \
  CMD='sudo grep -a "<OP>" /var/log/ares/recon.log /var/log/ares/credential_access.log | grep -a "tool=<T>" | head -20'

# End-to-end trace of one call_id across every worker log
task ec2:exec EC2_NAME=kali-ares \
  CMD='sudo grep -a "<call_id>" /var/log/ares/*.log'

# Sanity: is the binary the caller expects actually on the box right now?
task ec2:exec EC2_NAME=kali-ares \
  CMD='which netexec; ls -la /usr/local/bin/netexec /usr/bin/netexec 2>/dev/null; netexec --version 2>&1 | head -3'

Gotchas (do not skip):

1. task ec2:exec runs CMD through go-task's template engine. {{ ... }}, backticks, and some quoting silently fail with "CMD required" — that means the template ate the arg, not that CMD was empty. Workarounds: bind Q="…" locally and pass CMD="$Q"; keep single quotes on the outside; avoid {{. If you see "CMD required", simplify quoting before assuming the file is empty.
2. Per-role log files are ANSI-color-coded on disk. grep 'tool.name="X"' returns 0 hits even when the tool ran because the bytes are tool.name<ESC>[0m<ESC>[2m=<ESC>[0m"X". Anchor on invariant plain-text substrings: Executing tool, tool=<T> call_id=, err=failed to spawn, attack_operation_id="<OP>". grep -a (force text mode) is required — the escapes make grep treat these files as binary and go silent otherwise.
3. Per-role log files stay near-empty in steady state (see the intro's "worker per-role log mtimes are not a signal") — but executor OTel spans DO land there. recon.log and credential_access.log are the right files for tool-attribution greps even though they look sparse.
4. ingest.log is a firehose (multi-GB); do not grep it without a --max-count or a very narrow anchor.

Case study — was cross-routing broken on op-20260716-181136? credential_access called username_as_password, the runner pruned it after "spawn failed". The trace resolved it in three greps:

agent.role=recon
task_id=credential_access_de9f5fa0be53
err=failed to spawn 'netexec' — is it installed?

agent.role=recon proved routing fired (a recon worker picked up a credential_access-originated call — cross-routing correct). The err= matched executor.rs:219 verbatim, pinning the root cause on netexec missing from the box at that moment. Fix was ansible provisioning drift, not code. Without the span attributes there was no way to distinguish "routing bug" from "environment drift" — every hypothesis based on just the runner's WARN line would have been wrong.

Step 3 — wedge detection (objective state frozen)

The canonical wedge is NOT "tokens flatlined" — tokens almost always keep climbing during a wedge because the LLM re-evaluates the same frozen state every tick. The canonical wedge is "objective state frozen while tokens climb." Probe state, not tokens:

# Snapshot 1 — verb matches TYPE per the table in Step 0. `credentials` NOT `creds`.
task ec2:exec EC2_NAME=kali-ares \
  CMD='redis-cli hmget "ares:op:'"$OP"':meta" has_domain_admin has_golden_ticket target_ips initialized red_completed_at red_blocked_on_blue; echo ---; for k in credentials hashes vulns completed_tasks; do printf "%s=" "$k"; redis-cli hlen "ares:op:'"$OP"':$k"; done; for k in hosts users timeline; do printf "%s=" "$k"; redis-cli llen "ares:op:'"$OP"':$k"; done'
# wait 60s
# Snapshot 2 — same command. Diff the two. Identical = wedge.

Cross-check against tokens: pull ec2:runtime at both snapshots. Tokens climbing + state identical = textbook wedge. Tokens climbing + state changing = healthy. Tokens flatlined + state identical = orchestrator hung (rarer).

If wedged, two further probes pinpoint where:

# Outbound HTTPS from orchestrator — zero connections = LLM API stall
task ec2:exec EC2_NAME=kali-ares CMD='ORCH=$(pgrep -f "ares orchestrator" | head -1); echo "orch_pid=$ORCH"; sudo ss -tnp 2>/dev/null | grep "pid=$ORCH" | grep -v 127.0.0.1 | wc -l'

# Loki search for retry/throttle/dedup markers in the last 30 minutes
mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "clearing dedup for retry|KDC_ERR_|Task deferred|throttler|stale|wedge"'
  limit: 80

Remedy depends on root cause:

• Hot retry loop on a tool (clearing dedup for retry) → fix the dedup/blacklist logic in the relevant automation/auto_*.rs; in the meantime task ec2:stop-op ... LATEST=true to stop the burn.
• LLM API stall → check the model provider's status, then restart the orchestrator with task ec2:restart EC2_NAME=kali-ares (that is stop+start of ares-orchestrator.service and infra only — it preserves Redis but leaves workers untouched; add task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"' if the workers are the stalled party).
• State frozen but no signature → escalate to Tempo (Step 7) to find the slow span.

Step 3.5 — tool-pruning cascade (recon suddenly does nothing)

Distinct failure class from "wedge" and "crash." Signature: the LLM issues a normal task, workers stay active, but every recon/credential-access tool the LLM tries is immediately marked Tool binary not found (spawn failed) — removing from available tools and the LLM burns through its 24-tool list in seconds without any external effect. The op then presents as slow-vs-baseline with 0 creds / 0 hashes / 0 hosts.

Grep the LLM runner side for the pattern:

task ec2:exec EC2_NAME=kali-ares CMD="sudo grep -aE 'Tool binary not found \(spawn failed\)' /var/log/ares/orchestrator.log | grep -a '$OP' | grep -oE 'tool=[a-z_]+' | sort | uniq -c | sort -rn"

If a bunch of nxc/netexec-backed tools (nmap_scan, enumerate_users, enumerate_shares, smb_signing_check, check_rdp_reachability, check_winrm_reachability, username_as_password, smb_sweep) all show up, that's the cascade.

Mechanism (three separate files):

1. ares-tools/src/executor.rs:219 — real spawn failure emits failed to spawn '<binary>' — is it installed?.
2. ares-cli/src/worker/tool_executor.rs:332-333 (is_tool_unavailable_error) — classifies that string as "unavailable" and inserts the tool name into a per-process unavailable_tools: HashSet<String>. Every subsequent call to that tool on that worker skips the spawn entirely and returns the cached "Tool 'X' is not installed on this worker. Do not call this tool again — it failed to spawn previously." response (tool_executor.rs:318-328).
3. ares-llm/src/agent_loop/runner.rs:60 — dispatch_one flattens the worker's error field into output ("Error: {err}\n\nPartial output:\n{output}"), then runner.rs:532 detects output.contains("failed to spawn") and yanks the tool from the LLM's active list for the rest of this task, plus injects a [SYSTEM] message telling the LLM to stop trying.

The trap: one transient spawn failure poisons the tool for the worker's lifetime — no TTL, no re-probe. Runs whose spawn genuinely failed (a mid-deploy race, an apt lock, an ephemeral cgroup hiccup) leave dead tool entries that persist across every subsequent op the same worker handles.

Deploys restart workers only if their units are already active (.taskfiles/ec2/Taskfile.yaml:255-257), so task ec2:deploy usually clears the poison — but silently skips any worker whose unit is inactive, printing no ares@ worker units active — skipping restart. When that happens the worker keeps its poisoned unavailable_tools set and /proc/<worker-pid>/exe points at the pre-deploy inode with (deleted) on it (see the Step 8 deploy note).

Confirmation & fix:

# Check worker uptime — anything > a few hours across multiple ops is suspicious
task ec2:exec EC2_NAME=kali-ares CMD='systemctl show ares@recon.service -p ActiveEnterTimestamp,MainPID; ps -o pid,etime,cmd -C ares | head -10'

# Verify the binaries actually work from the shell (rules out "genuinely uninstalled")
task ec2:exec EC2_NAME=kali-ares CMD='which netexec nxc nmap; nxc --version 2>&1 | head -1; nmap --version 2>&1 | head -1'

# If binaries work but pruning still fires → bounce the WORKER units (keeps Redis).
# `task ec2:restart` will NOT do this — it never touches ares@ units.
task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"; systemctl is-active "ares@*.service" | sort | uniq -c'

If the pruning cascade repeats on the very next op with fresh workers, the spawn failure is reproducible — probe from inside the worker's cgroup for AppArmor denials, broken Python venvs (nxc/netexec is a pipx shim; python3 -c 'from nxc.netexec import main' is a direct test), or system-ares.slice restrictions.

Note: sprayhound-backed tools (password_spray, asrep_roast) use a different binary and are unaffected — seeing those still Executing tool in the recon.log while nxc-backed tools are pruned is the signature that isolates this to the netexec side.

Step 4 — worker crash loop

A specific role keeps respawning. Check systemd journal via SSM: