Exploring Signals scouts

A scout is a scheduled agent that wakes on its own interval, looks at one PostHog project, decides what's genuinely worth surfacing, and either writes it into the Signals inbox as a report or closes out empty (a real, valid outcome). PostHog ships a fleet of canonical scouts — a cross-product generalist (signals-scout-general) plus per-surface specialists (error tracking, logs, AI observability, experiments, feature flags, session replay, web analytics, surveys, and more). A project may also have custom scouts beyond the canonical fleet — any signals-scout-* skill a team authored (e.g. -brand-mentions, -mcp-feedback) shows up here too, so don't assume a fixed roster: scout-config-list is the authoritative roster for a project. (One caveat: a just-authored scout has no config row until the coordinator's next tick auto-registers one — or until someone registers it via the write-side scout-config-create — so a brand-new scout may briefly be missing from the list.)

This skill helps you understand and explore what a project's scouts are doing and how they're performing — entirely through read-only MCP tools. It is the observability counterpart to the authoring-scouts skill (which teaches writing and tuning) and to the inbox-exploration skill (which covers the inbox reports scouts feed into). (The scout tools were recently renamed from signals-scout-* to scout-*; if a scout-* name comes back unknown, the server may still expose it under the legacy signals-scout-* name — search the tool catalog and call whichever name it returns.)

Governed metric first

When the question asks for scout failure rate or cost per run, call posthog:metric-list before the scout tools and look for scout_run_fail_pct or scout_cost_per_run. Run an approved, non-drifted match with posthog:data-catalog-metric-run for the canonical headline. If the user also asks which scouts, runs, or failure modes drive the result, answer the headline first, then use the workflows below for a noncanonical breakdown. If no governed metric matches, say so and label the derived measure noncanonical.

A scout's output is inbox reports, written 1:1. Scouts list emit_report / edit_report in their allowed_tools and author or edit inbox reports directly; a run's output shows up as emitted_report_ids (reports it authored) and edited_report_ids (reports it updated). The run rows also carry emitted_count / emitted_finding_ids — legacy fields from the deprecated signal-emitting channel (weak emit_signal findings a pipeline consolidated). On a report-channel scout they stay 0 / empty even on a productive run; a non-zero tally means the run came from a scout still on the legacy channel (an old custom scout, or a canonical scout not yet ported) — real output for that run, not noise. When unsure of a scout's channel, check its allowed_tools via skill-get. Never read emitted_count: 0 as "did nothing" — check the report columns and the run summary first. A scout whose config carries a structured_output_schema has a third output channel next to reports: schema-validated measurement records, recorded as $scout_structured_output events in the project (only scalar top-level payload keys flatten to output_<key> properties — object and array fields live solely inside the full output property, so a missing output_<key> is not a missing value; subject names the judged entity) rather than as run-row columns. The events are the ground truth — metadata.derived.has_structured_output says the run had at least one batch accepted, which is a fast per-run screen but not delivery confirmation (a rare capture failure after acceptance leaves it true with fewer or no events behind it), so count the events when the number of records matters. See references/scout-data-model.md for the event shape. Each run also carries a metadata map. Top-level: the provenance set harness_prompt_version / report_channel (none, emit, edit, or both) / skill_origin / github_guidance, saying which instructions the run was given; plus routing keys (model / runtime_adapter / reasoning_effort / service_tier) only when a gate or pin overrode the default. Nested under metadata.derived: booleans the harness computes at the end of the run (has_emit_report, has_edit_report, has_self_improvement, has_chart, has_self_validation, has_structured_output). When comparing runs (before/after a prompt change, one model against another), segment on all four provenance values first: runs differing on any of harness_prompt_version, report_channel, skill_origin, or github_guidance were given different instructions and aren't a like-for-like population. Runs predating this field have none of them, so treat missing provenance as unknown and exclude those runs from a comparison rather than pooling them. For "what kind of run was this?" questions — did it author a self-improvement report, did it validate its follow-up queue — read derived rather than parsing the prose summary. It's computed server-side from what the run actually did, so it can't disagree with the run's own output — with one exception: has_structured_output tracks batches the run had accepted, not events delivered, so it alone can be true with fewer or no records behind it (count the events, as above). No derived map at all means unknown, not "all false" — the run predates the field, failed before finishing, or its stamp failed. Most runs from before this shipped have no map, so don't read their absence as a finding.

There are six things you can observe about the fleet, each with its own tool:

What you want to know	Tool	What it tells you
Which scouts run, how often, in what posture	scout-config-list	One row per scout: schedule (run_interval_minutes or run_cron_schedule), enabled, status / pause_reason, emit, output_destinations, write_scopes, mcp_gateway_server_ids, last_run_at, consecutive_failure_count, description, scout_origin, owners, tags
What the scouts actually did, run by run	scout-runs-list / -retrieve	Per-run status, timing, end-of-run summary, emitted_report_ids / edited_report_ids, error / failure_reason on a failed run, deep-link; scope with skill_name
What the fleet has learned across runs	scout-scratchpad-search	Durable per-team memory (baselines, noise, allowlists)
What the team has told the fleet	scout-notes-list	Steering notes humans/agents left for scouts (per-scout or fleet-wide, newest first)
Which reports a run wrote or edited	the run row itself	emitted_report_ids / edited_report_ids; resolve each id via inbox-reports-retrieve
What the scouts surfaced to the user	inbox-reports-list	The scout-written reports, as the user sees them (scout: "<skill_name>" for one scout; source_product: "signals_scout" for the fleet)

scout-config-list takes a tags parameter (comma-separated) to narrow the roster to the scouts carrying any of the given labels — useful on a large fleet when the question is scoped to one area, e.g. tags=revenue. scout-runs-list takes skill_name (and optionally skill_version) to scope the dump to one scout, which is the normal way to answer any question about a single scout without paging through the fleet.

The orienting tool is scout-project-profile-get — the deterministic snapshot of "what's true about this project" that every scout cold-starts from. When a scout found nothing, this is usually why.

Output handling: expect to offload to a file

Two of these tools — scout-runs-list and especially tasks-runs-session-logs-retrieve — routinely return payloads that overflow an MCP client's token budget and get spilled to a file. This is the normal path, not an error. Plan for it up front rather than discovering it after a failed call:

• Keep limit small on scout-runs-list (~10–15). Each row carries a long prose summary, and runs come back newest-first across the whole fleet, so even a modest page is large.
• Session logs are large by nature. A single run's log is hundreds of KB to a few MB. Fetch it with call --json (so the saved file is real JSON, not the pretty text format — jq-able) and read the saved file with jq / a script rather than inline.
• Don't hand-parse the session log. The bundled scripts/ do the reconstruction for you — see below.

Start here: is the fleet even set up?

Don't assume the project has scouts. The fleet only runs on teams enrolled via the signals-scout feature flag, and a project may have no configs, all-disabled scouts, or scouts stuck in dry-run. Run this first whenever a user asks about their scouts for the first time in a session.

scout-config-list

Read the result against three cases:

The config list is unpaginated — it comes back as { results: [...] } (a bare array), with no count field. Read the result against three cases:

• Empty (results: []) — no scouts are registered. The project isn't enrolled in the scout fleet (or hasn't ticked yet). Say so plainly; don't go fishing for runs. Point the user at the Signals scout settings / PostHog Desktop onboarding rather than inventing activity.

• Configs exist but all enabled: false — the fleet is registered but paused. Nothing is running. Tell the user which scouts exist and that they're all off — and say who switched each one off, which status carries: paused_by_user means a person (or a launch seed posture) turned it off, paused_by_system means an automatic pause with its cause in pause_reason (no_output / ignored / repeated_failures). Either kind resumes with enabled: true via scout-config-update. A repeated_failures pause is the failure breaker: the streak of scheduled failures (consecutive_failure_count; manual and workflow-triggered runs don't count) ran one past what the schedule fits in twelve hours, clamped to 5–25 (daily: 5, rolling hourly interval: 13, hourly cron: 15, since cron slots are counted over a window padded for daylight-saving shifts). It is half-open, so the coordinator probes the scout once a day and resumes it on a clean run (unless the project is at its enabled-scout cap, which leaves the row paused after a clean probe); read the newest run's failure_reason to say what kept failing.

• At least one enabled: true — the fleet is registered and that scout is allowed to run. For each enabled scout note its cadence (run_cron_schedule when set, else run_interval_minutes; the cron wins), emit (false = dry-run, runs but writes nothing to the inbox), and last_run_at. A status of pending_pause means the scout still runs but the system has flagged it to pause soon (cause in pause_reason); any config edit clears the warning. One caveat before reporting "it's live": runs are gated by the signals-scout feature flag, not by enabled. A project that was enrolled and later drained from the flag keeps its enabled: true rows, but the coordinator no longer plans runs for it — so a stale or null last_run_at on an enabled scout usually means the project is no longer enrolled, not that the scout is idle.

  last_run_at is a dispatch stamp, not proof a run executed. The coordinator advances it the moment it enqueues a child workflow for a due scout — before any worker picks the run up. Child dispatch is fire-and-forget, so if workers are saturated or down the children just queue and no run ever materializes, yet last_run_at keeps marching forward each tick. So a recent last_run_at means "dispatched this tick," not "a run is genuinely happening." The authoritative liveness signal is the newest actual run row in scout-runs-list, not the config stamp. Cross-check them: if last_run_at is fresh (minutes ago) but no run row has appeared for that scout in well over its run_interval_minutes, the fleet is dispatching but not running — workers backed up / down, or runs stranded — a real reliability problem, not a live scout. Don't report "it's running" off last_run_at alone.

A scout that is enabled: true but emit: false is the most common source of "my scout isn't doing anything" confusion: it is running and reasoning every tick, it just isn't allowed to post reports yet. Always surface the emit posture when reporting on a scout.

See references/scout-data-model.md for every field on a config, run, and scratchpad entry, the run status values, and how the pieces link together.

Workflow: survey the fleet

"What scouts do I have / what are they doing?" — lead with config-list, then enrich with the most recent run per scout so the user sees liveness, not just configuration.

1. scout-config-list — the roster.
2. scout-runs-list once with a small limit and pick the newest run per skill_name (runs come back newest-first across the whole fleet, so one call usually covers everyone); for a scout that doesn't appear in that page, call again with skill_name set. Report status and how long ago it ran.

Present it as a table the user can scan — scout, cadence, posture, last run, last outcome — and call out anything anomalous (never run, last run errored, stuck in dry-run for a long time).

Workflow: understand one scout end to end

"How does my error-tracking scout work / how is it doing?"

1. Read its config — find the row in config-list for signals-scout-error-tracking: schedule, posture, last run.
2. Read its body — posthog:skill-get {"skill_name": "signals-scout-error-tracking"} returns the team's actual instruction set (which may be a canonical default or a diverged, hand-edited row). This is what the agent is told to do every run — its signal-vs-noise discriminator, explore patterns, and disqualifiers. To understand why a scout behaves the way it does, read its body.
3. Read its recent runs: runs-list with skill_name set to the scout (add text to search its summaries for a topic). The end-of-run summary on each run is the scout's own account of what it looked at and decided; a failed run carries failure_reason instead.
4. Read what it remembered — scratchpad-search (see below). The memory entries a scout wrote reveal the baselines and noise it has internalized about this project.
5. Read what it was told — scout-notes-list {"skill_name": "signals-scout-error-tracking"} returns the steering notes humans left for this scout plus the general fleet-wide ones — exactly what its runs read as prior context. Each note carries an origin: human for one left directly, or a derived kind the inbox forwarded automatically: report_dismissal (a dismiss/snooze note), report_discussion (a question typed into a report's Discuss box), report_feedback (a note left with a thumbs rating), report_reviewer_correction (someone added or removed a suggested reviewer). Derived notes expire after ~30 days, and the list hides them from a caller without report read access (task:read), since they quote report content, so a credential with only signal_scout:read sees the human notes alone. Notes addressed to pipeline:report-research steer the report pipeline's research stage, not this scout, so they won't appear here. A behavior change that doesn't trace to a skill edit or a scratchpad entry often traces to a note. When asked to steer a scout with a note (rather than observe), hand off to the authoring-scouts skill, which covers the notes channel's write side.

Workflow: read recent runs

scout-runs-list returns the most recent runs across the whole fleet, newest first (capped at 100). Use it to answer "what happened lately?"

• Scope to one scout with skill_name (and skill_version to isolate the runs of one body version): the primary scoping path whenever the question is about a single scout.
• Scope to a window with date_from / date_to (ISO-8601; inclusive lower, exclusive upper on created_at). Walk backwards by passing an earlier date_to.
• Search summaries with text — a case-insensitive substring match on each run's end-of-run summary. This is how the headless scout dedupes, and it's how you find "did any run already look at the checkout error spike?"
• Filter by output with emitted — emitted=true returns only runs that authored at least one report (or, on legacy runs, emitted a finding), emitted=false only the runs that authored nothing. This is the direct way to answer "which runs actually wrote something?" without parsing prose. One caveat: a run that only edited an existing report doesn't count as emitted=true — check edited_report_ids before calling such a run quiet.

Each summary row carries run_id, skill_name, skill_version, status, started_at, completed_at, emitted_report_ids / edited_report_ids (the reports the run wrote or edited, its output), emitted_count / emitted_finding_ids (the legacy signal-channel tally, 0 / empty on current scouts), task_url (a deep-link into the Tasks UI for the full transcript), the summary prose, and, on a run that didn't complete cleanly, error (the full TaskRun error) plus failure_reason (a concise derived one-liner). Lead with the summary when narrating to the user — it's the scout's own plain-language close-out — and always offer the task_url for the full reasoning.

Workflow: drill into a single run

When the user wants the full story of one run (or pastes a run id / Tasks URL):

scout-runs-retrieve
{ "id": "<uuid>" }

Note the field name flip: runs-list returns each run's id as run_id, but runs-retrieve takes it as id. Pass the run_id value through as id.

Returns the full run: status, started_at / completed_at (compute duration from these), skill_name / skill_version (what ran, at what body version), the end-of-run summary, emitted_report_ids / edited_report_ids, metadata, and task_url. The transcript — the actual tool calls and reasoning — lives in the Tasks UI behind task_url, not in this payload; hand the user that link when they want to see every step. A failed run has an empty summary; read failure_reason first (a one-line diagnosis: a timeout at the run budget, a tool that kept erroring, a sandbox that never started) and error for the full TaskRun error text. Those two fields answer most "why did it fail?" questions on their own; reach for the transcript when the failure needs the sequence of calls that led to it.

You don't have to open the UI for that: tasks-runs-session-logs-retrieve returns the run's session log (every tool call, message, and reasoning step) as data — handy when you're diagnosing a failure or want to trace exactly what a run did without leaving the conversation. Pass the run's task_run_id as id and its task_id (both are on the run row).

The raw stream is large (hundreds of KB to a few MB) and will overflow inline, so fetch it with call --json and let it spill to a file, then run it through scripts/render_run_report.py rather than parsing it by hand. The tool returns 100 entries by default and at most 5000 per call, and its pagination state is not in the body, so pass limit: 5000 and repeat, advancing offset by the number of entries the previous page actually returned (not by limit), until a page comes back empty, then concatenate the pages before rendering; a short page is not the end, since a page also closes early at a byte cap, and stepping by limit past one would skip the entries it left out, so a single default page can drop the failure tail.

⚠️ Do not reach for exclude_types: "tool_call_update,…" to slim it down. It is tempting — the stream is dominated by incremental tool_call_update chunks — but each tool's actual input lives only in those chunks: the base tool_call event carries an empty rawInput, and the streamed updates build the input (and the final rawOutput) token by token. Excluding them leaves you with tool names but no idea what the scout actually queried. Fetch the full log and let the script reassemble each call (it groups by toolCallId, keeps the richest rawInput, and attaches the completion's rawOutput/status).

Whether a run wrote anything is a first-class field: emitted_report_ids / edited_report_ids. A non-empty emitted_report_ids lists the reports the run authored via emit_report, in order; edited_report_ids lists the reports it mutated via edit_report (which can target any inbox report, not just ones a scout authored). A productive run typically has one id there and a summary like Report authored: <id>; resolve any id via inbox-reports-retrieve to read the report itself. Don't parse the prose summary for output — a phrase like "already reported P1 … did not re-file" describes a prior run, so substring-matching the summary is unreliable; the id columns are the authoritative tally. One scout shape breaks the equivalence between "no report ids" and "wrote nothing": a measurement scout files no report on a normal run because its output is the record stream, so check metadata.derived.has_structured_output (and the events themselves) before calling such a run empty.

Legacy runs: emitted_count / emitted_finding_ids. Runs from the deprecated signal-emitting channel (a scout without the allowed_tools opt-in — an old custom scout, or a canonical scout not yet ported) tally their output as emitted_count weak findings instead; each finding_id maps to a Signal with source_id = run:<run_id>:finding:<finding_id>. For those runs only, scout-runs-emission-reports (pass the run_id) maps each emitted finding to the inbox report its signal grouped into (or null if it never surfaced). On report-channel scouts these fields are always 0 / empty — don't diagnose off them. See references/scout-data-model.md for the full field reference.