Agent Visual Feedback (Describe this)

Use this skill when the operator wants the agent to look at the current viewer and give actionable feedback — not a generic caption.

When To Use

• UI Describe this button (stays on the Rerun/viewer tab; chat opens as a drawer)
• Chat turns containing [npa-visual-feedback] or “describe this viewer/visual”
• Interpreting held-out Rerun frames, Isaac/GR00T-style sim views, rollout video, images, or Data-pane JSON
• For Sim2Real recordings, inspect summary/stage_progress and summary/policy_access alongside rollouts/outer_*/iter_*: camera and action entities share the same frame timeline; policy access is a link/download action and must never be described as policy execution inside Rerun.
• Metadata-only feedback when a frame cannot be captured

Model

1. Prefer a quality-captured frame (vision tier → MiniMaxAI/MiniMax-M3).
2. Wait for a non-blank canvas (skip uniform black/white/mid-gray; dense RGB, skeletons on dark grids, and meshes are valid). Cleared WebGL buffers often look mid-gray — never attach those. Blank detection must keep enough resolution to see sparse orange/cyan skeleton strokes (do not 80px-downscale them away).
3. Rerun capture must use the MediaStream bridge (canvas.captureStream → <video> → JPEG). Do not gate capture on sync drawImage blank checks — those false-negative on live WebGL/WebGPU.
4. If capture fails, send metadata/text and use the reasoning tier — never pretend pixels were seen.
5. Do not answer Describe-this from the grounded intent router.
6. Open the chat drawer and show Describe this — capturing… immediately; do not wait for capture before the user bubble appears.

Visual kinds (generalized — no URI allowlists)

Kind	Source	What to emphasize
rerun	Largest same-origin Rerun canvas after quality wait	Sim RGB, depth/seg, 3D mesh, tiled envs, policy strips — not “blank” by default
video	<video> current frame	Task progress, success/failure cues
image	Preview <img>	Scene contents, defects
data	<pre> / text excerpt	Report fields, success_rate, missing keys

Domain hints are inferred from free-text metadata tokens (artifact_key, notes, workflow name) such as isaac / gr00t / heldout / genesis / cosmos — never from a hardcoded path allowlist.

UX

• On the Viewer tab (Rerun/Video/Image/Data), chat is a bottom-right collapsible widget (viewer-focus + FAB).
• Clicking the Chat main tab from Viewer opens the widget (stays on the media pane); Full chat expands to the Chat tab.
• Describe this captures in-place, opens the widget, and queues the vision turn.
• Chat sends are queued (enqueueChatJob) so Describe + typed messages do not drop.
• Frame captures are JPEG-capped (maxChars) and nginx allows client_max_body_size 32m so multimodal POSTs do not 413 / Failed to fetch.
• Do not prefetch .rrd bodies (many runs). Soft-swap uses add_receiver only; warm wasm/js separately.

Agent API flow

POST /api/chat
{
  "session_id": "default",
  "visual_context": {
    "kind": "rerun",
    "run_id": "demo-workbench-ui",
    "artifact_key": "…/reports/sim2real.rrd",
    "frame_quality": "rendered",
    "capture": "frame"
  },
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "[npa-visual-feedback] Describe this rerun viewer…"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
      ]
    }
  ]
}

Reply shape (required)

1. What I see
2. Likely meaning
3. Operator feedback
4. Next actions

Implementation pointers

• Helpers: npa/src/npa/cli/agent_visual_feedback.py
• UI: describeVisual, waitForQualityRerunFrame, queueChatText, viewer drawer
• Contracts: AGENT_VISUAL_FEEDBACK_CONTRACT, AGENT_CHAT_QUEUE_CONTRACT, AGENT_VIEWER_CHAT_DRAWER_CONTRACT