UI harness — drive the app without a mouse
SkillDev toolsDrive and observe Quantick UI surfaces through hooks and screenshots.
Available today. Use it from your connected AI after setup.
No other account needed.
Connect ahel once, and every AI you use reads what you have installed.
Then ask your AI: use the UI harness — drive the app without a mouse skill
What this skill tells your AI
The instructions your AI receives, as published by milocaetano/quantick in .agents/skills/ui-harness/SKILL.md and read by ahel’s review.
The contract that makes autonomous visual work possible:
Every user-visible surface (panel, layer, tab, popup, demo flow) must be reachable from a fresh launch via environment hooks alone — zero clicks.
A PR that adds a surface without a hook leaves that surface untestable by
agents; that is a Should-fix in review. Hooks follow the existing family:
QUANTICK_<SURFACE>_AUTOSTART=1 reuses the exact code path of the manual
toggle — never a parallel activation path — and defaults to off, so a hook
never changes behaviour for a user who did not set it.
Hook registry
The registry lives in references/hook-registry.md, beside this file. It is
one row per hook — | QUANTICK_… | what it reaches | — so grep it for the
surface you need rather than reading it whole:
grep -i 'heatmap\|book' .claude/skills/ui-harness/references/hook-registry.md
It was moved out of this file rather than shortened, and no row has ever been dropped. The table is 69KB — five sixths of this skill — and it is data, looked up one row at a time by a run that drives one or two surfaces. Loading it whole to answer "what turns the heatmap on" was the single largest token cost in this repository's whole agentic flow, paid on every capture. A grep answers the same question from the same rows.
Read it whole when you genuinely need the whole thing: taking inventory of what is reachable, or auditing coverage before a release. That is the rare case, and it now costs the same as it always did instead of being charged to every run.
The registry is generated and cannot lie about what exists. Rows come from
the declare_hooks! line beside each read, fused with the prose in
docs/ui-harness/hook-prose.md; a hook read but not described, or described
but not read, fails cargo test -p quantick-guards. Edit the prose, never the
registry, then cargo run -p quantick-app -- --dump-hook-registry over it.
No one file owns the hooks. harness.rs holds 24 of the 126; the rest are declared where they are read, across 37 files (50 in
app/launch_hooks.rs, 8 in paper_trading.rs, 6 in
surfaces/drawing_chrome/mod.rs). The registry's Declared in column is the
answer. Every launch hook is applied in crates/app/src/app/launch_hooks.rs,
in the order its doc comment fixes — that module is the application point.
A QUANTICK_* nothing reads is logged at startup as UNKNOWN_HOOK.
Launch and capture workflow
- Own target dir, on a drive with room: build with
CARGO_TARGET_DIR=D:\quantick-agent-targetso the user's running exe is never locked and rust-analyzer never poisons fingerprints. It wasF:until that drive stopped existing — checkGet-PSDrive -PSProvider FileSystembefore trusting this line, and pick the drive with free space:C:runs into single-digit gigabytes with a few worktrees on it, and a build that dies of ENOSPC looks like a compile error until you read the message. - Fresh exe, proven fresh:
cargo build -p quantick-appimmediately before capturing, then compare the exeLastWriteTimeagainst your last edit.cargo testgreen does not imply the exe was rebuilt. - Launch via PowerShell
Start-Processwith hooks set andRUST_LOG=quantick=info, stderr to a log file. A bash background job produces a window whose GL surface never presents (pure-white captures). - Capture by PID, never by window title: use
tools/capture_window.ps1(PrintWindow with PW_RENDERFULLCONTENT) adapted to filter by the PID you launched — title matching grabs the wrong window when other instances or editors are open. - Gate on health before trusting a capture:
APP_HEALTH_SUMMARYprints every 2 s.fps≈59 / frame_avg≈16.7→ surface presents, capture is real.fps≈19 / frame_avg≈52 / frame_cpu≈3→ occluded or idle desktop, capture will be blank; wait for fps ≥ 50 in the log and recapture. Blank capture is an environment state, not a render regression — run amaincontrol build before blaming the change. - Verify by pixel when the eye can be fooled: read the PNG (e.g.
System.Drawing) to count/locate marks, match dash signatures, or compare two frames; usereadable_min_radiusfromconfig/bubbles.tomlas the "too small to read" reference. - Be a guest on the desktop: never fight the user — if the window gets
minimized or the mouse is active (
GetLastInputInfoidle ≈ 0), stop driving, keep the evidence you have. Do not inject input withSendInput. Never bind a second MT5 listener on the user's port (9100) — useQUANTICK_CONFIGwith an alternatelisten_addr. Close every instance you opened when done.
Reading the running app through the control plane
A screenshot shows what a window looks like. It does not say what the application believes — which market, which revision, how late the tape is, whether a control is disabled and why. The control plane answers that in structured data, and an assertion against it is worth more than an assertion against pixels: it does not move when a colour does.
Prefer this over reading a capture whenever the question has a structured answer. Keep the screenshot for the questions only pixels can answer — clipping, font, composition, "does this read".
The fixture. Launch with local access enabled and the scopes the read
needs. The scope IDs, and what each one reaches, are the
QUANTICK_CONTROL_ACCESS / QUANTICK_CONTROL_SCOPES /
QUANTICK_CONTROL_EVIDENCE rows of references/hook-registry.md — named
rather than pointed at, because "the table above" stopped being true the day
the registry moved out of this file:
$env:QUANTICK_CONTROL_ACCESS = "1"
$env:QUANTICK_CONTROL_SCOPES = "all-reads,observe.evidence,observe.screenshot"
The client. quantick-mcp is a STDIO MCP server; feeding it JSON-RPC lines
is a complete client, no extra tooling. It discovers the running instance
itself and never starts one.
Build it first — step 1 of the launch workflow builds quantick-app only, and
the adapter is a separate binary in the same target directory:
$target = "D:\quantick-agent-target" # the same one the launch used
cargo build -p quantick-mcp
$mcp = Join-Path $target "debug\quantick-mcp.exe"
$lines = @(
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"ui-harness","version":"1"}}}',
'{"jsonrpc":"2.0","method":"notifications/initialized"}',
'{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"quantick_get_scene","arguments":{}}}'
) -join "`n"
$lines | & $mcp --profile observer
Send a blank line first. Windows PowerShell 5.1 writes a UTF-8 preamble to
a child's stdin the moment Process.StandardInput is touched, and it lands on
line 1 — so the initialize frame comes back -32700 parse error: expected value at line 1 column 1 while every line after it parses, which reads as "the
adapter is broken" and is not. A leading newline takes the preamble:
$psi = New-Object System.Diagnostics.ProcessStartInfo
$psi.FileName = $mcp; $psi.Arguments = "--profile observer"
$psi.RedirectStandardInput = $true; $psi.RedirectStandardOutput = $true
$psi.UseShellExecute = $false
$m = [System.Diagnostics.Process]::Start($psi)
$nl = [char]10
$bytes = [System.Text.Encoding]::ASCII.GetBytes($nl + ($lines -join $nl) + $nl)
$m.StandardInput.BaseStream.Write($bytes, 0, $bytes.Length)
$m.StandardInput.BaseStream.Flush(); $m.StandardInput.Close()
$m.StandardOutput.ReadToEnd()
One instance at a time, or discovery answers control.instance_ambiguous
and names the ids rather than choosing. Clear strays by path, never by
process name: Get-Process quantick-app | Stop-Process takes the trader's own
window down with yours, which is the be a guest on the desktop rule above,
broken by a one-liner.
Every answer is one JSON line on stdout; result.structuredContent is the
capability's own result, and result.isError with a control.* code is a
refusal you can branch on. Useful calls:
| Ask | Call |
|---|---|
| What is on screen, by name | quantick_get_scene |
| Which market, which bars, which layout | quantick_get_snapshot with the scopes |
| Is the frame healthy, is the tape late | quantick_get_diagnostics |
| What changed since I looked | quantick_read_events / quantick_wait_for_change |
| Everything at one instant, hashed | quantick_capture_evidence |
Evidence bundles. quantick_capture_evidence freezes the named scopes, the
events around them and the effective configuration into one hashed bundle and
answers with a manifest. Read it back with quantick_invoke on
evidence.read, page by page, and concatenate the base64 chunks: the bytes are
the bundle's canonical JSON and their SHA-256 is the manifest's
content_digest. Two fields decide whether an assertion is sound:
coverage— what the capture left out, and why, as codes. A scope you did not name is inomitted_scopes; a field the application could not fill is inunavailable_fieldswith the JSON Pointer that finds it.completeis never true, and a capture never pretends to be the whole session.screenshot.capture_revision— equal to the bundle's owncapture_revision, which is what makesscreenshot.control_regionstrustworthy: each named control's rectangle in the image, in physical pixels, withwithin_imagesaying whether the window was clipping it. That is the pair a visual defect is diagnosed from — the picture plus the names.
Without a client on the socket, QUANTICK_CONTROL_EVIDENCE takes the same
capture from a launch and logs the manifest as CONTROL_EVIDENCE_CAPTURED.
Bundles live in memory for fifteen minutes, are cleared when access is turned
off, and are never written to disk.
Adding a new hook
New surface → new QUANTICK_* env hook in the same commit: read the var, call
the same function the manual toggle calls, default off. Then two more edits,
both enforced: add the name to that module's crate::hooks::declare_hooks![…],
and a row to docs/ui-harness/hook-prose.md. Regenerate the registry; do not
hand-edit it. Miss either and cargo test -p quantick-guards fails — a hook
nobody can find is a surface nobody can reach.
Where the var is read depends on what it reaches, and getting this wrong is now a build failure rather than a style note. There are exactly two homes, and neither of them is the trunk:
- A hook the window owns — a menu pressed open, a pointer parked, a demo
staged, a page of history asked for, a budget counted down over frames —
lives in
crates/app/src/harness.rs. One field onHarness, one line inHarness::from_env, one accessor named for what the hook is for. The trunk's own line is then the single call that asks for it. That module's header carries the argument. - A hook a floating surface owns — its hook lives in that surface's own
module under
crates/app/src/surfaces/, as anapply_env_hookthe registry calls. Not another line in the window's own file;crates/guards/src/size.rsfails a branch that adds it to the trunk instead.
So: a surface's hook goes beside the surface; every other hook goes in
harness.rs. If you are about to add a std::env::var call to the window,
the answer is almost always one of those two files instead.
Almost: about fifty launch reads stay in app/launch_hooks.rs, applied to
the built window and not debt in the same sense — see
docs/agentic-development.md. A hook that keeps a field and needs nothing but
its own parsed value belongs in the owner.
Prefer a defaulting field to a new variant. A hook that already exists and
needs a second dimension — "the same demo, but shared across the split", "the
same profile, but left selected" — becomes a field on that hook's struct
(DrawingsDemo, FrvpDemo, DrawingDraft), defaulting to "did not ask". It
does not become a new arm of an enum, which reopens every call site that
matches on it.
Signals
- GitHub stars
- 36
- Forks
- 4
- Last commit
- Sep 2026
Advanced
- Catalog kind
- skill
- Gateway key
ui-harness- Source
- github.com/milocaetano/quantick