DCC UI Control

The Agent Host owns user authorization. Core never enables the standalone Host's legacy native confirmation popup, including when menu invocation is allowed. Missing authorization remains a typed refusal; do not auto-approve or retry it through another input route. For bounded task authorization, use the project-owned DCC-CUA task bridge and its exact target/scope checks.

Use application UI automation only when a typed DCC skill, host API, or adapter script reports unsupported or capability_missing. Policy denial, user interruption, authentication, and desktop unavailability are stop conditions.

Backend order

1. Use a typed DCC-MCP tool whenever one exists.
2. For Chrome, Edge, launchers, and embedded webviews with DevTools access, set DCC_MCP_UI_CONTROL_BACKEND=chrome (or edge) and use CDP.
3. For native application UI, set DCC_MCP_UI_CONTROL_BACKEND=cua and use the standalone dcc-cua CLI/Host.
4. mock is an explicit deterministic test backend only; production defaults to dcc-cua and never silently falls back to mock automation.

Core does not package an automation Host in its own release assets. The official CLI installer reconciles the independently released dcc-cua companion; verify it with dcc-mcp-cli components status dcc-cua or repair it with dcc-mcp-cli components ensure dcc-cua --yes. For custom layouts, install dcc-cua 0.4.0 or newer separately. Core probes the configured DCC_MCP_INSTALL_DIR, the platform's standard dcc-mcp bin directory, versioned standalone installs, and then PATH; set DCC_MCP_CUA_BINARY to an absolute executable path only for a custom location. Core validates dcc-cua manifest, ensures the shared Host, and keeps one persistent JSONL bridge per active UI session. It prefers shared-memory image transport when the native Core extension is present and otherwise uses bounded binary attachments.

The CUA Host owns native accessibility, capture, banner/border/cursor markers, input scheduling, Escape interruption, browser CDP routing, and platform adapters. Core owns only DCC policy, normalized tool results, audit events, and artifact publication.

Exact application scope

DccServerBase injects its trusted DccServerOptions process/window context into every in-process ui-control call. Dedicated or custom servers may override that automatic binding with:

• DCC_MCP_UI_CONTROL_PROCESS_ID
• DCC_MCP_UI_CONTROL_WINDOW_HANDLE
• optional DCC_MCP_UI_CONTROL_PROCESS_NAME
• optional DCC_MCP_UI_CONTROL_WINDOW_TITLE

Request arguments may narrow that scope but cannot widen it. Native mouse and keyboard input are enabled inside that exact scope by default; operators can disable them with DCC_MCP_CUA_ALLOW_RAW_INPUT=false. Semantic accessibility actions remain preferred. For a multi-window process, pass a window_title that narrows the trusted PID to one window. The Host resolves that title once, then returns and enforces an exact window capability; use an explicit handle if the title still matches more than one window.

A redundant process_name may accompany a trusted PID or HWND as metadata or an additional constraint. It never replaces the exact native binding: a process name or title by itself cannot mint a capability, and a process name that conflicts with an operator-trusted name fails with invalid_target.

Multiple agents may control different applications concurrently. Each logical session has its own grant, window capability, observation fences, and bridge. The shared Host isolates session state, serializes raw input, and broadcasts Escape as the operator stop signal.

Observe, act, verify

Follow this loop:

1. Call ui_control__snapshot.
2. Use ui_control__find for semantic controls when needed.
3. Call ui_control__act once with the current snapshot_id.
4. Call ui_control__wait_for for an explicit condition or take a fresh snapshot.
5. Call ui_control__stop_computer_use when the application task is complete.

Never reuse an observation after an action, target change, display change, or resume. CUA actions are fenced by both observation_id and accessibility_state_id; stale actions fail closed.

For a native application menu bar, use ui_control__act(action="invoke_menu", menu_path=[...]) when a semantic menu click or Alt mnemonic cannot prove that a Qt popup opened. This exact-window native path does not require a prior snapshot, fails closed on missing or ambiguous levels, and invalidates any current observation. Check verification_required, then take a fresh snapshot and verify the popup or resulting application state before another mutation.

Prefer semantic control_id actions. Coordinate clicks, drag paths, raw typing, shortcuts, and game navigation are native input and require the raw-input gate. Do not use UI Control to enter credentials, solve authentication challenges, or cross a secure-desktop boundary.

Recording

Trajectory recording is session-scoped and intentionally asynchronous:

1. Call ui_control__recording_start with an absolute output_dir and optional record_video=true.
2. Perform the UI actions that should become evidence.
3. Optionally call ui_control__recording_state.
4. Call ui_control__recording_stop to finalize the trajectory.
5. Preserve the finalized artifacts and structured state returned by CUA.

This replaces the removed synchronous JPEG-sequence record_clip contract. Core does not duplicate CUA's recording format.

Semantic profiles and trusted handoff

Run dcc-cua profiles, then inspect the chosen application with dcc-cua profile --id <id>. Profile target IDs stay stable across languages; localized aliases are matching inputs, not translated identifiers. Dispatch a browser_dom surface through the exact-bound dcc-cua browser route, never the in-app Browser skill. A fallback such as ue/fab/download to fab/launcher_download requires a new Epic Games Launcher binding and fresh observation. Cloudflare challenges, authentication, purchases, and operating system security confirmations require trusted human action; full agent access does not bypass them.

Browser control

CDP is the first choice for browser content because it exposes DOM semantics, stable selectors, network state, and background operation. Use native CUA only for browser chrome, file pickers, permission prompts, canvas-only content, or when CDP is unavailable. Never attach a second automation stack to the same page concurrently.

Error handling

Treat these as recoverable only when the stated precondition can be restored:

• stale_observation: take a fresh snapshot.
• missing_window or invalid_target: rediscover and rebind the exact app.
• desktop_unavailable: wait for the interactive desktop to return.
• backend_unavailable: install/configure dcc-cua or restore CDP.
• user_interrupted: stop until the operator explicitly resumes.
• policy_disabled or permission_denied: do not bypass policy.

Successful snapshots include capture_provenance. Preserve it with any visual evidence. Audit logs redact typed text and other sensitive action payloads.