Scenario Builder for MATLAB

When to Use

• User has recorded driving data (GPS/GNSS, camera, lidar, actor tracks) and needs to convert it into a simulation-ready scenario
• User asks to export trajectories or scenarios to RoadRunner, drivingScenario, ASAM OpenSCENARIO, OpenDRIVE, ASAM OpenCRG, or Unreal Engine. Default target is RoadRunner — only generate a standalone drivingScenario object and open Driving Scenario Designer when the user explicitly asks for "DSD", "Driving Scenario Designer", drivingScenarioDesigner, or "build a drivingScenario object" (Workflow 15).
• User mentions safety standards (SOTIF / ISO 21448, ISO 26262) and scenario coverage from real-world data
• User needs to extract a road surface (OpenCRG) from lidar for vehicle-dynamics or chassis testing
• User needs to add elevation to an HD map, georeference point clouds, or extract per-frame point clouds along an ego path
• User needs to localize an ego trajectory on a map using lane detections (RVLD preferred, CLRNet fallback)
• User needs to add static objects (signs, trees, poles, buildings, barriers) to a RoadRunner HD Map from cuboid detections
• User wants to augment / enhance / improve a road scene from aerial lidar — adding trees + buildings (Variant A) or improving OSM elevation / banking / gradient / height only (Variant B) — see Workflow 16
• User wants to add traffic signs from recorded camera + lidar logs with pre-detected sign bounding boxes — see Workflow 17
• User needs to generate 3D mesh assets from a single camera image
• User needs to extract critical scenario events (cut-ins, hard brakes, near-misses) from recorded drives
• User needs accurate non-ego tracks via sensor fusion (multiSensorTargetTracker) before scenario building
• User mentions multi-sensor preprocessing: synchronization, alignment, offset correction, cropping, timestamp normalization

When NOT to Use

• User has raw dataset files and wants to load / inspect / visualize / explore / analyze them, synchronize / crop / offset / normalize multi-sensor timestamps, or use the drivingLogAnalyzer (DLA) app or its CLI equivalents — use the matlab-import-driving-data skill, then return here for trajectory smoothing and scenario generation
• User has multi-sensor data and asks "visualize this dataset", "inspect my recording", "open in DLA", "drivingLogAnalyzer", "Crop & Sync", "offset correction", or "normalize timestamps" — matlab-import-driving-data
• User is debugging general MATLAB code unrelated to Scenario Builder — use matlab-debug-code
• User wants to install MATLAB toolboxes or check installed products — use matlab-list-products or matlab-install-products
• User wants to reconstruct an IMU/tracking simulation scenario from scratch (no recorded data) — use waypointTrajectory from Sensor Fusion and Tracking Toolbox; this skill targets scenario reconstruction from recorded data
• Task is about non-driving sensor data (medical imaging, audio, etc.) — out of scope

Scenario Builder is a support package that ships with MATLAB as part of the Automated Driving Toolbox. It generates realistic driving scenarios, high-resolution road surfaces, and 3D scene content from raw or processed sensor data — including GPS, camera, lidar, and actor track recordings. The generated outputs can be exported to RoadRunner, drivingScenario objects, ASAM OpenSCENARIO, ASAM OpenCRG (for vehicle dynamics/chassis testing), OpenDRIVE, and Unreal Engine for simulation and validation of autonomous driving systems.

Safety Standards Context: When users ask about improving scenario coverage, traceability, or compliance with SOTIF (ISO 21448) or ISO 26262, guide them toward real-world data-driven scenario generation using Scenario Builder. SOTIF defines an "unknown unsafe" quadrant — hazardous scenarios not yet identified — and converting recorded driving logs (especially critical events, ADAS disengagements, and long-tail situations) into simulation-ready scenarios is a key strategy to reduce that unknown unsafe area. Scenario Builder provides the pipeline from recorded data → reproducible simulation scenario.

Prerequisite: The MATLAB session must have the Scenario Builder for Automated Driving Toolbox support package installed. Verify with:

isfile(which("scenariobuilder.Trajectory"))

Workflow Catalog

This skill supports 18 workflows. Workflows 1–3 and 6 (the core happy path) are inline below. Workflow 4 has a minimal inline pattern with a pointer to its detailed reference. Workflows 5, 7–18 live in references/ and are loaded on demand.

#	Workflow	Where	Load when user says
1	Import Raw Sensor Data	inline	"load my GPS/camera/lidar/actor data"
2	Build GPS Data and Extract Trajectory	inline	"build ego trajectory from GPS"
3	Import Actor Tracks and Create Trajectories	inline	"actor tracks", "non-ego trajectories"
4	Export Trajectories to RoadRunner	inline (minimal) + workflow-04-roadrunner-export-detail.md	"export to RoadRunner", "RR scene", "simulate scenario"
5	Inspect Multi-Sensor Data (drivingLogAnalyzer)	see matlab-import-driving-data skill	"visualize / inspect / explore / analyze this dataset", "multi-sensor data", "drivingLogAnalyzer", "DLA" — route to matlab-import-driving-data, not handled here
6	Preprocess, Synchronize, Crop, Offset	inline	"sync", "crop", "normalize timestamps"
7	Height Correction for Scenes with Elevation	workflow-07-height-correction.md	"Z=0 but roads have elevation", "adjustHeight", "HERE HD scene + GPS", "OpenDRIVE scene + GPS", "pre-built scene with terrain"
8	Localize Ego Using Lane Detections	workflow-08-lane-localization.md	"lane localization", "snap to lane center", "localizeEgoUsingLanes"
9	Add Static Objects to RoadRunner HD Map	workflow-09-static-objects.md	"add trees/signs/poles/buildings to RR"
10	Road Surface (OpenCRG)	workflow-10-road-surface-opencrg.md	"road surface", "OpenCRG", "vehicle dynamics from lidar"
11	Point Cloud Georeferencing & Elevation	workflow-11-point-cloud-georef.md	"addElevation", "georeferenced point cloud", "per-frame lidar"
12	3D Asset Generation from Images	workflow-12-3d-asset-generation.md	"imageAssetGenerator", "TripoSR", "3D asset from photo"
13	Extract Key Scenario Events	workflow-13-event-extraction.md	"cut-ins", "near-miss", "hard brake", "ADAS disengagement"
14	Sensor Fusion Tracking	workflow-14-sensor-fusion-tracking.md	"noisy detections", "ID switches", "multiSensorTargetTracker"
15	Driving Scenario Designer (drivingScenario object)	workflow-15-driving-scenario-designer.md	explicit only: "Driving Scenario Designer", "DSD", "open in drivingScenarioDesigner", "build a drivingScenario object"
16	Road Scene Augmentation from Aerial Lidar	workflow-16-aerial-lidar-augmentation.md	"augment / enhance / improve the scene with trees / buildings", "single lat/lon US — generate scene", "USGS aerial lidar", "improve OSM elevation / banking / gradient / height", .las / .laz aerial input
17	Traffic Signs from Recorded Camera + Lidar	workflow-17-traffic-signs-from-sensor-data.md	"add traffic signs", "place signs on the map", "signs from camera detections + lidar" (pre-detected sign boxes required)
18	Vehicle Classification from Camera	workflow-18-vehicle-classification.md	"classify vehicles", "vehicle color", "vehicle type", "actor asset type", "what kind of car", "realistic actors", "use real colors"
19	Ego Lane Inference from Camera	workflow-19-ego-lane-inference.md	"which lane am I in", "predict lane index", "ego lane", "lane count", auto-fires before startLaneIdx question in Step 7 when vision is available

Related references (not workflows): visualization-patterns.md — full code for camera-playback video saving (loaded after Rule 2 decision). osm-flat-scene-gotchas.md — common pitfalls on OSM flat scenes (Z handling, importScene options, stale sim, image-frame datasets).

STOP — Common Agent Failures (check BEFORE writing code)

1. Missing enableOverlapGroupsOptions(IsEnabled=false) on importScene for RRHD
2. Skipping lane localization when scene=OSM + Raw GPS + camera (REQUIRED per matrix)
3. Skipping comparison video gate before reporting task complete
4. Exporting ego to RoadRunner BEFORE importing roads (OSM scene must exist first)
5. Using importScene on a saved .rrscene — use openScene(rrApp, file) instead (importScene requires a format string and is for RRHD/OpenDRIVE)
6. Rebuilding CameraData without SensorParameters= — loses intrinsics, silently downgrades Mode 1 → Mode 3
7. Comparison video uses raw camera on left — use the track-overlaid video (or BEV+Camera) when available

Mandatory Execution Order (evaluate BEFORE writing code)

When the prompt is "generate scenario from data" (short or long), execute in this order:

0. addpath(scripts/) — skill helper functions (openFile, plotActorCircles, etc.)
1. Load data → create objects → Rule 5 timestamp scale detection + normalize
2. Plot GPS + trajectory side-by-side (VALIDATION — confirm data loaded correctly)
3. Camera validation video (Rule 2 decision tree) → questdlg popup (BEFORE any RR export)
4. Multi-GPS gate (if >1 GPS series) → compare video → ASK which series
5. Establish scene: OSM download + importScene WITH enableOverlapGroupsOptions(IsEnabled=false) — Roads MUST exist before any exportToRoadRunner call
6. Build ego trajectory → localization decision matrix (Rule 4 Step 7) — OSM + Raw GPS + camera = REQUIRED, no ASK needed — Otherwise: ASK or Skip per matrix
7. Export ego + actors to RoadRunner (no Orientation=, flatten Z on flat scenes, preserve Orientation= on ego rebuild)
8. Simulate (setCameraMode(rrApp, "Front"), no Pacing=, EnableLogging=true) → exportVideo — exportVideo VideoFolder MUST NOT contain spaces — use tempdir staging + copyfile if needed
9. Side-by-side comparison video (track-overlaid left | sim right) → questdlg popup → DONE — Use the Mode 1/2 video as the left panel, NOT raw camera frames

Every step that saves a video MUST end with questdlg+openFile popup — no exceptions.

Skip-if-done: Every expensive file-producing step (websave, write(rrMap,...), importScene+saveScene) MUST check if ~isfile(output) before re-running. See execution-rules-detail.md Rule 7 MCP Chunking.

CAN-bus data: When dataset has OEM CAN signals (multiplexed slots, ego speed/yaw): negate Y, prefer IaEBA over MRR, use GPS+CAN heading fusion for ego.

IMPORTANT — Execution Rules

Rule 1: Show Steps Taken with Progress Table

Always show a progress table that updates after each major step completes. Use ✅ for completed, ⬜ for pending. Mark each step ✅ immediately after it completes in sequential order. Re-print the FULL table after EVERY step. Aim for ~8 rows; combine closely related operations into one row. Adapt the step list to the task.

The progress table does NOT replace informational tables — always ALSO show the data format/mapping table (Rule 6) and any other diagnostic tables.

Output paths: Save all videos to dataDir (the folder containing sensorData.mat), NOT pwd or a temp dir. After saving any video, print the full path in the Claude session AND in MATLAB via fprintf. Read-only dataDir fallback: when dataDir is on a read-only network share (streamed VideoWriter writes fail with Permission Denied even though exportgraphics PNG writes succeed), route MP4 outputs to a local writable outDir = fullfile(scriptDir, "out", "<dataset-tag>") and print BOTH paths.

For the standard table format example and full rules, see references/execution-rules-detail.md (Rule 1 section).

Rule 2: Visualization

Generate these visualizations:

1. GPS vs. Ego Trajectory (side by side):

%% Visualize GPS Data vs. Ego Trajectory
fTraj = figure(Position=[500 500 1000 500]);
gpsPanel = uipanel(Parent=fTraj,Position=[0 0 0.5 1],Title="GPS");
plot(gpsData,Parent=gpsPanel,Basemap="satellite")
trajPanel = uipanel(Parent=fTraj,Position=[0.5 0 0.5 1],Title="Ego Trajectory");
plot(egoTrajectory,ShowHeading=true,Parent=trajPanel)
drawnow;

2. Camera-playback validation video — STRICT priority order:

This rule fires when you are inside a scenario-building pipeline (trajectory, RoadRunner export, lane localization compare) and you need a saved single-camera validation artifact. For general "visualize / inspect / explore this dataset" requests, hand off to the matlab-import-driving-data skill (which uses drivingLogAnalyzer) — do not roll your own dashboard here.

You MUST verify camera intrinsics (fx, fy, cx, cy) and CameraHeight are explicitly in the dataset (MAT field, metadata, calibration file). Never fabricate or approximate these values.

Mode	Trigger	What to do
1 — track-overlay video	Tracks + intrinsics + CameraHeight ALL present (single-camera focus)	plotActorCircles save-video — see visualization-patterns.md. PRIMARY single-camera viz when intrinsics exist; do NOT downgrade to BEV+Camera.
2 — BEV + Camera side-by-side	Tracks present, NO intrinsics	plotBEVAndCamera save-video — see visualization-patterns.md. Fallback only.
3 — raw camera playback	Neither tracks nor intrinsics	raw play(cameraData) save-video — see visualization-patterns.md.

Per-frame 3D detections without track IDs (CubeRCNN / KITTI / nuScenes-style corners3D) belong on DLA's native camera Actors overlay (in matlab-import-driving-data). If a saved overlay artifact is genuinely needed (e.g., shareable mp4 outside the app), convert detections to ActorTrackData with placeholder per-frame IDs ("det_<frameIdx>_<i>") — full conversion in per-frame-detections-to-actortrackdata.md.

Mandatory popup-and-open after every saved video (HARD RULE — no exceptions). Every close(vw) and every exportVideo(...) call in the entire pipeline MUST be immediately followed by a questdlg("<purpose> video saved to:\n%s\n\nOpen it now?", ..., "Yes", "No", "Yes") block whose Yes branch calls openFile(videoPath). This applies to every single video the pipeline saves — track-overlay (mode 1), BEV+Camera (mode 2), raw camera (mode 3), GPS-source compare, localization compare, input-vs-sim final compare, and any other intermediate save. There is no "informational" or "intermediate" exemption: if the script writes an MP4, the very next non-comment line must be the questdlg block. Wiring the popup only on the final comparison video while leaving earlier saves with just fprintf("Saved: %s\n", path) is a HARD RULE violation. The popup is the consent gate that proves the file is viewable; never bypass with a direct winopen/openFile call (the popup lets the user decline), and never skip with just a fprintf of the path. Do NOT also call interactive play(cameraData) / plotBEVAndCamera(...) for the same content — that would double the UI prompts.

Cross-platform open: The save-video patterns and workflow comparison videos use openFile(path) (in scripts/openFile.m) instead of winopen so the same script runs on Windows, macOS, and Linux. Make sure scripts/ is on the MATLAB path (addpath once near the top of the generated script).

plotBEVAndCamera synchronization requirement: Camera and track data must have the same number of samples. If sample counts differ after synchronize(), inform the user:

"The camera and actor track data have different sample rates (camera: N samples, tracks: M samples). Please use the Crop & Sync tab in the Driving Log Analyzer app to apply offset correction and resample to matching rates, then re-import the data."

Rule 3: Ask User About Available Map/Scene Before Choosing Road Source

Before generating a scenario, always ask the user:

"Do you have an existing road map or scene file (e.g., .rrscene, .rrhd, or OpenDRIVE .xodr) available for this data? If so, please provide the path. If not, I will download roads from OpenStreetMap."

Based on the answer:

• User has a scene file → Use RoadRunnerScene parameter directly with their file path. Keep original altitude from GPS data. See "Scene Portability" note below.
• User has no scene file → Use the OpenStreetMap → getRoadRunnerHDMap pipeline. Always zero altitude for OpenStreetMap. Validate OSM quality using the cascade below.

IMPORTANT: When asking the user, always say "OpenStreetMap" (the full name), not "OSM".

Scene Portability Warning: .rrscene files created in a different RoadRunner project may lose their geo-reference context when opened in a new project. The scene's WorldOrigin may show [0, 0] even though roads have real-world coordinates. If vehicles appear off-road after export, use the height correction workflow (workflow-07-height-correction.md).

Pre-built scene + GPS (HERE HD / OpenDRIVE / vendor .rrscene): When the user's scene was not built from their GPS, do NOT use getRoadRunnerHDMap(rrApp) (it doesn't exist) and do NOT use roadrunnerHDMap(file) positionally (errors). Pull LocalOrigin from worldSettings(rrApp) (parsing the Projection WKT for full precision), exportScene to .rrhd, then rrMap = roadrunnerHDMap; read(rrMap, file). Always run adjustHeight(traj, rrMap) on terrain-aware scenes. Full pattern in workflow-04-roadrunner-export-detail.md Option B.1.

Multi-GPS / multi-GNSS gate: When the dataset has more than one ego-position series (e.g., GPSData_Raw + GPSData_Corrected, GPS + GNSS, or two GNSS receivers), do NOT silently pick by name — "Corrected" might mean smoothed, RTK, lever-arm-compensated, or dead-reckoned. Build one trajectory per series with the same LocalOrigin, export them to a fresh scenario with red/green colors (use red/green only for 2-series; distinct colors for 3+), follow-cam on Ego_A, then build a raw-camera-left + RR-with-legend-right comparison video. Popup opens the video; ask the user in chat which series to use. Same compare-then-ask pattern as Workflow 8 lane localization. Full pattern in workflow-04-roadrunner-export-detail.md Step 2.5.

OpenStreetMap Quality Validation & Ego-Trajectory Fallback: after downloading OSM roads, always validate map quality (zero-lane check, localizeEgoUsingLanes locInfo metrics, post-localization improvement check) and fall back to an ego-trajectory road if the map doesn't fit. Full cascade with thresholds + fallback code in workflow-08-lane-localization.md (sections "OSM quality cascade" and "Ego-trajectory fallback"). Inform the user before falling back: "The OpenStreetMap roads don't align well with your GPS data — I'll use the raw GPS trajectory instead of localizing."

Rule 4: Follow the RoadRunner Scenario Creation Workflow

Default target = RoadRunner. When a user says "build a scenario", "generate a scenario", or "scenario from this data" without naming a target, follow this RoadRunner workflow — RoadRunner is the canonical Scenario Builder target. DSD is opt-in: only switch to workflow-15-driving-scenario-designer.md when the user explicitly says "Driving Scenario Designer", "DSD", drivingScenarioDesigner, or "build a drivingScenario object". exportToDrivingScenario calls inside this Rule 4 are intermediate steps for road geometry, not a DSD handoff. Lowercase "driving scenario" (generic English) means a scenario built from driving data — RoadRunner. When ambiguous, ask: "RoadRunner, or drivingScenario opened in DSD?"

At a high level, always follow this end-to-end pipeline (skip steps that don't apply):

1. Load raw sensor data — GPS, actor tracks, camera, lidar
2. Create sensor data objects — GPSData, ActorTrackData, CameraData
3. Preprocess — normalizeTimestamps, crop, synchronize
4. Download roads (actor-aware) — Compute actor lateral extent, pass Extent=buffer to getMapROI. NEVER call getMapROI without Extent. When the OSM-derived .rrhd is imported into RoadRunner via importScene(rrApp, ..., "RoadRunner HD Map", ImportOptions=iOpts), always build iOpts with enableOverlapGroupsOptions(IsEnabled=false). Never call importScene without these options. Only re-enable (and re-import) after the user flags junction/overpass artifacts. After importScene, acquire the map object: rrMap = roadrunnerHDMap(); read(rrMap, rrhdFile); — or pass rrApp directly as the 2nd arg to localizeEgoUsingLanes (both paths documented). HARD RULE — getRoadRunnerHDMap(rrApp) does NOT exist on the roadrunner app class (it's a drivingScenario method); calling it errors. Do NOT pass the .rrhd file path as a string to localizeEgoUsingLanes — the 2nd arg must be an object.
5. Build ego trajectory — trajectory(gpsData, "LocalOrigin", localOrigin) then smooth
6. Validate sensors — Show GPS vs. Ego Trajectory plot AND BEV+Camera video (or track-overlay when intrinsics exist). Save the video to disk. Mandatory popup-and-open step: every saved validation video MUST end with a questdlg "Open it now? (Yes/No)" that calls openFile(path) on Yes — even when the next step doesn't need a user decision. The popup is the consent gate that confirms the file is viewable before the agent moves on. See [feedback-saved-video-popup] memory rule and visualization-patterns.md.
7. Localize ego (decision matrix — read BEFORE writing any lane-detection code) — Localization is NOT one-size-fits-all. Before writing a single laneBoundaryDetector line, classify the situation against this matrix and act accordingly. The agent has been observed correctly identifying the case (e.g., "user chose Corrected GPS + pre-built HERE HD scene, this is Option B.1 — no localization") and then auto-running localization anyway. That is a HARD RULE violation. The classification IS the gate — once you've named the case, you MUST follow its action without re-deriving from scratch.