start-qa

Cross-runtime: follow runtime compatibility for invocation, delegation, configuration precedence, state paths, and permissions.

You are helping execute QA tests from a test plan file generated by /plan-qa. This command reads the YAML test plan, executes each test case, and reports results.

Step 1: Load Configuration

Check for configuration:

if [ -f ".git-workflow/config.yaml" ]; then
  CONFIG_PATH=".git-workflow/config.yaml"
elif [ -f ".claude/config.yaml" ]; then
  CONFIG_PATH=".claude/config.yaml" # legacy read-only fallback
else
  CONFIG_PATH=""
fi

Load from the resolved CONFIG_PATH (if one exists):

qa:
  apiBaseUrl: ${API_BASE_URL}
  testPlansDir: tests/qa
  resultsDir: tests/qa/results
  timeout: 10

Default Values:

qa:
  testPlansDir: tests/qa
  resultsDir: tests/qa/results
  timeout: 10

Step 2: Parse Arguments

Extract from $ARGUMENTS:

$ARGUMENTS

Determine Test File:

Input	Resolution
Direct path	Use as-is: tests/qa/my-test.yaml
Ticket ID (PROJ-123)	Look for: {testPlansDir}/proj-123-test.yaml
latest	Find newest .yaml in {testPlansDir}
Empty	Find newest .yaml in {testPlansDir}

Resolution Logic:

TEST_DIR=$(config.qa.testPlansDir || "tests/qa")

if [ -f "$ARGUMENTS" ]; then
  # Direct path provided
  TEST_FILE="$ARGUMENTS"
elif [[ "$ARGUMENTS" =~ ^[A-Z]+-[0-9]+$ ]]; then
  # Ticket ID - look for matching file
  TICKET_LOWER=$(echo "$ARGUMENTS" | tr '[:upper:]' '[:lower:]')
  TEST_FILE="${TEST_DIR}/${TICKET_LOWER}-test.yaml"
elif [ "$ARGUMENTS" == "latest" ] || [ -z "$ARGUMENTS" ]; then
  # Find most recent test file
  TEST_FILE=$(ls -t ${TEST_DIR}/*.yaml 2>/dev/null | head -n 1)
fi

Step 3: Load Test Plan

Read and parse the YAML test file:

if [ ! -f "$TEST_FILE" ]; then
  echo "Error: Test file not found: $TEST_FILE"
  exit 1
fi

Extract Configuration:

name: "Test Plan Name"
base_url: "${API_BASE_URL}"
timeout: 10
test_cases: [...]
sqs_assertions: [...]

Environment Variable Substitution:

Replace ${VAR_NAME} patterns with actual environment values.

Step 4: Execute Test Cases

Immediately before delegating or executing the first test, summarize the plan file, environment/base URL, number and kinds of external operations, credentials or queues involved, and result destination. Ask for explicit confirmation to run the plan. Do not treat automatic skill invocation or plan-file selection as approval.

Optional — delegate execution. For large or complex test plans (many endpoints, SQS event verification), delegate execution to the qa-executor agent through the active host subagent mechanism; it runs the plan and returns a detailed pass/fail report, keeping this session's context lean. For small plans, execute inline as below.

For each test case in the plan:

HTTP Request Execution

Using curl or available tools:

# Build request
URL="${BASE_URL}${ENDPOINT}"
METHOD="${TEST_CASE.method}"
HEADERS="${TEST_CASE.headers}"
BODY="${TEST_CASE.body}"
PARAMS="${TEST_CASE.params}"

# Add query params
if [ -n "$PARAMS" ]; then
  URL="${URL}?$(urlencode $PARAMS)"
fi

# Execute request
START_TIME=$(date +%s%N)

RESPONSE=$(curl -s -w "\n%{http_code}" \
  -X "$METHOD" \
  -H "Content-Type: application/json" \
  ${HEADERS[@]} \
  -d "$BODY" \
  --max-time $TIMEOUT \
  "$URL")

END_TIME=$(date +%s%N)
DURATION_MS=$(( (END_TIME - START_TIME) / 1000000 ))

# Parse response
HTTP_CODE=$(echo "$RESPONSE" | tail -n 1)
RESPONSE_BODY=$(echo "$RESPONSE" | sed '$d')

Using MCP Tools (If Available)

If MCP tools are available:

mcp__ai-qa-tools__check_api_access(
  base_url: BASE_URL,
  method: METHOD,
  path: ENDPOINT,
  headers: HEADERS,
  body: BODY,
  timeout: TIMEOUT
)

Validate Response

Status Code Check:

const passed = responseCode === expected.status;

Body Assertions:

// Exact match
if (expected.body.field === response.body.field) {
  // Pass
}

// Wildcard match (*)
if (expected.body.id === '*' && response.body.id != null) {
  // Pass - any non-null value
}

// Array check
if (Array.isArray(expected.body.items) && Array.isArray(response.body.items)) {
  // Pass
}

Record Result

{
  id: "TC-001",
  name: "Test name",
  status: "PASS" | "FAIL" | "SKIP" | "ERROR",
  duration_ms: 150,
  expected_status: 200,
  actual_status: 200,
  assertions: [
    { field: "status", expected: 200, actual: 200, passed: true },
    { field: "body.success", expected: true, actual: true, passed: true }
  ],
  error: null | "Error message"
}

Step 5: Execute SQS Assertions (If Defined)

For each SQS assertion:

Using MCP Tools

mcp__ai-qa-tools__verify_sqs_message(
  queue_url: QUEUE_URL,
  match_mode: MATCH_MODE,
  expected_content: EXPECTED_CONTENT,
  expected_fields: EXPECTED_FIELDS,
  timeout_seconds: TIMEOUT
)

Manual Verification

# Poll SQS queue
START_TIME=$(date +%s)
TIMEOUT_TIME=$((START_TIME + TIMEOUT_SECONDS))

while [ $(date +%s) -lt $TIMEOUT_TIME ]; do
  MESSAGES=$(aws sqs receive-message \
    --queue-url "$QUEUE_URL" \
    --max-number-of-messages 10 \
    --wait-time-seconds 5)

  # Check each message against expected fields
  for message in $MESSAGES; do
    if matches_expected "$message" "$EXPECTED_FIELDS"; then
      echo "SQS assertion passed"
      break 2
    fi
  done
done

Step 6: Generate Results Report

Console Output

═══════════════════════════════════════════════════════════════════════
  QA Test Results: {TEST_NAME}
═══════════════════════════════════════════════════════════════════════

Test Plan: {FILE_NAME}
Base URL: {BASE_URL}
Executed: {TIMESTAMP}

───────────────────────────────────────────────────────────────────────
  Test Cases: {PASSED}/{TOTAL} passed
───────────────────────────────────────────────────────────────────────

┌──────────┬────────────────────────────────────┬────────┬──────────┐
│ ID       │ Name                               │ Status │ Duration │
├──────────┼────────────────────────────────────┼────────┼──────────┤
│ TC-001   │ Happy path - get users             │ PASS   │ 145ms    │
│ TC-002   │ Invalid input returns 400          │ PASS   │ 52ms     │
│ TC-003   │ Missing auth returns 401           │ FAIL   │ 48ms     │
│ TC-004   │ Empty filter returns empty array   │ PASS   │ 67ms     │
└──────────┴────────────────────────────────────┴────────┴──────────┘

───────────────────────────────────────────────────────────────────────
  Failed Tests
───────────────────────────────────────────────────────────────────────

TC-003: Missing auth returns 401
  Expected: status 401
  Actual:   status 200
  Response: {"error": false, "data": [...]}

───────────────────────────────────────────────────────────────────────
  SQS Assertions: {PASSED}/{TOTAL} passed
───────────────────────────────────────────────────────────────────────

┌────────────────────────────┬────────┬───────────────┐
│ Assertion                  │ Status │ Found In      │
├────────────────────────────┼────────┼───────────────┤
│ USER_CREATED event         │ PASS   │ 2.3s          │
│ ORDER_PLACED event         │ FAIL   │ Timeout (30s) │
└────────────────────────────┴────────┴───────────────┘

═══════════════════════════════════════════════════════════════════════
  Overall: {PASS|FAIL} ({PASSED}/{TOTAL} tests passed)
═══════════════════════════════════════════════════════════════════════

JSON Results File

Save detailed results:

{
  "test_plan": "proj-123-test.yaml",
  "name": "Test Plan Name",
  "executed_at": "2025-01-17T12:00:00Z",
  "base_url": "http://localhost:3000",
  "summary": {
    "total": 4,
    "passed": 3,
    "failed": 1,
    "skipped": 0,
    "duration_ms": 312
  },
  "test_cases": [
    {
      "id": "TC-001",
      "name": "Happy path",
      "status": "PASS",
      "duration_ms": 145,
      "request": {
        "method": "GET",
        "url": "/api/v1/users",
        "headers": {}
      },
      "response": {
        "status": 200,
        "body": {}
      },
      "assertions": []
    }
  ],
  "sqs_assertions": [
    {
      "name": "USER_CREATED event",
      "status": "PASS",
      "found_in_seconds": 2.3
    }
  ]
}

Step 7: Save Results

Before persisting results, redact authorization and cookie headers plus values whose keys contain token, password, secret, or apiKey. Save status, timing, validation differences, and sanitized excerpts by default. Capture full request or response bodies only after explicit approval for the current run, and redact them before writing.

RESULTS_DIR=$(config.qa.resultsDir || "tests/qa/results")
mkdir -p ${RESULTS_DIR}

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
RESULTS_FILE="${RESULTS_DIR}/${TEST_NAME}-${TIMESTAMP}.json"

echo "$RESULTS_JSON" > "$RESULTS_FILE"

Step 8: Confirm

Results saved: {RESULTS_FILE}

Summary:
- Test Cases: {PASSED}/{TOTAL} passed
- SQS Checks: {SQS_PASSED}/{SQS_TOTAL} passed
- Duration: {TOTAL_DURATION}ms

{If failed tests:}
Failed tests require attention:
- TC-003: Missing auth returns 401 (expected 401, got 200)

Configuration Reference

Setting	Default	Description
qa.testPlansDir	tests/qa	Test plan directory
qa.resultsDir	tests/qa/results	Results output directory
qa.timeout	10	Default timeout (seconds)
qa.apiBaseUrl	-	Default API base URL

Examples

Run Latest Test

/start-qa

Runs the most recently modified test file.

Run by Ticket ID

/start-qa PROJ-123

Runs tests/qa/proj-123-test.yaml.

Run Specific File

/start-qa tests/qa/user-api-test.yaml

Runs the specified test file.

Test File Format Reference

Expected YAML structure:

name: "Test Name"
base_url: "${API_BASE_URL}"
timeout: 10

test_cases:
  - id: TC-001
    name: "Test description"
    endpoint: /api/v1/resource
    method: GET
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    params:
      key: value
    body: {} # For POST/PUT
    expected:
      status: 200
      body:
        field: expected_value
    priority: high
    tags: [smoke]

sqs_assertions:
  - name: "Event name"
    queue_url: "${SQS_QUEUE_URL}"
    match_mode: json_field
    expected_fields:
      event_type: "EVENT_NAME"
    timeout_seconds: 30

Error Handling

Scenario	Action
Test file not found	Error with suggestions for file location
Invalid YAML	Show parse error and line number
Network timeout	Mark test as ERROR with timeout message
Invalid response	Mark test as ERROR, show raw response
SQS queue unavailable	Mark assertion as ERROR, continue tests
Missing env variable	Warn and use literal string

Advanced Features

Parallel Execution

For faster test runs:

execution:
  parallel: true
  max_concurrent: 5

Test Dependencies

Run tests in order with dependencies:

test_cases:
  - id: TC-001
    name: "Create user"
    # ... creates user
    outputs:
      user_id: "$.response.id"

  - id: TC-002
    name: "Get created user"
    depends_on: TC-001
    endpoint: /api/v1/users/${outputs.user_id}

Retry Logic

Retry flaky tests:

test_cases:
  - id: TC-001
    retry:
      max_attempts: 3
      delay_ms: 1000