QA Methodology

Senior-to-principal QA and SDET methodology: test strategy, automation, regression, risk-based prioritization, exploratory testing, quality gates, AI code quality gates for agentic Spec-Driven Development, agentic eval design, career leveling, and SDET engineering.

Ownership

You own	You don't own
Test strategy — what to test, at what level, with what priority	Root-cause debugging — route to systematic-debugging
Test automation — framework selection, parallelism, flaky management	Security implementation and threat modeling — route to secure-software-engineering
E2E automation strategy and coverage decisions	Operating a browser test tool (Playwright) — authoring/running specs, selectors, network mocking, scraping — route to playwright
Regression suites — selection, impact analysis, suite evolution	Spec pipeline mechanics and gate verdicts — route to spec-driven-development
Quality gates — blocking vs advisory, metrics, DORA	Eval framework governance and statistics — route to agent-evals-and-observability
Risk-based testing — P×I scoring, prioritization, registers	Verification verdicts against explicit criteria — route to verification-methodology
Exploratory testing — SBTM charters, heuristics, tours	Feature implementation — that's the developer
AI code quality gates — independent verification, AC testability	Production monitoring and incident response — that's SRE
Mutation-guided test hardening — bounded mutation review evidence and survivor triage	Verification verdicts against explicit criteria — route to verification-methodology
Agentic eval design — dataset design, judge bias, flaky-eval discipline
QA career levels — Senior/Staff/Principal scope progression
SDET engineering — test infrastructure, gTAA, CI/CD integration

Core Principles

If it isn't tested, it's broken. Untested code is code whose failure mode hasn't been discovered yet.

Quality is a property of the process, not the artifact. Testing at the end doesn't create quality. Quality is designed in through strategy, automation, and gating throughout the cycle.

Test behavior, not implementation. Tests coupled to behavior survive refactoring; tests coupled to implementation break on it.

Risk drives priority. Not everything deserves equal test investment. Score probability × impact, then allocate accordingly.

Flaky tests are worse than no tests. A nondeterministic failure trains teams to ignore all failures. Quarantine on detection; rerun once, never twice.

Independent verification is non-negotiable. The implementing agent (or developer) must not self-verify. Separate session, fresh context, no shared priors.

Loading Guide

File	Load when
references/test-strategy.md	Designing a test strategy — pyramid shape, shift-left/right, cost-of-failure, coverage as diagnostic
references/test-automation.md	Selecting frameworks, parallelism/sharding, flaky quarantine, predictive ML test selection, mutation-guided hardening
references/quality-gates-and-metrics.md	Designing quality gates (blocking vs advisory), DORA metrics, vanity-vs-actionable metrics, mutation testing
references/regression-testing.md	Building regression suites — impact analysis, selection math, suite evolution, shift-right feedback
references/test-data-management.md	Test data strategy — fixtures, factories, time-travel, masking, GDPR/PII rules
references/performance-testing.md	Load/stress/soak testing — k6/Locust/Gatling/JMeter, SLO thresholds, CI cadence
references/security-testing.md	Security testing — OWASP Top 10:2025, STRIDE, SAST/DAST/SCA, supply chain/SBOM
references/ci-failure-triage.md	CI is red — exit-code taxonomy (1/2/126/127/137/139/143), git bisect, flake-vs-failure protocol
references/test-debugging.md	A test that should pass is failing — CI-vs-local divergence, ordering/shared state, mock binding
references/risk-based-testing.md	Prioritizing by risk — P×I formula, 5×5 matrix, risk workshop, register, reassessment triggers
references/exploratory-testing.md	Exploratory testing — SBTM, charter writing, SFDIPOT/HICCUPPS heuristics, tours
references/test-design-techniques.md	Choosing test design techniques — EP, BVA, decision tables, state transition, pairwise, error guessing
references/qa-career-levels.md	QA career growth — Senior/Staff/Principal scope, leveling mechanics, archetypes, misconceptions
references/sdet-engineering.md	SDET role and skills — gTAA/TAF architecture, POM, SOLID for tests, build-vs-buy, testability
references/ai-code-quality-gates.md	Reviewing AI-generated code — independent verification, AC testability, agent-test quality, human-in-the-loop
references/agentic-eval-design.md	Designing agent evals — dataset test design, judge bias, flaky-eval discipline, CI gate tiers, replay
templates/test-strategy.md	Producing a test strategy document — fill in scope, risk tiers, level allocation, automation targets
templates/risk-register.md	Recording risk assessment results — fill in items, P×I scores, owners, mitigations
templates/exploratory-charter.md	Writing an SBTM charter — fill in target, resources, discovery goal, timebox
templates/bug-report.md	Filing a structured bug report — fill in reproduction steps, expected vs actual, severity
templates/verification-plan.md	Planning independent verification — fill in AC-to-method traceability, verifier assignment, exit criteria
templates/mutation-review.md	Recording bounded mutation review scope, classifications, survivor tests, and independent evidence
assets/risk-matrix-grid.md	Scoring risks during a workshop — 5×5 P×I grid with zone thresholds
assets/test-design-techniques-checklist.md	Selecting techniques for a feature — quick-reference checklist mapping scenario type to technique
assets/qa-definition-of-done.md	Defining release readiness — QA contribution to definition of done
scripts/risk-prioritize.py	Computing P×I rankings from a risk-items JSON file
scripts/check-ac-testability.py	Checking acceptance criteria for vague verbs and missing observable outcomes
evals/evals.json	Running output-quality evals for this skill (schema v1, 10 cases)

Scripts

Script	Invocation	Purpose
risk-prioritize	python3 scripts/risk-prioritize.py --json <input.json>	Reads risk items (probability, impact), computes P×I scores, emits ranked JSON
check-ac-testability	python3 scripts/check-ac-testability.py <spec.md>	Scans acceptance criteria for untestable language, exits non-zero if any are flagged

Triggers

Load this skill when the task involves:

• Test strategy — designing what/how/priority to test for a project or feature
• Regression testing — building, selecting, or evolving regression suites
• CI triage — diagnosing CI failures, exit codes, flake-vs-real classification
• Test automation — framework selection, parallelism, flaky quarantine, ML selection
• Quality gates — gate design, blocking vs advisory, metrics, DORA
• Mutation-guided test hardening — diff-aware mutation scope, surviving mutants, weak assertions, and review evidence
• Risk-based testing — P×I scoring, risk registers, prioritization workshops
• Exploratory testing — SBTM charters, oracle heuristics, session debriefs
• Agentic evals — eval dataset design, judge bias, flaky-eval discipline, CI tiers
• SDD gate review — QA ownership at spec-driven gates, AC testability, independent verification
• SDET — test infrastructure engineering, gTAA, CI/CD integration, career scope

When not to use

Route to the named sibling skill instead:

• spec-driven-development — writing specs, running the SDD pipeline, gate verdict format, revision loops
• agent-evals-and-observability — eval framework governance, statistical comparisons, telemetry and privacy controls, grader implementation
• verification-methodology — collecting evidence and rendering verdicts against explicit pass/fail criteria
• release-engineering — composing test evidence into release-candidate readiness, promotion, go/no-go, production rollout, and rollback decisions; QA owns test strategy and gate semantics
• systematic-debugging — root-cause analysis of production incidents, bug reproduction, fault localization
• secure-software-engineering — security implementation, threat modeling, secure defaults, dependency evaluation
• playwright — operating the Playwright tool itself: authoring and running E2E specs, selector robustness, network mocking, headless scraping, and headed debugging

Stop and Exit Conditions

• Test strategy complete when: strategy document names risk tiers, level allocation, automation targets, and exit criteria for each tier.
• Risk assessment complete when: every identified risk has a P×I score, an owner, and a mitigation or acceptance decision recorded in the register.
• CI triage complete when: failure is classified (flake vs real, env vs code), root cause is localized, and a fix or escalation path is identified.
• Gate review complete when: every acceptance criterion maps to a verification method, the verifier is independent of the implementer, and evidence is attached.
• Bounded escalation: stop after three non-converging diagnostic passes and report the evidence collected so far.