Design Testing Strategy

A reference manual for designing a fit-for-purpose, fit-for-criticality testing strategy.

This skill is decision-oriented, not philosophical: every gate is deterministic (ON when X / OFF when Y), every schema is enforced (field ordering matters), every example is worked end-to-end.

How To Use This Skill

1. Read Decision Gates in order (Gate 0 -> Gate 6). Each gate is independent — you may finish with any subset of test types ON.
2. Apply Strategic Skip Heuristics to remove ON gates that would yield low ROI for this artifact.
3. For each ON gate, fill the Test Matrix Schema (selected_types entry) — the field order is load-bearing.
4. List rejected types in rejected_types and deliberate skips in deliberately_skipped.
5. Produce a Test Cases to Cover markdown bullet list using ISTQB techniques from Case Design Techniques.
6. Cross-check against the matching Worked Example (A pure function / B HTTP+DB endpoint / C UI component).

---

Decision Gates

Apply gates in numeric order. Each gate produces an independent boolean (applies: true|false). Gates do NOT veto each other — a single artifact may have unit + integration + contract + property-based all ON.

#	Type	ON when	OFF when	Source
0	Skip All	Criticality is NONE (docs-only, comments, formatting, generated code, config without logic, throwaway prototypes)	Anything with branching, computed output, side effects, or user-visible behavior	Pragmatic Programmer — "Test ruthlessly and effectively" implies effective skipping when ROI is zero
1	Unit	Code contains any logic: branches, loops, conditionals, computation, transformation, parsing, validation, formatting	Pure declarative wiring (DI registration, route table) with no behavior	Test Pyramid (Vocke) base layer + Beck TDD Red-Green-Refactor unit
2	Integration	Boundary crossing: HTTP call, DB query, external SDK, message queue, filesystem I/O, OR collaboration with >=2 distinct collaborators where unit doubles distort behavior	Pure function with no I/O and 0-1 stable collaborators	Testing Trophy (Dodds) — integration is the highest-ROI layer; Google "Follow the User"
3	Component or E2E	UI surface AND criticality >= MEDIUM-HIGH AND user-facing critical path (signup, checkout, auth, payment, primary CTA)	Internal admin-only screens, dev tooling, or non-critical UI	Test Pyramid top + ISO/IEC/IEEE 29119 risk ranking + Google e2e principles
4	Contract	Public API consumed by >=1 distinct clients (mobile + web, multiple internal services, external partners) AND independent deploy cadence	API where consumer and provider deploy together	Pact / CDC + Pactflow CDC explainer
5	Smoke	Deployable surface (web app, API, service) AND a deploy/CI pipeline exists where post-deploy validation is meaningful	Library, internal helper, or no deploy pipeline	Google "What Makes a Good End-to-End Test" — smoke = minimal e2e for deploy gate
6	Property-Based	Input domain is large or unbounded (numeric ranges, strings, lists, parsers, serializers, encoders, math) AND invariants are stable (round-trip, idempotency, monotonicity, commutativity) AND criticality >= MEDIUM-HIGH	Small finite input domain, unstable invariants, or LOW criticality	Hypothesis / QuickCheck

Gate Application Algorithm

for gate in [Gate 0, Gate 1, ..., Gate 6]:
    if gate.ON_condition_met(artifact):
        result[gate.type] = applies: true
    else:
        result[gate.type] = applies: false

if Gate 0 is true:
    short-circuit: emit empty selected_types, document criticality=NONE, stop

Criticality Scale (used by Gates 3 and 6):

Level	Definition
NONE	Docs, formatting, generated code, throwaway code, configs without logic
LOW	Internal dev tooling, admin-only screens, logging formatters
MEDIUM	Standard CRUD, internal APIs with a single team consumer, non-critical UI, helpers and utilities
MEDIUM-HIGH	User-facing UI on critical paths, public APIs with multiple consumers, business workflows
HIGH	Money movement, auth/authz decisions, security-critical validation, data integrity, regulated domains

---

Test Type Reference

Type	Use when	Do NOT use when	Frameworks	Typical dependencies	Google Size
unit	Pure logic, single function/method/class, deterministic inputs	Code is just I/O orchestration with no logic	vitest, jest, pytest, go test, JUnit, xUnit, RSpec	None (or in-memory fakes)	Small
integration	Boundary crossing (DB, HTTP, queue, FS); multiple collaborators where mocking distorts behavior	Pure function with no boundary	vitest, jest, pytest, go test, JUnit + Testcontainers, supertest, TestRestTemplate	Real Postgres/Redis/Kafka via Testcontainers, in-process HTTP server, real FS in tmpdir	Medium (single machine, localhost OK)
component	UI rendering + interaction within a single component, no full app context	Backend-only logic; multi-page user flow	React Testing Library, Vue Test Utils, Angular TestBed, Storybook interaction tests	jsdom or happy-dom, mocked network at fetch/axios level	Small to Medium
e2e	Full user path through running app: real browser, real backend, real DB	Internal helper, single component, non-critical UI	Playwright, Cypress, Selenium	Real running app + Testcontainers-backed DB or seeded staging	Large (multi-process, possibly multi-machine)
smoke	Post-deploy go/no-go: hit / health, key endpoints respond, login works	Detailed correctness; smoke is shallow by design	Playwright (1-3 critical paths), HTTP probe scripts, k6 minimal scenarios	Real deployed environment	Large
contract	Public API consumed by 2+ distinct clients with independent deploy cadence	Single-consumer internal API; provider and consumer deploy together	Pact, Spring Cloud Contract, OpenAPI schema validators	Pact broker or contract files in repo	Medium
property-based	Large/unbounded input domain with stable invariants (parser, serializer, encoder, math)	Small finite input space; unstable invariants	Hypothesis (Python), fast-check (TS), QuickCheck (Haskell), jqwik (Java), proptest (Rust)	Same as unit	Small

Google Test Size Mapping

Google Test Sizes (Bland) and SWE at Google Ch.11 classify tests by resources (size), independent of scope (paths covered):

Size	Process model	Network	Filesystem	Time budget	Notes
small	Single process, single thread	None	None (in-memory only)	< 100ms	Fast, hermetic, parallelizable
medium	Single machine, multiple processes allowed	localhost only	tmpdir allowed	< 1s	Testcontainers fits here
large	Multi-machine	External network allowed	Persistent FS allowed	< 15min	Full e2e
enormous	Distributed	Wide network	Anywhere	longer	Cluster / chaos

A test's type (unit/integration/e2e) and size (small/medium/large) are orthogonal: a small integration test (Testcontainers Postgres in same process via JDBC) is legitimate.

Playwright vs Cypress (UI e2e)

Dimension	Playwright	Cypress
Browsers	Chromium, Firefox, WebKit	Chromium, Firefox, WebKit (limited)
Multi-tab / multi-origin	Yes	Limited
Parallelism	Built-in shards	Paid dashboard or external
Network interception	Robust route-level	cy.intercept
Default	Choose Playwright for new projects unless team already standardized on Cypress	Choose Cypress when team has heavy investment

---

Case Design Techniques

Use ISTQB Foundation Level black-box techniques to derive what to test inside each chosen test type. References: ISTQB BVA white paper, ASTQB black-box techniques.

1. Equivalence Partitioning (EP)

Divide input domain into partitions where the system is expected to behave the same way; ONE test per partition is sufficient.

Worked example — discount(orderTotal: number) -> number:

Partition	Range	Representative test input	Expected
Below threshold	0 <= total < 100	50	0% discount
Mid tier	100 <= total < 500	250	5% discount
Top tier	total >= 500	1000	10% discount
Invalid (negative)	total < 0	-1	throw / error

Four tests cover all partitions. EP alone misses boundaries — combine with BVA.

2. Boundary Value Analysis (BVA)

Bugs cluster at boundaries. For every boundary value B, test B-1, B, B+1 (or for floats, the smallest representable step).

Worked example — same discount function, boundary at 100:

Test input	Why	Expected
99 (= B-1)	Last value of "below threshold" partition	0% discount
100 (= B)	First value of "mid tier" partition	5% discount
101 (= B+1)	Confirms not off-by-two	5% discount

Repeat for boundary at 500: test 499, 500, 501. Total: 6 boundary tests + 4 EP tests = 10 cases.

The B-1 / B / B+1 triplet has the same shape across boundaries (vary input, vary expected output, identical assertion); this is a natural fit for a table-driven test (see sub-section 5 below).

3. Decision Tables

When output depends on combinations of conditions. Each column is a rule.

Worked example — canCheckout(cartHasItems, paymentValid, addressOnFile):

Condition / Rule	R1	R2	R3	R4
cartHasItems	T	T	T	F
paymentValid	T	T	F	*
addressOnFile	T	F	*	*
Result	allow	block:address	block:payment	block:cart

Four tests, one per rule (* = don't care, dropped via merging).

4. State Transition

When behavior depends on history. Identify states, events, and forbidden transitions.

Worked example — Order state machine with states {draft, submitted, paid, shipped, cancelled}:

From	Event	To	Test
draft	submit	submitted	happy path
submitted	pay	paid	happy path
paid	ship	shipped	happy path
draft	cancel	cancelled	early cancel
paid	cancel	reject	forbidden — refund flow required, NOT direct cancel
shipped	submit	reject	forbidden

Cover one test per legal transition + one per forbidden transition (negative path).

5. Table-Driven Tests

When EP, BVA, or decision-table analysis yields 3+ cases with the same shape (same setup, same assertion, only inputs and expected outputs differ — e.g., parsing valid/invalid date formats; computing tax across brackets; routing rules) collapse them into a single table-driven test. The cases become rows in a data table; the test body iterates the rows and runs one assertion per row. References: Dave Cheney, Prefer table-driven tests; Go wiki: TableDrivenTests.

Do NOT force a table when setup, framework calls, or the assertion shape varies substantially across cases. Forced uniformity hides real differences behind a single name and produces obscure failure messages — keep those as separate, individually named tests.

Worked example — six EP+BVA cases for discount(orderTotal) (boundary at 100) collapsed into one table-driven unit test (TS / vitest syntax; the same pattern applies to Go t.Run, JUnit @ParameterizedTest, pytest parametrize):

describe("discount", () => {
  const cases: Array<{ name: string; input: number; expected: number }> = [
    { name: "EP: below threshold (typical)", input: 50,  expected: 0    },
    { name: "BVA: B-1 at boundary 100",      input: 99,  expected: 0    },
    { name: "BVA: B at boundary 100",        input: 100, expected: 0.05 },
    { name: "BVA: B+1 at boundary 100",      input: 101, expected: 0.05 },
    { name: "EP: mid tier (typical)",        input: 250, expected: 0.05 },
    { name: "EP: top tier (typical)",        input: 1000, expected: 0.10 },
  ];

  for (const c of cases) {
    it(c.name, () => {
      expect(discount(c.input)).toBe(c.expected);
    });
  }
});

The name column is mandatory: each row must produce an individually addressable test so failures point to the specific case, not "row 3 of 6". Rows that need a different assertion (e.g., the negative-input case throws) stay as separate tests outside the table.

---

Dependency Decision

For Gate 2 (Integration) and Gate 3 (Component/E2E), choose dependencies deliberately. The goal is maximum realism that still runs deterministically in CI.

Dependency style	Use when	Avoid when	Notes
Real infra via Testcontainers	DB/Redis/Kafka/Browser, dev needs real driver behavior, hermetic CI required	Cold-start budget < 1s, no Docker available	Default for integration tests on Postgres / Redis / Kafka / Localstack
In-memory fake	Owned interface, semantics are simple (key-value, list), test speed critical	Fake diverges from real — silent bugs at integration boundary	Acceptable for repository ports in hexagonal architectures, IF the port has its own contract test against real infra
Mock (test double)	Single collaborator with pure interface; test focuses on protocol (was X called with Y)	You're mocking >2 collaborators or mocking data structures (anti-pattern: incomplete mocks)	Mocks are tools to isolate, not things to test
Stubbed HTTP	Calling external SaaS where Testcontainers / Localstack option doesn't exist	When Pact / CDC is needed (use contract tests instead)	nock (Node), responses (Python), WireMock (JVM)
Real external service	Smoke test in staging only	Unit / integration / CI — always non-deterministic	Reserve for smoke tests against staging

Tradeoff summary: Testcontainers > in-memory fake > mock, but cost goes the same direction. Pick the cheapest level that doesn't lie about the boundary's behavior.

---

Strategic Skip Heuristics

Explicit "don't bother" rules. Skipping these is not laziness — it is risk-adjusted ROI per ISO/IEC/IEEE 29119 risk-based testing and Risk-Based Testing.

Skip	Rule
No e2e for internal helpers	If artifact has no UI surface and no user-facing path, skip e2e. Unit + integration is sufficient.
No contract test for bound by deploy consumer API	If only one client consumes the API and they deploy together, contract testing adds maintenance with no decoupling benefit.
No property-based on small finite domains	If input space is enum {A, B, C}, EP + BVA already covers it; property-based adds infra without finding more bugs.
No integration test for pure functions	Adding a Postgres container to test a formatCurrency helper is waste. Unit only.
No component test for static markup	If the component has no state, no events, no conditional rendering, a snapshot is enough — or skip entirely.
No unit test for declarative wiring	DI bindings, route registration, schema declarations: assert at integration level (does the route serve the right handler) instead.
No e2e for things integration covers reliably	Per Google e2e principles: the smaller the test you can use to cover a behavior, the better. e2e is the exception, not the default.
No tests for spike/throwaway code	Per Beck TDD: if the artifact will be deleted within hours, document the exception with the human partner. Then write tests on the kept version.
No "and" tests	If a test name contains "and", split it into separate tests (one assertion per behavior).

---

Test Matrix Schema

Every test strategy MUST be expressed as the YAML block below. Field ordering inside each list entry is load-bearing — judges and downstream tools parse the first key as the critical one (rationale / reason / why), and the second key as the categorical one (type / what).

Schema

test_strategy:
  artifact: "<path or short identifier>"
  rationale: "Why this test strategy is being applied to this artifact (specific, evidence-based)"
  criticality: "NONE | LOW | MEDIUM | MEDIUM-HIGH | HIGH"

  selected_types:
    - rationale: "Why this type is being applied to this artifact (specific, evidence-based)"
      type: "unit | integration | component | e2e | smoke | contract | property-based"
      size: "small | medium | large | enormous"
      framework: "vitest | jest | pytest | go test | JUnit | playwright | cypress | pact | hypothesis | ..."
      dependencies:
        - "List of dependencies: real Postgres via Testcontainers, in-memory fake, mocked HTTP via nock, etc."
      gate: "Gate N (the gate that triggered this selection)"

  rejected_types:
    - reason: "Why this type does NOT apply to this artifact (cite Strategic Skip Heuristic or gate that did not trigger)"
      type: "unit | integration | component | e2e | smoke | contract | property-based"

  deliberately_skipped:
    - why: "Cost / risk justification for skipping despite a partial signal"
      what: "A specific category of test cases being skipped (e.g., 'browser compatibility on IE11', 'load testing beyond 100 RPS')"

Worked YAML Example

test_strategy:
  artifact: "POST /users (user registration endpoint)"
  rationale: "User registration is a critical user-facing path; can be used by web and mobile apps independently of each other."
  criticality: "MEDIUM-HIGH"

  selected_types:
    - rationale: "Endpoint contains validation logic (email format, password rules, uniqueness) — Gate 1 ON for branch coverage"
      type: "unit"
      size: "small"
      framework: "vitest"
      dependencies: ["in-memory user repository fake"]
      gate: "Gate 1"
    - rationale: "Endpoint writes to Postgres and emits user.created event to Kafka — Gate 2 ON, real boundary behavior matters"
      type: "integration"
      size: "medium"
      framework: "vitest + supertest + Testcontainers"
      dependencies: ["Postgres 15 via Testcontainers", "Kafka via Testcontainers"]
      gate: "Gate 2"
    - rationale: "Consumed by mobile app and web app on independent deploy cadences — Gate 4 ON, prevents drift"
      type: "contract"
      size: "medium"
      framework: "Pact"
      dependencies: ["Pact broker"]
      gate: "Gate 4"

  rejected_types:
    - reason: "No UI surface in this artifact — Gate 3 OFF"
      type: "component"
    - reason: "No UI surface — Gate 3 OFF; e2e covered by web/mobile apps separately"
      type: "e2e"
    - reason: "Input domain (email, password) is large but invariants are well-covered by EP+BVA at unit level — property-based ROI is low at MEDIUM-HIGH criticality, only triggers Gate 6 partially"
      type: "property-based"

  deliberately_skipped:
    - why: "Project does not have post-deploy probe pipeline yet; smoke would be no-op"
      what: "Smoke test for /users after deploy"
    - why: "Non-functional load testing is out of scope for this task; tracked separately in performance backlog"
      what: "Load test verifying p99 < 200ms at 1000 RPS"

Field ordering checklist (judges check this verbatim):

• test_strategy: artifact BEFORE rationale BEFORE criticality.
• selected_types[*]: rationale BEFORE type BEFORE size BEFORE framework BEFORE dependencies BEFORE gate.
• rejected_types[*]: reason BEFORE type.
• deliberately_skipped[*]: why BEFORE what.

---

Case Listing Schema

After the matrix, produce a flat markdown bullet list of test cases to be implemented. This is separate from the YAML matrix because:

• a. it lists what to test, not how
• b. it links back to acceptance criteria

Format

## Test Cases to Cover

### AC-N: [criterion title]
- [type] description
- [type] description

### AC-N: [criterion title]
- [type] description
- [type] description

Where:

• type matches one of selected_types[*].type from the matrix
• description follows AAA / Given-When-Then (Dan North BDD) shape — see Bill Wake AAA (2001)
• AC-N references the acceptance criterion the case verifies (omit if non-AC-bound, e.g., infrastructure smoke)

Worked Example

## Test Cases to Cover

### AC-1: Discount returns the correct percentage based on the total
- [unit] discount returns 0% when total = 0 [EP partition: below threshold]
- [unit] discount returns 0% when total = 99 [BVA: B-1 at boundary 100]
- [unit] discount returns 5% when total = 100 [BVA: B at boundary 100]
- [unit] discount returns 5% when total = 101 [BVA: B+1 at boundary 100]

### AC-2: Discount fails when total is invalid
- [unit] discount throws when total = -1 [EP partition: invalid]

### AC-3: /orders saves the order to the database
- [integration] POST /orders persists order to Postgres and returns 201 with order id

### AC-4: /orders rejects duplicate idempotency key
- [integration] POST /orders rejects duplicate idempotency key with 409

### AC-5: /orders/:id returns order by id
- [contract] GET /orders/:id returns schema matching mobile-app pact

---

Sources & Further Reading

These 14 sources back every gate and rule above. When in doubt, consult the source linked at that gate.