write-e2e

Produce a mergeable E2E test for a change: correct scope, right environment, assertions that survive real infrastructure, and CI wiring that runs it.

Ownership: the team owning test/new-e2e/tests/<area>/ is paged when the test fails, not @DataDog/agent-devx, which watches main and release branches and dispatches flakes back to that team. That is why the JOBOWNERS entry in Step 8 and the reliability gates in Step 9 are not optional.

References

The body carries the procedure; the references carry the lookup tables. Load a file when its trigger fires.

File	Load when
references/environments.md	The target is anything other than a Linux host, you need a non-default OS, architecture, or cloud, or you want a provisioner variant without the agent or without fakeintake.
references/templates.md	You need a skeleton other than the Step 5 one, you are splitting a suite across operating systems, or you need to mark a known flake or collect diagnostics on failure.
references/fakeintake.md	The test asserts on a payload — before writing the first such assertion, or when the payload you need has no client method yet.
references/ci-wiring.md	You are adding a new test package or changing which build artifacts a test consumes.
references/api-traps.md	You are working from an internal Confluence E2E page, an older branch, or any snippet whose form you cannot find in use under test/new-e2e/tests/.

The repo is the source of truth; this skill is the procedure. Resolve an option by grepping its definition (grep -n '^func With' <provisioner-dir>/*.go) rather than recalling it.

Step 0 — Establish scope

Given an argument, treat it as the scope and skip to Step 1. Otherwise derive it:

git status --short
git diff --name-only $(git merge-base HEAD origin/main)

Classify each changed path:

Class	Examples	Action
User-visible behavior	a metric or log payload, a config key, CLI or service behavior, packaging, permissions	Candidate for coverage
Framework	test/e2e-framework/**	No test; validate by running an existing suite
Test-only, docs, CI	test/new-e2e/tests/**, docs/**, .gitlab/**	Out of scope

Resolve the owning team and its <area> directory from .github/CODEOWNERS, then report the scope table before writing anything. When several behaviors are candidates and covering all of them would be a large change, ask which to cover. When nothing in the diff is user-visible, say so and stop — that is a correct answer, not a failure.

Step 1 — Gate the scope

Invoke e2e-audit with the behavior. Continue to Step 2 only if the verdict is that E2E is justified; on any other verdict, report it, name the package where the test belongs instead, and stop. Gating before Step 2 is deliberate — reading implementations for the literals a test would assert on is wasted if the behavior belongs in a unit test.

Some areas own their own template and lifecycle rules through a directory-scoped skill: ls test/new-e2e/tests/<area>/.claude/skills/*/SKILL.md. If one exists, invoke it by its name and stop; directory-scoped skills are only offered when the working directory matches, so read the file directly when it is not in the skill list. Sibling skills named elsewhere in this file live under .agents/skills/<name>/SKILL.md.

Step 2 — Extend an existing suite before creating one

Provisioning dominates E2E cost, and it is paid per suite. A Kubernetes suite blocks on cluster and agent readiness before its first assertion — tests/containers allows up to ten minutes for it, twenty when FIPS or cri-o makes the images slower to pull — and a new suite beside it pays that again for one assertion. A new test method inside an existing suite is close to free.

Read the implementation now for the exact observable it produces, because the assertion depends on the literal string rather than on the feature name: a metric name, log service, check name, or tag if the behavior ships a payload; a service name, file path, registry key, package name, or CLI output if it does not. Then look for existing coverage and a suite to join:

ls test/new-e2e/tests/<area>/
grep -rn "<metric-or-config-name>" test/new-e2e/tests/<area>/
grep -rln "e2e.BaseSuite\[environments.<Env>\]" test/new-e2e/tests/<area>/

Widen the metric grep to all of test/new-e2e/tests/ only when the area turns up nothing.

Situation	Do
A suite in the area already provisions the environment you need	Add a test method to it
Same environment, but the suite needs a different agent config	Add a sibling entry point reusing the shared suite body
No suite in the area, or a genuinely different environment	Create one

Read any AGENTS.md inside the target directory first (ls test/new-e2e/tests/<area>/AGENTS.md) — several areas carry local rules that override the defaults below.

Step 3 — Choose the environment and cloud

Signal in the behavior	Environment	Provisioner
Host metrics, agent CLI, files, services, packaging	environments.Host	awshost.Provisioner()
Windows host behavior	environments.Host with a Windows OS descriptor	awshost.Provisioner()
Active Directory, Defender, FIPS mode, test signing, MSI install	environments.WindowsHost	winawshost.Provisioner()
Agent in a container, Docker integrations	environments.DockerHost	awsdocker.Provisioner()
DaemonSet, Cluster Agent, admission, k8s tagging	environments.Kubernetes	kind (.../aws/kubernetes/kindvm)
ECS or Fargate task behavior	environments.ECS	.../aws/ecs
Two hosts, or a host plus a separate workload	custom struct	e2e.WithPulumiProvisioner[Env]

Ordinary Windows behavior does not need WindowsHost. Suites like tests/agent-subcommands/config_win_test.go run on plain environments.Host with ec2.WithOS(e2eos.WindowsServerDefault), which is also what the cross-OS split in references/templates.md produces — so a Windows test can extend an existing suite instead of provisioning a new one. Move up to WindowsHost only for the scenario components it carries.

Cloud: use the cloud the feature is specific to, and AWS otherwise. Windows included — the Windows CI templates depend on AWS-side MSI artifacts. Azure has a working Windows provisioner and reportedly boots Windows faster, but no test uses it and no CI job wires it, so choosing it means solving both problems yourself.

Kubernetes flavor: use kind. It provisions faster, costs less, and fails less than EKS. Reserve EKS for behavior that is specific to EKS, and expect an EKS job to run on main and nightly rather than on pull requests.

Every stock provisioner ships a fakeintake by default, and on AWS it is an ECS Fargate task you would otherwise pay for and never query. When the test asserts only on host state, drop it. Host provisioners take a ProvisionerNoFakeIntake constructor; every other provisioner spells it differently and a few cannot do it at all, so use the table in references/environments.md rather than guessing.

Reach for a custom provisioner last; test/new-e2e/codereview_guideline.md says to avoid them. Check the option surface of the stock provisioner first — most needs are already an option.

Step 4 — Place the files

Tests live in test/new-e2e/tests/<area>/. In an existing area, copy the package clause from a file already there — the usual form is the directory with its separators removed (agent-runtimes holds package agentruntimes), but not every area follows it. Read test/new-e2e/AGENTS.md §§ "Layout", "Each entry point needs its own suite type", and "Build tags" for the rest: which <area>, how a Linux/Windows pair splits across files, and why every entry point needs its own suite type — getting that last one wrong silently puts two parallel suites on one stack. references/templates.md § "Splitting one suite across operating systems" has the code for the split.

Fixtures: a YAML snippet of ten lines or fewer goes inline as a const; anything longer, and any script, goes in fixtures/ loaded with //go:embed (which needs _ "embed" in the import block).

Step 5 — Write the suite

// Unless explicitly stated otherwise all files in this repository are licensed
// under the Apache License Version 2.0.
// This product includes software developed at Datadog (https://www.datadoghq.com/).
// Copyright 2016-present Datadog, Inc.

// Package myarea contains e2e tests for <feature>.
package myarea

import (
	_ "embed"
	"testing"
	"time"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"

	"github.com/DataDog/datadog-agent/test/e2e-framework/components/datadog/agentparams"
	e2eos "github.com/DataDog/datadog-agent/test/e2e-framework/components/os"
	"github.com/DataDog/datadog-agent/test/e2e-framework/scenarios/aws/ec2"
	"github.com/DataDog/datadog-agent/test/e2e-framework/testing/e2e"
	"github.com/DataDog/datadog-agent/test/e2e-framework/testing/environments"
	awshost "github.com/DataDog/datadog-agent/test/e2e-framework/testing/provisioners/aws/host"
)

//go:embed fixtures/myfeature.yaml
var myFeatureConfig string

type myFeatureSuite struct {
	e2e.BaseSuite[environments.Host]
}

func TestMyFeature(t *testing.T) {
	t.Parallel()
	e2e.Run(t, &myFeatureSuite{},
		e2e.WithProvisioner(awshost.Provisioner(
			// AWS agent options nest inside WithRunOptions. The flat
			// awshost.WithAgentOptions(...) reads naturally and is what
			// Azure and GCP use, but on AWS it does not exist.
			awshost.WithRunOptions(
				ec2.WithEC2InstanceOptions(ec2.WithOS(e2eos.Ubuntu2204)),
				ec2.WithAgentOptions(
					agentparams.WithIntegration("myfeature.d", myFeatureConfig),
				),
			),
		)),
	)
}

func (s *myFeatureSuite) TestMetricReachesIntake() {
	s.EventuallyWithT(func(c *assert.CollectT) {
		metrics, err := s.Env().FakeIntake.Client().FilterMetrics("myfeature.points")
		require.NoError(c, err)
		assert.NotEmpty(c, metrics, "no myfeature.points received yet")
	}, 2*time.Minute, 10*time.Second)
}

The common agentparams options are WithAgentConfig, WithIntegration, WithLogs, WithFile, WithTags, and WithHostname; references/environments.md has the rest and the container equivalents.

Invariants, each with the reason it exists:

• Embed e2e.BaseSuite[Env] and start through e2e.Run — that is what builds s.Env() and registers teardown.
• Call t.Parallel() first in the entry point so suites overlap instead of serializing their provisioning.
• Keep sibling test methods independent. Testify does not guarantee ordering, so a method relying on a sibling's side effects fails unpredictably; put shared setup in the suite, and use s.T().Run(...) subtests when you genuinely need sequence.
• Leave the environment as you found it. Retries reuse the same infrastructure when they can, and fall back to a full reprovision when a leftover artifact breaks the second attempt.
• When overriding SetupSuite, BeforeTest, or AfterTest, call the embedded method first. Do not add defer s.CleanupOnSetupFailure() — BaseSuite already registers a t.Cleanup hook for it, and calling it from a defer consumes the panic so testify reports the recovered value with no stack, costing you the file and line of the failure. Older suites still carry the defer; leave them alone rather than copying them.
• Log what a future debugger needs on failure only. The infrastructure is destroyed after the run, and E2E_DEV_MODE=true is a local-only escape hatch. Write large dumps to s.SessionOutputDir() as an artifact rather than into the job log.

Step 6 — Assert

Decide first what the behavior is observable as. Most suites in tree assert on host state — services, packaging, permissions, CLI output, resolved configuration — reached through Env().RemoteHost. Payload assertions through fakeintake are the minority; reach for them when the behavior is the payload.

Either way, anything that takes time to settle is asserted inside s.EventuallyWithT(func(c *assert.CollectT) {...}, timeout, interval) as in the Step 5 skeleton, and three rules apply to every such poll: require on anything later code dereferences, so the iteration aborts instead of accumulating misleading errors from a nil result; MustExecuteOn(c, ...) rather than MustExecute, so a transient SSH error retries; and never time.Sleep or a ticker, which is either a flake or wasted minutes.

Asserting on payloads — load references/fakeintake.md first. It has the payload-to-client-method table, the matcher catalog, the timeout budgets, how to prove a negative, and flush ordering around a restart.

Not asserting on payloads — drop the intake. On AWS it is an ECS Fargate task, so provisioning one the test never queries is pure cost against the budget Step 2 is trying to protect. How to drop it depends on the provisioner, as Step 3 notes.

Step 7 — Keep it reliable

Read test/new-e2e/codereview_guideline.md now — it is the authoritative reliability contract, and three of its sections shape code you are about to type:

• § "Docker image pulls" — every image comes through the ECR cache. CI is moving to block outbound internet, so an unqualified image: or FROM resolves to DockerHub and will start failing.
• § "Pin your dependencies" — an unpinned image is latest, and apt install <pkg> silently tracks whatever the mirror carries today.
• § "Other dependencies / Internet accesses" — third-party artifacts come from the internal S3 bucket via RemoteHost.HostArtifactClient, and remote Kubernetes manifests are vendored locally with their image references rewritten, because a remote manifest pulls both itself and its images at runtime.

Step 8 — Wire into CI

Needed when you created a new package, and whenever a test starts consuming a build artifact its job does not already pull — adding a container-based test to a package whose job extends a deb-only template leaves it waiting on an image nobody built. A test in a package no job references never runs at all. Load references/ci-wiring.md for the rule template, the artifact-dependency table, the Windows matrix convention, and which branches a job ends up running on. Two things bite regardless of which template you extend: overriding needs replaces the template's list rather than adding to it, so re-reference the template's own entry; and .gitlab/JOBOWNERS is what routes a failure notification, while .github/CODEOWNERS only routes review.

Step 9 — Verify

Needs dda, pulumi, ~/.test_infra_config.yaml (created by dda inv e2e.setup), sandbox AWS credentials, and AppGate connected; run-e2e covers setup failures. Run the gates in order and stop at the first failure.

Gate	Command	Catches
G1	dda inv linter.go --module=test/new-e2e --targets=./tests/<area>	Compile errors, including nonexistent provisioner options
G2	grep -rn -e 'docker\.io' -e 'apt-get install' -e 'apt install' -e 'yum install' -e 'curl http' test/new-e2e/tests/<area>/	External dependencies and unpinned installs
G3	grep -rn 'TARGETS:.*<area>' .gitlab/test/e2e/*.yml .gitlab/windows/test/e2e/*.yml .gitlab/windows/test/e2e_install_packages/*.yml and the JOBOWNERS entry	A test that never runs, or fails silently to no owner
G4	The dev-mode session below	Provisioning, timing, real Agent behavior, hidden inter-test dependencies, missing cleanup
G5	Compare G4's wall time to 15 min (PR-gated) and 30–40 min (main/nightly)	A suite that will slow every pipeline

G2 flags candidates for review, not violations. A parameterised registry such as ${DD_REGISTRY:-docker.io} resolves to the cache in CI and is fine; a hardcoded docker.io/... or a package install on the VM is not. G3 is a grep, so it comes before G4 — there is no point proving a test works before knowing whether anything runs it.

G4 is the only gate that provisions real cloud infrastructure, and it takes 10–40 minutes. Dev mode keeps the stack alive between runs, so all three live checks share one provision:

# 1. One subtest on fresh infrastructure — proves it does not depend on a sibling.
E2E_DEV_MODE=true dda inv new-e2e-tests.run --targets=./tests/<area>/... --run '^TestX$/^SubB$'
# 2. The whole suite, reusing that stack.
E2E_DEV_MODE=true dda inv new-e2e-tests.run --targets=./tests/<area>/...
# 3. Again, without dev mode — a second green run proves each test left the
#    environment as it found it, and the suite destroys its own stack on the way out.
dda inv new-e2e-tests.run --targets=./tests/<area>/...

Dropping E2E_DEV_MODE on the last run is what releases the infrastructure, and it releases only this suite's stack. Reach for dda inv new-e2e-tests.clean -s only as deliberate housekeeping: it enumerates every local e2elocal stack and destroys all of them, including ones belonging to work you are not touching.

Show these commands and get agreement before running any of them. Compilation alone is not evidence the test works, so treat skipping G4 as a gap to report rather than a shortcut.

Output

Scope        behaviors covered, behaviors dropped and why, e2e-audit verdict
Files        created and modified
Environment  env struct, cloud, OS, and one line on why that combination
Assertions   observable -> how it is checked -> timeout, one row each
Local run    command, result, wall time
CI           the rule/job/JOBOWNERS diffs, or "none needed" and why
Follow-ups   flake exposure, budget headroom, QA label, PR-branch coverage

Filled in, for a diff that added a disk.free metric:

Scope        Covers disk.free reaching the intake. Dropped the config-parsing
             change — e2e-audit called it a unit test. Verdict: E2E justified.
Files        test/new-e2e/tests/agent-runtimes/disk_free_test.go (new)
Environment  environments.Host, AWS, Ubuntu2204 — host metric, no container
             or cluster behavior involved.
Assertions   disk.free -> FilterMetrics + WithMetricValueHigherThan(0) -> 2m/10s
Local run    dda inv new-e2e-tests.run --targets=./tests/agent-runtimes/...
             --run '^TestDiskFree$'  — passed, 11m4s
CI           None needed. new-e2e-agent-runtimes already covers ./tests/
             agent-runtimes and pulls agent_deb-x64-a7.
Follow-ups   Test-only change, so qa/no-code-change. Runs on PR branches
             via .on_arun_or_e2e_changes. 11m of the 15m budget used.

Recommend a QA label, and note that the checks accept exactly one. A pull request that only adds tests or CI wiring takes qa/no-code-change, whichever branches its job runs on. qa/rc-required is for changes that can only be validated on a release candidate — a workload that cannot be emulated, or behavior observable only during RC deployment — so reach for it based on what the change needs, not on whether the new job happens to skip pull-request branches. docs/public/guidelines/contributing.md has the full list.

When a repository document turns out to be wrong, correct it in the same change — the root AGENTS.md asks for exactly that.