Dev-DevOps — Production Infrastructure & Delivery

Backend handoff rule: When a deploy/SRE gate needs app behavior, dev-backend implements the hook (health handler, readiness dependency check, trace/span/log fields, migration compatibility, shutdown hook). dev-devops defines the operational gate, rollout/rollback behavior, alert policy, and release proof. This skill owns deployment strategy, rollback proof, observability operations, health/readiness operational gates, SLOs, incident response, and infrastructure/runtime delivery.

Build reliable, secure, and automated infrastructure and delivery pipelines. This skill is a routing role that activates by change-surface: whenever the work primarily touches containers, CI/CD, deployment, cloud/runtime infrastructure, Kubernetes, IaC, release engineering, or SRE operations, use this skill and then load the relevant references.

C0/C1 work (small local patches): See dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.

dev is canonical: dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.

Severity and rule authority are distinct (dev §0.2). Safety/correctness and release proof remain mandatory; architecture/tool preferences need project-specific justification.

Modular References

File	When to Read	What It Covers
references/docker.md	Container build/deploy	Multi-stage builds, distroless, Docker Scout/Trivy, BuildKit secrets, SBOM/Cosign
references/package-release.md	Package publishing / release auth	npm/PyPI trusted publishing, Bun-to-npm, registry auth model, downstream distribution table
references/cross-platform-release.md	Cross-platform release proof	CI matrix vs local OS proof, Windows App/RDP prompts, desktop verification boundaries
references/homebrew.md	Homebrew distribution	Formula vs Cask, audit/test, livecheck, artifact trust, install/uninstall proof
references/platform-engineering.md	Platform / DORA / provider routing	DORA capabilities, platform guardrails, provider table rows, SLSA handoff
references/kubernetes.md	K8s deployment	Gateway API (v1.6+), Kustomize overlays, HPA/VPA, Helm, ArgoCD GitOps
references/ci-cd-deploy.md	Deploy pipeline	GHA reusable workflows, deploy strategies, rollback, GitOps, progressive delivery
../dev/references/stacked-prs.md	Stacked/dependent PRs or unexpected CI runs	DEV-STACK-03/06/07: native membership preflight and CI diagnosis; also reached globally through dev
references/branch-lifecycle.md	Branch/worktree cleanup	Closed-PR branch automation, per-branch deletion evidence, worktree dirty audit, stacked-PR safety
references/iac.md	Infrastructure code	OpenTofu/Terraform modules, Pulumi, state encryption, blast radius isolation
references/sre-foundations.md	Operations/incidents	SLO/SLI/error budget, burn-rate alerting, incident response, blameless postmortem
references/edge-serverless.md	Edge/serverless work	Edge request shaping, auth at edge, Cloudflare Workers, Vercel Edge, edge AI triage
references/ml-infra.md	ML infrastructure	GPU cluster mgmt, model registry, scaling, edge inference, MLOps platform patterns

Read package-release.md for package publishing, registry auth, npm/PyPI trusted publishing, Bun-to-npm release decisions, and downstream package channels. Read cross-platform-release.md when a release claim depends on OS-local behavior that CI may not prove. Read homebrew.md for Formula/Cask distribution work. Read platform-engineering.md for broader DevOps capability refresh, DORA, provider routing, and platform guardrails. Read docker.md + ci-cd-deploy.md first for containerized deploy workflows. For K8s-specific work, add kubernetes.md. For SRE/on-call, add sre-foundations.md.

When release, registry-auth, provider-doc, service-status, image/platform version, or package-manager behavior depends on current external evidence, read the active search skill and follow its source-fetch and evidence-status rules instead of relying on stale memory or copied snippets.

---

§1 Container Builds

§1.1 Dockerfile Rules (STRICT)

Rule	Detail
Multi-stage	Separate build and runtime stages; final image has no build tools
Base image	Pin version + SHA256 digest: node:22-slim@sha256:abc...
Distroless	Prefer gcr.io/distroless/* for runtime; no shell, no package manager
Non-root	USER nonroot:nonroot (distroless) or create dedicated user
Dependency-first copy	COPY package.json bun.lock ./ → install → COPY . . for layer caching
BuildKit secrets	RUN --mount=type=secret,id=token ... — never use ARG for secrets
.dockerignore	.git, node_modules, .env*, *.log, dist/, coverage/, __pycache__/

For canonical Dockerfile templates, read references/docker.md §1.

§1.2 Image Security (STRICT)

CRITICAL/HIGH image findings block push under this image policy. General checklist exception language does not waive this gate: changing it needs a separately approved, predeclared security policy, never an exception invented in the failing release report. Read references/docker.md §4 for scan/SBOM/sign command examples, and ../dev-security/references/supply-chain-sbom.md for deeper SBOM/signing policy.

§1.3 Anti-Patterns

Banned	Symptom	Fix
FROM node:latest	Irreproducible builds	Pinned version + digest
USER root in final stage	Attack surface	Non-root user
COPY . . as first instruction	Cache invalidation on every change	Dependency files first
ARG SECRET=xxx	Exposed in image history	BuildKit --mount=type=secret
No scan before push	Vulnerable images in prod	Trivy/Scout CI gate
apt-get install without cleanup	Bloated image	--no-install-recommends && rm -rf /var/lib/apt/lists/*

---

§2 Deploy Pipeline

§2.1 Pipeline Stages (DEFAULT)

[dev-testing §5]  lint → typecheck → test → contract → e2e
[dev-devops]      build-image → scan → push-registry → deploy-staging → smoke → promote → deploy-prod

Stacked-PR CI preflight (DEFAULT). For stack sizing, repeated runs or missing checks, follow DEV-STACK-03/06/07 in cxc-dev's canonical stack reference. Stack recognition also applies without this DevOps router; the global dev entry owns it. Ordinary PRs/manual chains are the default. Do not suggest or adopt GitHub native stacks without the user's clear, strong request for that feature in this task (DEV-STACK-OPT-IN-01). Existing membership is a safety fact, not new authorization. Keep membership and CI decisions in the canonical owner.

§2.2 GHA Reusable Workflows (DEFAULT)

# .github/workflows/ci.yml (caller)
jobs:
  build:
    uses: org/templates/.github/workflows/build-test.yml@v2
    with:
      service: payments
    secrets: inherit

Rule	Detail
workflow_call	Central CI template, max 10-level nesting
Permissions	Caller cannot escalate; downgrade only
Environment	environment: production + required reviewers + prevent self-review
Promote	Digest-based (image@sha256:...), never mutable tags

§2.3 Deploy Strategies (DEFAULT)

Strategy	Tool	When	Risk
Rolling update	K8s Deployment	Stateless, low risk	Low
Blue-green	Argo Rollouts blueGreen:	Instant rollback, no DB migration	Medium
Canary	Argo Rollouts canary: steps:	Traffic % control, metric-based promote	Medium
Progressive	Flagger	Auto analysis + rollback, A/B testing	Medium-High
Feature flag	LaunchDarkly / Unleash	Code-level gradual rollout	Low

§2.4 Rollback Rules (STRICT)

• Every deployment must be rollback-capable within 5 minutes
• Digest-based promote only — mutable tags are banned
• DB migrations: forward-only + backward-compatible (expand-contract pattern)
• Post-rollback: automatic Slack/PagerDuty notification

§2.5 Secret Management (STRICT)

Rule (DEVOPS-AUTH-01): Prefer OIDC, workload identity federation, trusted publishing, or other short-lived credential flows before static long-lived tokens. When static tokens are unavoidable, scope narrowly, store in the managed secret system, and rotate on schedule or incident.

Source	Usage
GHA Secrets / Vault / AWS SM	CI pipeline secrets
External Secrets Operator	K8s → Vault/AWS SM sync
.env files	Never committed — generated in CI
Rotation	90-day cycle or immediate on incident

§2.6 GitOps (DEFAULT)

• Actions = CI, ArgoCD = CD — separation of concerns
• Actions updates deploy repo (image digest PR/commit) → ArgoCD reconciles
• Self-heal: ArgoCD auto-reverts drift
• Environment protection: GitHub Environments for prod approval gate

§2.7 Release Proof Contract (STRICT)

Rule (DEVOPS-RELEASE-PROOF-01): A release claim must name the artifact digest, workflow/builder identity, deploy target/environment, smoke-test evidence, and rollback evidence. Keep the proof at router level; detailed package, platform, and SLSA mechanics live in package-release.md, cross-platform-release.md, and platform-engineering.md.

---

§2.8 Freeze & GO/NO-GO Gates (STRICT)

DEVOPS-RELEASE-PROOF-01 governs the proof bundle for an artifact you already published. This section governs the decision to publish at all — the readiness report, and the gates it claims to have passed.

Sources: the OpenCodex v2.32.1 hotfix train and the operator-visibility train that followed it (devlog/_plan/260824_v2_32_1_hotfix_train/, devlog/_plan/260825_operator_visibility_train/). A freeze audit rejected the first GO report there on three counts — unresolved review threads on merged PRs, a red gate argued into an exception, and missing frozen-head receipts (900_go_nogo_readiness_report.md:37-49).

Rule	Severity	Statement
DEVOPS-FREEZE-SHA-01	STRICT	Pin the readiness report to the code SHA its gates describe. If the head moved after the freeze, prove the delta is docs-only (git diff --name-only <freeze> <head>) and keep every gate receipt on the freeze SHA.
DEVOPS-GATE-WEAKEN-01	STRICT	A red named gate is never excused inside the report that gate failed. Make the original command green, or replace it with a pre-declared equivalent CI actually runs — and declare the swap before the verdict, not after the failure.
DEVOPS-REVIEW-THREADS-01	STRICT	Unresolved review threads on merged PRs are a GO blocker. Count them after merge: a thread opened minutes before merge still counts until it is fixed or explicitly dismissed.
DEVOPS-GATE-OWNER-01	STRICT	A mandatory GO gate needs an implementing work-phase and a recorded terminal outcome — pass, not-reproduced, or explicitly deregistered. A gate nobody implements is not a gate; it is a wish.

Per-rule sources: FREEZE-SHA 900:3-7; GATE-WEAKEN 900:47-49; REVIEW-THREADS 900:40-46 and 080_wp8:47; GATE-OWNER 090_wp9:6-8 and 000_baseline_scope_and_roadmap.md:243-245.

Why gate-weakening is the load-bearing rule. In the case that produced it the suite was red, the red tests were known to be load-sensitive, and the fix was real — so the report explained the exception. The audit rejected that, correctly: an exception argued after a gate fails is indistinguishable from an exception argued because it failed. The honest move was to decompose the gate to match what CI actually runs (DEVOPS-SUITE-PARTITION-01), which turned the general suite green. One local api-usage failure remained and was waived separately, on the DEVOPS-BASELINE-DEFECT-01 triple — not by the partitioning.

Operational mechanics — suite partitioning, baseline-versus-defect attribution, instrument stability, and exact-head evidence — live in references/ci-cd-deploy.md §6. Runtime and operator-signal evidence rules live in references/sre-foundations.md §7.

---

§2.9 Branch Lifecycle Hygiene (STRICT)

This section guides explicitly requested branch-lifecycle work. A review or routine feature change does not authorize changing host settings, creating scheduled jobs, or deleting refs. Propose missing automation first; enact it only when authorized.

Delivery repositories accumulate dead refs, and the cost is not disk. Stale branches make git branch -r unusable for triage, keep superseded heads reachable by tooling that resolves names, and hide the handful of branches that actually still matter. Treat branch lifecycle as delivery infrastructure.

Rule	Severity	Statement
DEVOPS-BRANCH-AUTODELETE-01	STRICT	Enable host-side head deletion on merge (delete_branch_on_merge on GitHub) and close the gap it leaves. That setting fires only on merge; a pull request closed without merging keeps its head branch forever, so the closed-PR case needs its own scheduled automation.
DEVOPS-BRANCH-DELETE-EVIDENCE-01	STRICT	Never bulk-prune. Before deleting any ref, prove per branch that it is not protected, not an open PR head, not the base of an open PR, not a fork head, and not carrying unique commits. A name pattern is not evidence.
DEVOPS-BRANCH-SNAPSHOT-01	STRICT	Snapshot git for-each-ref (SHA + refname) for every local and remote ref to scratch space before the first deletion. Deleted remote branches are restorable with git push origin <sha>:refs/heads/<name> only while you still hold the SHA.
DEVOPS-WORKTREE-DIRTY-01	STRICT	Check every attached worktree for uncommitted work before removing it, and remove worktrees before their branches — an attached branch cannot be deleted, and --force on a dirty tree discards work no reflog will return.

Why the merged/closed distinction is load-bearing. delete_branch_on_merge reads as complete branch hygiene, and a repository with it enabled looks solved. It is not: OpenCodex had the setting on and still carried 59 dead remote branches, because closed-unmerged PRs are outside what that setting covers. The failure is silent and compounds — nothing reports it, and the branch list simply degrades until triage stops using it.

Why stacked PRs break naive cleanup. A stacked child PR targets its parent's head branch. Deleting a closed parent closes the open child, so "the PR that owned this branch is closed" is insufficient grounds for deletion. The base of any open PR is protected regardless of its own PR state.

Fork heads are out of scope, and identity is by repo id. A fork's head lives in the contributor's repository; deleting refs there is neither permitted nor intended. Compare repository ids, not names — a fork commonly carries the same branch names as upstream, so name comparison silently misclassifies it.

Mechanics, the deletion-plan algorithm, and a worked audit live in references/branch-lifecycle.md.

---

§3 Kubernetes Basics

§3.1 Minimum Viable K8s (DEFAULT)

Resource	Purpose
Deployment	Pod template + replica management
Service	Internal networking
HTTPRoute (Gateway API)	External traffic routing — not Ingress
ResourceQuota	Request/limit enforcement
Probes	Liveness + readiness + startup
Namespace	Environment isolation (dev/staging/prod)

§3.2 Gateway API (v1.6+, verified 2026-07-02 — TCPRoute/UDPRoute GA in v1.6)

Gateway API is the successor for new routing while Ingress remains GA but feature-frozen. Role separation: platform team owns GatewayClass + Gateway, app team owns HTTPRoute.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments-route
spec:
  parentRefs:
    - name: shared-gateway
  hostnames: ["payments.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      backendRefs:
        - name: payments-svc
          port: 8080

§3.3 Scaling (DEFAULT)

Mechanism	Scope	Watch
HPA	CPU/memory + custom metrics	Don't combine with VPA on same metric
VPA	Request auto-tuning	Use Off mode for recommendations only
PDB	Disruption budget	minAvailable: 50% or maxUnavailable: 1

§3.4 Anti-Patterns

Banned	Fix
Ingress (new projects)	Gateway API HTTPRoute
No resource limits	Always set requests + limits
image: app:latest	SHA digest or pinned SemVer
Single replica in prod	Minimum 2 + PDB
Secrets in ConfigMap	K8s Secret + External Secrets Operator
Annotation-based routing	Gateway API native fields

---

§4 Infrastructure as Code

§4.1 OpenTofu/Terraform Rules (DEFAULT)

Rule	Detail
State	Remote backend required (S3+DynamoDB / TF Cloud / OpenTofu)
Encryption	OpenTofu native state/plan encryption via KMS
Blast radius	Separate state per app/layer/env
Modules	Purpose-built (vpc, iam, ecs-service), typed I/O
Apply	plan → PR review → apply; auto-apply staging only
Versions	Pin provider + module versions explicitly

§4.2 Tool Selection (HEURISTIC)

Tool	Best For	2026 Status
OpenTofu (HCL)	Open-source/licensing-neutral IaC default (MPL-2.0, Linux Foundation)	✅ Recommended for OSS neutrality
Terraform / HCP Terraform (BSL)	Vendor support or HashiCorp platform integration	✅ Active (check BSL competitive-use terms)
Pulumi (TS/Python)	Teams preferring programming languages	✅ Active
AWS CDK	AWS-only infrastructure	✅ Active (AWS only)
CDKTF	—	❌ Deprecated 2025-12-10; repo archived/read-only, no further fixes

§4.3 Anti-Patterns

Banned	Fix
Local state file	Remote backend required
Manual console changes	All changes via code
Hardcoded values	Variables + tfvars
Monolithic main.tf	Modular decomposition
CDKTF (new projects)	OpenTofu or Pulumi
Unpinned provider versions	Explicit version constraints

---

§5 SRE Foundations

§5.1 SLO/SLI (DEFAULT)

SLI	Measurement	Typical SLO
Availability	Success requests / total	99.9% (28-day rolling)
Latency	p50/p95/p99 response time	p99 < 500ms
Error rate	5xx / total	< 0.1%
Freshness	Data update delay	< 5min (pipelines)

SLIs measure user experience, not infrastructure metrics. "CPU is fine ≠ users are fine."

Error budget = 1 − SLO (99.9% → 0.1% budget).

DORA 2025 (verified 2026-07-02): AI acts as an amplifier — returns depend on the underlying sociotechnical system. For AI-agent-heavy delivery invest in golden paths, guardrails, observability, provenance, and review gates.

§5.2 Error Budget Policy (DEFAULT)

Budget State	Action
Normal (>50%)	Continue releases, routine monitoring
Accelerated burn (20–50%)	Heightened alerts, slow releases, reliability triage
Exhausted (≤0%)	Feature freeze — security/bugfix only; VP exception required

Single incident consuming >20% → mandatory postmortem. Two consecutive window misses → architecture review.

§5.3 Incident Response (DEFAULT)

1. Detect → Triage (S1/S2/S3) → Stabilize → Fix → Postmortem
2. Roles: IC, Primary Responder, Comms Lead, Scribe
3. Mitigation first, diagnosis second during active incidents
4. All S1/S2 → mandatory blameless postmortem within 5 business days
5. Status updates: S1 every 15min, S2 every 30min

§5.4 Runbook Template (HEURISTIC)

## [Service] — [Symptom]
### Diagnosis
1. Logs: `kubectl logs -l app=<name> --tail=100`
2. Metrics: Grafana → [dashboard URL]
3. Dependencies: `curl -s http://<dep>/health | jq .`
### Emergency Mitigation
1. Rollback: `argocd app rollback <app>`
2. Traffic block: ...
### Root Fix
1. ...
### Escalation
- Owner: @team-sre
- PagerDuty: [policy]

§5.5 Anti-Patterns

Banned	Fix
Infrastructure-only SLIs	User-experience-based SLIs
SLO without consequences	Error budget policy with freeze gate
Too many SLIs (>5 per service)	2-4 meaningful SLIs
Page on every deviation	Burn-rate multi-window alerting
Blame individuals	Blameless postmortem, system improvement
No error budget policy	Define 3-stage policy (normal/accelerated/exhausted)

---

§6 Cross-References

Topic	Canonical Owner	What dev-devops defers
Test strategy & CI test stages	dev-testing §5	Test pyramid, coverage gates
Backend observability code patterns	dev-backend observability.md	OTel SDK setup, structured logging
Security hardening (app-layer)	dev-security	OWASP, auth, input validation
SBOM/signing depth	dev-security references/supply-chain-sbom.md	Supply-chain evidence policy beyond image scan gates
Architecture module boundaries	dev-architecture	Coupling taxonomy, barrel discipline
Scaffolding conventions	dev-scaffolding	File naming, project structure
Frontend build/bundle	dev-frontend	Vite/webpack config, SSR

dev-devops owns: container builds, deploy pipelines, K8s manifests, IaC modules, SRE/incident response, edge infra, ML infra. DevOps owns operational scan execution and release gates; dev-security owns security policy, severity thresholds, and required evidence. dev-backend owns: application-layer observability code, API design, health check implementation. Overlap: observability alerting rules (dev-devops §5) ↔ observability code instrumentation (dev-backend observability.md). Cross-ref both.

Pre-flight Checklist

Before submitting infrastructure changes:

• Dockerfile is multi-stage with distroless/slim final image and non-root user
• Image scanned (Trivy/Scout) with CRITICAL/HIGH gate — no unresolved findings
• SBOM generated and attestation signed (Cosign) for production images
• Deploy pipeline uses digest-based promotion, never mutable tags
• K8s manifests have resource requests/limits, probes, PDB, and use Gateway API (not Ingress)
• IaC uses remote state, pinned versions, and modular decomposition
• Secrets are managed through Vault/AWS SM/GHA Secrets — no .env commits, no ARG secrets
• SLO/SLI defined with error budget policy and burn-rate alerting
• Rollback plan documented and tested — <5min rollback capability confirmed
• Runbook exists for critical failure scenarios