Ansible

Playbook Structure

• Entry point: pb_all.yml (NOT site.yml)
• Decommission: pb_decommission.yml
• Vars: group_vars/all.yml — single source of truth for cluster-wide settings
• Inventory: hosts.yml
• ArgoCD apps: kubernetes-services/templates/; extra manifests in kubernetes-services/additions/. Reusable ingress sub-chart at kubernetes-services/additions/ingress/.
• SSH to nodes: ssh ansible@<node> (not root).

Common Tags

• servers — system packages, workstation taint/label tasks, NVIDIA container runtime on GPU nodes. Not update_packages — that tag silently does nothing.
• cluster — ArgoCD root app + all child apps (branch switching, service toggles). Sanctioned path for touching ArgoCD-managed state.
• k3s — K3s install/configure.
• known_hosts — populate /etc/ssh/ssh_known_hosts.

Operational patterns

• Ansible reaches nodes via the host SSH agent, forwarded into the devcontainer by VS Code Dev Containers (SSH_AUTH_SOCK is set automatically). Make sure your ansible key is loaded into the host agent before opening the container; the sandbox running Claude cannot see it, but ordinary devcontainer terminals can.
• Commits need uv run — pre-commit hooks live in the uv venv.
• Adding a node: run /add-node.
• Full bootstrap: run /bootstrap-cluster.
• Rebuild cluster: run /rebuild-cluster.

Branch switching

repo_branch in group_vars/all.yml must always be main — never edit it. To point the cluster at a feature branch, use just switch-branch <branch>, which passes -e repo_branch=<branch> as an override. Revert with just switch-branch main.

Check current tracked revision before branching new work

The cluster may already be on a feature branch (e.g. mid-validation of an open PR), not main. Branching new work off origin/main then running just switch-branch <new-branch> will silently roll back any commits that exist only on the previously-tracked branch. The most painful case is sealed secrets re-sealed during a rebuild: the new branch applies older (un-decryptable) versions from main, the controller logs no key could decrypt secret, and consumer pods drift from their live Secrets.

Before creating a new working branch:

kubectl -n argo-cd get application all-cluster-services \
  -o jsonpath='{.spec.sources[*].targetRevision}'

If it's not main, stop and ask the user whether to: (a) merge the tracked branch to main first, then branch off main, or (b) base the new work on the currently-tracked branch.

Topology rules

Multi-homed nodes

K3s and flannel auto-detect the IP from the default route, which may be the wrong subnet. Set node_ip and flannel_iface in hosts.yml for any node with multiple NICs — otherwise CNI traffic goes out the wrong interface and pod networking silently breaks.

Control-plane NoSchedule taint

The control plane (e.g. node01) is tainted NoSchedule. DaemonSets without a matching toleration won't schedule there, so it can safely be skipped when running --tags servers for node-level drivers (DRA plugins, etc.).

Workstation taint (ws03)

Workstation nodes carry workstation=true:NoSchedule because they may reboot unexpectedly. This has three consequences you must design for:

• DaemonSets that need ws03 must tolerate the taint. The nvidia-device-plugin template already does; audit any new DaemonSet that needs to run there.
• Longhorn does not tolerate the taint — no Longhorn storage runs on ws03. Anything with a Longhorn PVC must not be scheduled there.
• Monitoring statefulsets (Grafana, Prometheus) tolerate the taint so they can scrape workstation metrics, but must have a nodeAffinity rule excluding ws03. Otherwise their Longhorn PVCs get provisioned on ws03 and fail to attach.
• amd64-only workloads land on nuc2 by default. ws03 and nuc2 are the only amd64 nodes; ws03's taint means a chart with nodeSelector: kubernetes.io/arch: amd64 pins to nuc2 without any explicit hostname selector. Useful for charts whose schema forbids kubernetes.io/hostname overrides (e.g. thoth).

Longhorn replica count vs node count

Replica counts in kubernetes-services/values.yaml must match the number of Longhorn-capable nodes (i.e. excluding ws03). Going too high leaves volumes permanently Degraded; going too low under-replicates on adds. Update after any /add-node run.

BMC power operations

When a node is NotReady and SSH times out, ansible -m reboot and the graceful procedures in docs/how-to/node-operations.md cannot help — both the kubelet and sshd on the node are gone. Power-cycle via the Turing Pi BMC instead.

The BMC is reachable as {{ tpi_user }}@turingpi (default root@turingpi, defined in group_vars/all.yml). Each slot maps to a node via slot_num in hosts.yml: node01→1, node02→2, node03→3, node04→4. nuc2 and ws03 are not on the BMC.

# What the BMC thinks of all four slots
ssh root@turingpi 'tpi power status'

# Power-cycle a single node (off → wait → on) — the form used by roles/flash
ssh root@turingpi 'tpi power off -n 4 && sleep 15 && tpi power on -n 4'

Notes:

• Don't try kubectl drain first — the node is unreachable, so the drain will just hang. Just power-cycle and let workloads reschedule (or come back, for DaemonSets pinned to that node like rkllama).
• Hard power-cycle is unclean for in-flight writes. If the node hosts live local-PV data (see CLAUDE.md "Local PV data paths are sacred": Prometheus on node02, Grafana on node03, Open-WebUI on node04), expect fsck on boot — check pod logs once the node returns.
• For the Claude sandbox specifically: the BMC SSH key isn't reachable from the sandbox (the --clearenv bwrap wrapper strips SSH_AUTH_SOCK and the BMC host key isn't in the sandbox's known_hosts). The ! prompt prefix runs inside the same sandbox, so it doesn't help — ask the user to run ssh root@turingpi 'tpi power ...' from a regular devcontainer terminal or the host shell.

Foot-guns

• known_hosts task must run serial: 1 — parallel writes cause race conditions on the shared file.
• Traefik is disabled — --disable=traefik with NGINX Ingress. Do not assume Traefik CRDs exist.
• No automated tests — validate by running playbook tags against the live cluster.
• Ansible k8s module merges annotations — it never removes annotations that were previously set on an Ingress. If you remove annotations from a template (e.g. ssl-passthrough), you must kubectl delete the old Ingress first, then re-run the playbook to recreate it cleanly.
• ArgoCD valuesObject overrides values.yaml — for child apps like open-brain-mcp, the image tag is set in templates/*.yaml under valuesObject, not in additions/*/values.yaml. Editing values.yaml alone has no effect — check the template first.
• admin_emails is duplicated — must be kept in sync between kubernetes-services/values.yaml (Helm) and group_vars/all.yml (Ansible). After changing the Ansible copy, re-run --tags cluster.
• Prometheus admission webhook secret — kube-prometheus-stack's TLS secret (grafana-prometheus-kube-pr-admission) is not created by the Helm hook job under ArgoCD. The cluster role creates it via scripts/create-prometheus-admission-secret. If the operator is still CrashLooping after a rebuild, run just create-prometheus-admission-secret directly then delete the stuck pod.
• Decommission before ArgoCD — when tearing down the cluster, the decommission playbook deletes ArgoCD Applications (orphan cascade) before scaling down workloads. Doing it in the other order means ArgoCD reconciliation re-creates pods faster than you can remove them. After controller uninstall, strip finalizers from Longhorn CRD resources (volumes, engines, etc.) since the controller is gone.
• gh pr edit fails on this repo — classic projects warning causes a GraphQL error. Use gh api repos/OWNER/REPO/pulls/N -X PATCH -f body=... instead.
• local-nvme PVs get stuck Released after ArgoCD app delete — chart-generated PVCs (StatefulSet volumeClaimTemplates) are owned by the consuming ArgoCD app, so argocd app delete cascades to the PVC via resources-finalizer.argocd.argoproj.io. The static PV (Retain policy, pre-bound via claimRef) keeps the dead PVC's uid in spec.claimRef.uid. New PVCs get fresh UIDs and can't bind → pod Pending, scheduler reports didn't find available persistent volumes to bind. Fix: kubectl patch pv <name> --type=json -p '[{"op":"remove","path":"/spec/claimRef/uid"},{"op":"remove","path":"/spec/claimRef/resourceVersion"}]'. (This is the rare authorised exception to the no-mutate-ArgoCD- managed rule — you're clearing kube-managed runtime state, not changing template intent.) Long-term fix tracked in issue #397. Prefer argocd app sync --replace over argocd app delete/recreate for the five local-nvme-backed apps: open-webui, grafana, prometheus, plus the three supabase-* claims.

Key files

• pb_all.yml — main playbook
• pb_decommission.yml — teardown
• group_vars/all.yml — all variables
• hosts.yml — inventory with node vars
• roles/ — Ansible roles