Freeshard Failure Archaeology

This is the incident chronicle for shard_core (this repo, shard_core/ — the per-VM FastAPI service that manages Docker apps behind Traefik on a "shard", a customer's personal cloud VM). It records what broke, why, and what was decided, so a fresh session does not re-attempt a dead end or undo a settled decision.

Terms used throughout:

• controller = FreeshardBase/freeshard-controller, the central service that provisions shard VMs and delivers core versions (v1…v18: per-shard bundles of docker-compose.yml + host migration scripts). Production shards run what the core version pins, NOT what this repo's docker-compose.yml says.
• call_backend / call_peer = shard_core's sign-and-forward HTTP proxies (shard_core/web/protected/, shard_core/web/internal/call_peer.py) that relay requests to the controller / other shards with HTTP-signature auth.
• grind = the overnight agent loop: one headless AI session per GitHub issue, own worktree, branch from main, PR for human review. Nothing auto-merges.

Entry format: SYMPTOM → ROOT CAUSE → EVIDENCE → STATUS. All issue/PR numbers without a repo prefix are in FreeshardBase/freeshard. Dates of volatile facts are as of 2026-07-03.

---

1. Backup / rclone pipeline

The single most incident-dense subsystem. Read this whole section before touching shard_core/service/backup.py.

1.1 rclone rollout incident chain, 0.28.0–0.28.6 (seven releases in three days, 2024-04-20..22)

• SYMPTOM: auto-backup feature shipped in 0.28.0; production backups failed repeatedly, each fix revealing the next failure.
• ROOT CAUSE (serial): Docker image shipped without the rclone binary; then rclone's multi-line output broke parsing; then strict Pydantic validation of rclone's log JSON broke on missing fields; then a wrong SAS-token path.
• EVIDENCE: commits cba5806 "install rclone to docker image", 2d2ec22 "use last line of rclone output", 9666db5 "make all rclone log fields optional", b4c5eda "Use new backup sas token path"; tags 0.28.0 (2024-04-20) through 0.28.6 (2024-04-22).
• STATUS: fixed by 0.28.6, but the parsing class of bug recurred — see 1.2.

1.2 rclone-output validation recurrence (2026) — the opaque-dict rule

• SYMPTOM: backups failed validation again two years later, right after the Postgres release.
• ROOT CAUSE: same class as 9666db5 (2024): Pydantic-validating rclone's stats JSON, whose shape is not a stable contract.
• EVIDENCE: commit b9dfcfd "treat rclone stats as opaque dict to prevent backup validation failures" (released 0.38.3, 2026-04-18).
• STATUS: fixed. Standing rule: never Pydantic-validate rclone stats/log output — treat it as an opaque dict. It changed shape twice; assume it will again.

1.3 The 0.37.5 backport anomaly — tag order ≠ commit topology

• SYMPTOM (for archaeologists): tag 0.37.5 (2026-04-18) is newer in time than 0.38.0 (2026-04-17).
• ROOT CAUSE: the freshly released 0.38 Postgres line was evidently not trusted for immediate fleet rollout, so the rclone-stats fix was cherry-picked (0a5f249) onto branch origin/release/0.37.5 (3392ff3) and shipped on the old TinyDB line too.
• STATUS: historical fact. Never assume tag order equals commit topology in this repo. Resolve with git branch -a --contains <tag>.

1.4 CronTask silent death kills backups permanently

• SYMPTOM: scheduled backups stopped forever with no error surfaced; only a restart revived them.
• ROOT CAUSE: CronTask background scheduler died on the first uncaught exception and never rescheduled.
• EVIDENCE: issue #58, fixed in PR #61 (merged 2026-04-14).
• STATUS: fixed. Rule: any background/periodic task must catch and log exceptions, or it dies silently and permanently.

1.5 No external backup-recency signal → marker blob

• SYMPTOM: the controller could not tell whether a shard's backups were actually running (see 1.4 for why that mattered).
• FIX: write a marker blob after each successful backup; the controller polls it.
• EVIDENCE: issue #59, PR #62 (merged 2026-04-14).
• STATUS: fixed. Any change to backup flow must keep the marker-blob write.

1.6 datetime in JSONB

• SYMPTOM: post-Postgres backups broke on serializing the directories structure.
• ROOT CAUSE: datetime objects are not JSON-serializable when writing JSONB columns.
• EVIDENCE: PR #74 "fix(backup): serialize datetime in directories JSONB" (commit 935250b, released 0.38.4).
• STATUS: fixed. Explicitly serialize datetimes before they hit JSONB.

1.7 Azure list-ops cost fix → v18 release-blocker regression

• SYMPTOM (first): Azure "List and Create Container Operations" meter cost several times the actual stored backup data.
• FIX (first): add --fast-list and --azureblob-no-check-container to the rclone sync, PR #106 (merged 2026-06-13).
• SYMPTOM (second): all backups on core v18 failed — the rclone binary pinned in the shipped image did not know --azureblob-no-check-container.
• EVIDENCE: release-blocker issue #117, fixed by PR #119 dropping the unsupported flag (2026-06-30).
• STATUS: fixed. Rule: any new rclone flag must be validated against the rclone version pinned in the Docker image before merge.

Also in this subsystem: empty directories were silently lost by backup/restore — fixed in PR #52.

---

2. Database: three dead migrations, one success, and the rollout fallout

2.1 The THREE dead DB-migration attempts before PR #56

TinyDB→relational-DB was attempted four times. Before restarting ANY storage work, read the dead branches (section 10 checklist).

Attempt	Branch	Last commit	Fate
1. SQLite route	origin/feature/sqlite	2025-01-13	Abandoned mid-way (per-store migration commits: peers, tours, terminals, app-usage, backup-stats). No in-repo rationale for abandonment.
2. Copilot Postgres	origin/copilot/migrate-database-to-postgres	2026-02-01	PR #29 closed unmerged.
3. Agent Postgres	origin/issue/25-replace-tinydb-with-postgres	2026-02-24 (tip lineage includes ab73e25 "WIP claude interrupted")	PR #34 closed unmerged.
4. Success	origin/replace-tinydb-with-postgres	—	PR #56 merged 2026-04-17 (commit 2980bfb, merge 0837749), released 0.38.0. Driving issue #25 (closed 2026-06-29 in bulk backlog cleanup).

2.2 Postgres rollout fallout, 0.38.0–0.38.4 (2026-04-17..23)

• SYMPTOM: four patch releases within six days of the biggest storage change in project history.
• FIX SEQUENCE: 0.38.1 c824636 (CI app-install timeout flakiness), 0.38.2 dc45ced (LifespanManager shutdown_timeout teardown flakiness), 0.38.3 b9dfcfd (rclone opaque dict, see 1.2), 0.38.4 935250b (datetime JSONB, see 1.6). Plus 358a908 "Fix CI test hang: commit migration SQL and guard pool cleanup" inside the migration branch itself.
• STATUS: all fixed. Pattern to expect: any infra-level minor release is followed by a burst of 2–4 patch releases within days. Plan review and rollout capacity accordingly.

2.3 DB pool exhaustion under forwardAuth bursts

• SYMPTOM: batches of 500s on private apps; SPA page loads (~30 parallel requests, e.g. Actual Budget) exhausted the small connection pool via Traefik forwardAuth calls to /internal/auth, requests waited ~30 s then failed.
• ROOT CAUSE: /internal/auth runs on EVERY private-app request and did 2–3 DB queries per call against a 4-connection pool.
• FIX: pool bump (max_size=20) + in-process auth-path caches; then a follow-up commit on the same PR fixing cache invalidation correctness (signal-driven, no TTLs).
• EVIDENCE: issue #89 (was labeled high-priority), PR #90 (merged 2026-05-22); commits 1e0e7c1 (cache added) then 868e690 (invalidation fixed).
• STATUS: fixed. If you touch identity or app state, check whether the auth caches need an invalidation signal — the first cut of the cache shipped without one and had to be patched.

---

3. Docker lifecycle: stale state fixed three times

Docker objects (containers, networks) survive shard_core updates and go stale. This bit three separate times before the lesson stuck:

#	Fix	Commit / PR	Date
1	Recover from stale Docker network references on app start	d66d889, PR #54	2026-04-08 (0.37.4)
2	Handle stale container name conflicts on docker_start_app (symptom fix)	fefd2f4, PR #63	2026-04-14
3	Fix root cause of stale containers on core update	751678d (same PR window as #63)	2026-04-14

• STATUS: fixed, but structural. Rule: any new docker-start code path must recover from name conflicts and dangling network references — do not assume clean Docker state after a core update.

Related settled work: zombie healthcheck curl processes fixed by tini as PID 1 (PR #55, commit 57ac53d); standalone docker-compose v1 → docker compose v2 plugin (issue #35, PR #87); app compose templates re-rendered on startup (PR #49).

3.1 python-on-whales migration — stalled, not dead, not approved

• Oldest open issue: #24 "Integrate Python on Whales or some other docker python lib" (2025-12-19). PR #88 has been open with zero reviews since 2026-05-18.
• STATUS: open/stalled. Do not start a competing docker-lib migration, and do not treat PR #88 as accepted direction — it has never been reviewed.

---

4. Traefik / ingress / proxying

4.1 fd-exhaustion outage arc (June 2026) — the worst recent incident

• SYMPTOM: shards "up but not serving" — containers healthy, nothing reachable. Traefik logged accept4: too many open files (EMFILE) in a hot loop; the error spam itself filled the root disk (controller issue #306: unbounded container logs bricked shard upgrades).
• ROOT CAUSE — two factors, both required:
  1. Production Traefik was pinned v2.6, built with a pre-1.19 Go (measured go1.17.10; the "Go 1.16" figure in controller PR #322's prose is imprecise — see freeshard-proof-and-analysis-toolkit Recipe 1). Go only self-raises RLIMIT_NOFILE from 1.19 on, so Traefik ran with the soft limit.
  2. Newly provisioned shards moved to an Ubuntu 26.04 base image (controller PR #299, forced by OVH image rotation — see 6.3) where the effective per-container nofile soft limit was 1024. Older shards had the identical socket leak (readTimeout=0 on an internet-scanned IP) but survived only because their fd ceiling was higher — "old shards are fine, must be luck" was investigated and disproven; it was ceiling height, not luck.
• FIXES (multi-repo):
  • shard_core: finite readTimeout on Traefik entrypoints stops the leak — commit fd1d05b, PR #108 (merged 2026-06-21, released 0.39.4 on 2026-06-24).
  • controller: core v16 log rotation (#309) — which silently did nothing at first, see 6.1; core v17 nofile ulimit (#311) and daemon restart so config actually applies (#312).
• DECISION — stay on Traefik v2: bump to v2.11 (built with Go 1.25, self-raises nofile ≈1024→64000) instead of the "obvious" risky v3 migration. Canonical statement is the closing comment on PR #112 (closed unmerged 2026-06-24): v3 migration folded into a needs-Intent P3 backlog ticket. Confirmed in production: core v18 compose pins traefik:v2.11.
  • Do not resurrect v3-prep changes (HostRegexp→Host() style); local branch fix/traefik-dashboard-host-rule (2026-06-23) is that dropped work.
  • TRAP: this repo's own docker-compose.yml says image: traefik:v3.6 (bumped e78862d, 2025-12-16) — that is the dev/self-hosted compose only. Production shards run the controller core-version pin (v2.11 as of core v18, 2026-07-03). Do not "fix" either to match the other without reading PR #112 first.
• PROCESS FALLOUT: the readTimeout fix was merged but sat in no released tag while shards kept failing — issue #111. Merged-is-not-shipped; see freeshard-change-control. A redundant version-bump PR #113 was closed unmerged because main had already been bumped directly — always git fetch and check origin/main + published releases before opening a release PR.
• STATUS: fixed (0.39.4 + core v17/v18); Traefik-v3 migration = open backlog, decided-not-now.

4.2 Content-Type proxy landmine

• SYMPTOM: every proxied write to the controller returned 422; reads worked fine. Red herring chased: body double-encoding.
• ROOT CAUSE: call_backend relayed bodies but dropped the Content-Type header (Python requests sets none for data=<bytes>). Harmless for years, until the controller's FastAPI upgrade made JSON parsing strict about the header — a latent bug armed by a dependency bump in a different repo.
• EVIDENCE: issue #93 (high-priority), PR #94 (merged 2026-05-27). Diagnosed with tcpdump in the container netns because Traefik swallowed the detail.
• STATUS: fixed.

4.3 call_backend/call_peer is a repeat offender — full rap sheet

Bug	Evidence	Fixed
Query strings not forwarded to controller	PR #19	2025-12-13
Slow proxying (buffered instead of streamed bodies)	issue #38, PR #39	2026-03-22
Content-Type dropped on writes	#93/#94	2026-05-27

Checklist when touching any sign-and-forward proxy code here: forward query strings, forward Content-Type (and other load-bearing headers), stream bodies. Note _get_app_for_ip_address in shard_core/web/internal/call_peer.py is lru_cache'd and carries a "todo: test this" — untested hot path.

4.4 Splash-screen stalemate (#91/#92) — UNRESOLVED, do not code around it

• GOAL: embed the real upstream error response (hidden) in the splash page so it's visible in dev tools — issue #91.
• CONFLICT: PR #92 implements an internal reverse proxy; maintainer max-tet requested changes ("too complex — no need for a separate endpoint"). The agent's counter-argument: Traefik's errors middleware discards the upstream body, so some proxy is technically required. No movement since 2026-05-27.
• STATUS as of 2026-07-03: open architectural disagreement. Do not merge-adjacent work that presumes either outcome; if you pick this up, resolve the disagreement in the issue thread first (see freeshard-change-control for how decisions get made).

---

5. CI and release-process incidents

5.1 GitLab→GitHub CI bring-up chain, 0.31.0→0.32.0 (six tags in three days, 2025-02-24..26)

• CONTEXT: GitLab registry credentials had already died once (0.30.4–0.30.6, commits 584113e, 4d6e30f, 2025-01-20..21). Full migration: b33eada "delete .gitlab-ci.yml" + 0729394 "add github actions" (2025-02-24).
• SYMPTOM: pipeline debugged by trial and error directly on main: b258fbd "fix pipeline env vars", d67abc6 "fix pipeline permissions", d2acc45 "try another way to handle slug vars", 59c4e07 "use sha as label temporarily", 42cf5c6 "hard code registry path", d309113 "skip json-schema and api-docs for now".
• STATUS: pipeline works, BUT the "temporary" skip from d309113 (2025-02-26) is still in place — the json-schema job is commented out in .github/workflows/release.yml (line ~75) as of 2026-07-03. If your change relies on published JSON schemas or API docs, they are not being published.

5.2 Chronic CI flakiness (multi-year)

tests/conftest.py is the most fix-touched file in the repo (33 fix-commit touches). Landmarks: dcf6207 skip flaky peer-call tests (2022), 28964fd network not torn down between tests, aad605a telemetry test converted to unit test, c824636 install-timeout bump (2026), dc45ced LifespanManager shutdown_timeout. Before "fixing" a flaky test, check freeshard-testing-and-qa — most flake classes here have a known cause.

5.3 Reverts are rare — treat them as signals

Only ONE true git revert exists in ~784 commits: 3ee5a71 (reverts e0c4bed, 2023 test-infra). The project's failure mode is not revert-thrash; it is fix-tails (patch bursts after infra releases) and abandoned branches. When you see an unmerged branch, assume "pending decision or dead end", never "free code to merge" — check the PR/issue state first.

---

6. Controller-side incidents shard work must know about

These live in FreeshardBase/freeshard-controller but bite shard_core work directly.

6.1 Docker daemon.json reload-vs-restart trap

• SYMPTOM: core v16 shipped daemon-wide log rotation (max-size/max-file); it silently did nothing.
• ROOT CAUSE: it was applied with systemctl reload docker. SIGHUP reloads only a whitelisted subset of daemon.json — log-driver/log-opts and default-ulimits are NOT in it, and containers keep stale in-memory values even when recreated after a reload.
• EVIDENCE: controller PRs #309 (v16, ineffective) → #312 "v17 daemon-wide docker config + restart (rotation actually applies, fd ceiling)".
• STATUS: fixed in core v17. Rule: log-opts/ulimit changes need systemctl restart docker (bounces all containers — maintenance window), never reload.

6.2 Migration-delivery trap — the sentinel guards the wrong thing

• SYMPTOM: host migrations silently never ran on fresh shards and on version-skipping upgrades (e.g. v15→v17); a broken shard was found where the nofile migration had never executed.
• ROOT CAUSE: script delivery was version-scoped (only the target version's scripts were uploaded), and the .migrations_applied sentinel only prevented re-running — it gave zero protection against never-delivered scripts.
• EVIDENCE: controller issue #307, fix PR #308 (cumulative union of all versions' scripts, globally-unique vN-NN-description.sh names, numeric ordering); follow-up #323 replaced run-once sentinels with idempotent reconcilers run on every converge.
• STATUS: fixed. Design rule: prefer idempotent reconcile-on-every-converge over run-once sentinels.

6.3 OVH base-image rotation broke provisioning

• SYMPTOM: shard provisioning failed with a cryptic list index out of range; auto-provisioning disabled itself.
• ROOT CAUSE: OVH removed the non-LTS "Ubuntu 25.04" image from the region without notice; code did exact-name match + [0] on an empty list.
• EVIDENCE: controller PRs #299 (bump image), #300 (image name becomes a controller-managed runtime setting), #301 (clear error when image/key not found).
• STATUS: fixed; lessons: pin LTS images only; provider provisioning inputs are runtime settings, not static config. Side effect: the Ubuntu 26.04 image was factor 2 of the fd-exhaustion outage (4.1).

6.4 OVH SDK singleton auth-brick

• SYMPTOM: after one transient OVH token-endpoint 500, ALL subsequent OVH calls returned "401 You must login first" until controller restart; two shard deletions failed and the VMs kept billing.
• ROOT CAUSE: process-level ovh.Client singleton cached a stale token with no recovery path.
• EVIDENCE: controller issue #275 (closed 2026-06-05).
• STATUS: fixed. Pattern: SDK singletons with token caching need auth-error reset+retry.

6.5 Lexical version sort — "everything was fine until v10"

• SYMPTOM: the system thought core v9 was newer than v10; bit in four places at once (backend latest-version, /core-versions list, frontend latest, upgrade dropdown).
• ROOT CAUSE: version directory names sorted as strings: "v10" < "v9".
• EVIDENCE: controller PRs #260, #262 (merged 2026-05-26).
• STATUS: fixed. Rule: never sort version identifiers lexically; use a numeric ordinal, and the frontend trusts backend order. Related: shard_core's move to real semver is open — issue #118.

6.6 Pricing rounding divergence — Python round() vs JS Math.round()

• SYMPTOM: displayed price label could disagree with the actual PayPal charge by 1 cent on half-cent results.
• ROOT CAUSE: Python round() is banker's rounding, JS Math.round() is half-up; they agreed under the old formula and diverged once the 19%-VAT factor produced half-cent values.
• EVIDENCE: controller PR #298 — switched backend to half-up int(x + 0.5), bit-identical to Math.round for positive values; sibling PRs web-terminal#30 and landing-page#15 (formula is manually duplicated across repos and must change in lockstep).
• STATUS: fixed. Rule: never use Python round() in pricing code.

6.7 PayPal quantity-pricing workaround