Deploy

The release process for the beet website. Validate the SAME site across three environments in sequence, each in its own sub-agent, each with an IDENTICAL verification pass:

1. Local proves the build (serve on localhost, verify, stop).
2. Dev proves the cloud path (deploy to dev.beet.org, verify, then tear down, since a standing dev environment is a real monthly cost).
3. Prod publishes the permanent production site (beet.org + www.beet.org).

Run each step as a SEPARATE sub-agent. Do them in series: Dev only after Local is green, Prod only after Dev is green and torn down.

Dev MUST be torn down after verification. Only prod stays up. Dev is a temporary proving ground for the cloud path, not a standing environment: a live dev stack is a real recurring monthly cost. The moment dev verification is green (or fails, or is abandoned), run just beet-destroy and confirm no beet-site--dev--* S3 buckets, no dev Lightsail instance or static IP, and no dev.beet.org DNS record remain. Never leave dev running.

Topology

The deploy block is stage-aware (LightsailBeetSiteBlock, crates/beet_extra/src/infra/templates.rs). One small_3_0 Lightsail box (2 GB, flat monthly price) per stage carries everything: no load balancer, no container registry, no ACM certificate, no VPC of its own.

• http: Caddy on the box terminates TLS for every hostname with an automatic Let's Encrypt cert and reverse-proxies to the beet binary on its app port. Cloudflare fronts it PROXIED (an A record at the static IP) and runs Full-strict, so the edge verifies Caddy's cert.
• ssh: the beet TUI listens on port 22. Cloudflare does not proxy raw TCP, so ssh rides a DNS-ONLY app.* hostname pointing straight at the static IP. The box's own management sshd moves to 2222 (reachable with the stack's Lightsail key pair). Note this means the Lightsail browser console, which only dials 22, lands on the beet TUI rather than a shell.
• DNS (stage-aware, and this is why --stage matters): dev publishes ONLY dev.beet.org (proxied) + app.dev.beet.org (DNS-only). prod publishes the apex beet.org + www.beet.org (proxied) + app.beet.org (DNS-only). A dev deploy can never touch production apex DNS.

The generic beet binary reads the whole site from ONE bucket at runtime, the per-stage app bucket beet-site--<stage>--app: main.bsx, routes/, templates/ and assets/ all live in it, replicated from the checkout's site/ by the deploy sync. No deployed binary reads beet-site--shared--assets; that bucket is the developers' source of record for site/assets, owned by just site-shared pull|push and untouched by any stage deploy. (The separate beet--shared--assets bucket backs the WORKSPACE ./assets tree the examples, tests and wasm builds read; just beet-shared pull|push. The two trees are not mirrors: blog/ and branding/ belong to the site, and a few workspace-built files — the wasm binary, the geoip database, the robot faces — are borrowed into site/assets by the <DirCopy/> manifest in site/main.bsx, which runs ahead of every publish.) The deployed binary is built with --features aws_sdk,ssh,geoip (see <LightsailBeetSiteBlock features="aws_sdk,ssh,geoip"/> in site/main.bsx), so the served site includes the ssh terminal and country lookups. aws_sdk alone would serve http only.

The site entry declares its own infrastructure. site/main.bsx is a one-shot <CliServer{always:true}> dispatcher whose site is the serve route (the box launches app --repo=s3://.. --server=http,ssh serve, and a developer launches beet --main=site serve --server=http: one process, one argv, the same verb). Its raw analytics, rollup, and archive stores are declared ONCE under the stage <Stack> as writable <S3BucketBlock/> declarations: the deploy provisions them and runtime consumers reach them through StoreRef($analytics), RollupStoreRef($rollup), and ArchiveStoreRef($archive). Each composes <app>--<stage>--<label> from the app_name on the entry's <PackageConfig/>; locally the same declarations attach FsStores under target/stores/<label>. The deploy verbs do not load in the deployed binary at all: the <Stack> carries bx:cfg="feature:infra && feature:extra", which excludes the whole subtree before the build walk in a binary that links neither crate, leaving a tombstone that reports the condition if anything dispatches into it. A Router is a url space, so the site's urls stay rooted at / and no http/ssh request can reach a deploy route (curl /deploy -> 404).

The box is replaced only when the BOX changes. The instance's user_data renders machine config alone (the blueprint, Caddy, the sshd relocation, the CloudWatch agent, the systemd unit), never a version, so a code-only deploy plans no change to aws_lightsail_instance. The binary instead reaches the box through the artifacts bucket's stable release pointer, s3://beet-site--<stage>--artifacts/current/main-lightsail.env, which names this deploy's id and its versioned artifact key. The unit's ExecStart is a launcher (/usr/local/bin/beet-site-run) that re-reads that pointer at every start, so systemctl restart IS the deploy, and <LightsailRelease/> (after <TofuApply/>) performs it over the management sshd on 2222 and gates on the box actually serving: the RUNNING process reports this deploy's id (read from /proc/<MainPID>/environ), one request through the app's own port on loopback is answered, and NRestarts is unchanged across that whole check. A fetch that fails leaves the installed binary serving rather than killing the unit.

Editing LightsailBlock::build_user_data (bumping CADDY_VERSION, changing the unit, moving the management port) IS a rebuild: terraform cannot update user_data in place, so the instance is destroyed and recreated, and the IAM access key rotates with it (the rotation trigger is keyed on a digest of the rendered script, so the key rotates with every machine-config change; a rebuild from a non-script change, ie a bundle resize, keeps its key). Budget several minutes AFTER the instance reports running for cloud-init to install Caddy, pull the binary from S3, obtain the Let's Encrypt cert and start the unit; the static IP re-attaches so the address and DNS are stable. There is still no circuit breaker and no automatic rollback: a binary that fails to boot leaves a broken box. Rolling back is re-pointing the release pointer (beet rollback, which rewrites it) and restarting the unit.

This matters beyond the outage window. small_3_0 is burstable at a 20% baseline per vCPU, and a fresh instance starts at ZERO burst capacity, so a rebuild does the heaviest work of the instance's life while clamped to baseline. That is what produced the cluster of AWS SDK dispatch failure errors within ~75 seconds of unit start on a freshly deployed box. Covered by code_only_deploy_renders_one_box and machine_config_change_rebuilds_and_rotates in crates/beet_infra/src/blocks/lightsail.rs.

beet-deploy exiting 0 does NOT mean the SITE serves. The deploy's last steps are a timed log tail and a cache purge, neither of which gates on readiness, and cloud-init failures do not propagate back to it. <LightsailRelease/> now proves the APP serves — an active unit carrying this deploy's id that answers a request on its own port with no restart underneath — so the old failure where a crash-looping box reported is serving release .. after ~340 restarts is closed, and a deploy exiting 0 while LightsailWatch prints a fatal boot error in that same log should not recur. What the gate deliberately cannot see is everything in FRONT of the app: it probes loopback, so Caddy, DNS and certificate state are all still unproven. ALWAYS curl the site after the deploy returns. Diagnosis path for a bad origin:

• Cloudflare 521 = the edge reached DNS but nothing accepted on the origin's 443. 526 = Caddy answered but its cert is not valid for the Full-strict edge.
• Get a real shell on the management sshd (port 2222, the stack's Lightsail key pair, user ec2-user). Every deploy materialises that key beside the rendered config as target/infra/beet-site/deploy_key.pem (mode 600) for <LightsailRelease/>'s own ssh, so reach for it rather than digging the tofu state's aws_lightsail_key_pair out by hand: ssh -i target/infra/beet-site/deploy_key.pem -p 2222 ec2-user@app.<stage-host>. It is the stage of the LAST deploy, so re-run the deploy's stage before trusting it. Then: systemctl is-active caddy beet-site, ss -lntp, and sudo grep -iE "error|fail" /var/log/cloud-init-output.log.
• beet-site active on its app port while caddy is missing means the TLS terminator failed to install and everything else is fine.

REGRESSION GUARD (fixed): the deploy shipped a binary it never built. A block is declared under its <Stack> rather than as a step in the deploy sequence, so nothing dispatches it and its BuildArtifact's action never ran; <TofuApply/> went on reading that artifact's file off disk, hashing it and uploading it, so every deploy silently shipped whatever an earlier deploy had left in target/ (and on a machine with no prior artifact would fail on a missing file instead). The box then crash-looped on failed to load entry main.bsx: no component, resource or template registered for tag Stack, warning that types newer than the stale binary were unregistered, while beet-deploy reported success. Building now belongs to the step that CONSUMES the artifact: TofuApply calls BuildArtifact::build() immediately before reading each artifact's bytes, so an artifact that is uploaded but never built is unrepresentable. Covered by build_runs_the_declared_process in crates/beet_infra/src/actions/build_artifact.rs. A deploy log with no building: cargo-zigbuild .. line is this bug back.

REGRESSION GUARD (fixed): a rebuild reset the box's firewall and took prod down. aws_lightsail_instance_public_ports names the instance by NAME, which survives a rebuild, so nothing in its own config diffs and the plan leaves it untouched while the replaced instance comes up with Lightsail's DEFAULT rules (22 + 80). State then claims ports AWS has closed. Live, the --store -> --repo argv change replaced the prod box and shut both 443 (522 at every hostname, the site down with a perfectly healthy app behind it) and the management ssh port, so <LightsailRelease/> sat on waiting for ssh on <ip>:2222 (attempt N/60) for 25 minutes; opening the two ports by hand let the release pass INSTANTLY with beet: already serving, which is the tell that the box was never the problem. The ports resource now carries replace_triggered_by on the instance, exactly as the static IP attachment already did. Covered by rebuilding_the_box_reapplies_the_firewall in crates/beet_infra/src/blocks/lightsail.rs. Diagnose with aws lightsail get-instance-port-states --instance-name beet-site--<stage>--main-lightsail-instance: a rebuild deploy whose plan does not list the ports resource is this bug back.

KNOWN, NOT FIXED: a boot-time Let's Encrypt failure leaves the box serving TLS the edge rejects, with no retry gate and nothing in the deploy that notices. On the rebuild above, http-01 validation SUCCEEDED for beet.org and www.beet.org and the chain download then failed (HTTP 404 urn:ietf:params:acme:error:malformed - Certificate not found), so Caddy held a cert for app.beet.org alone and both proxied hostnames served 525 (distinct from 521: something IS listening on 443, its cert is just not one Full-strict accepts). <LightsailRelease/> cannot see this, since it probes loopback. The recovery is one line — sudo systemctl restart caddy on the management sshd, which re-obtains immediately — so on any 525 check sudo find /var/lib/caddy -name '*.crt' for a MISSING hostname before looking anywhere else.

REGRESSION GUARD (fixed): Caddy was installed via its cloudsmith rpm repo, whose setup script writes an amzn/2023 baseurl that Caddy does not publish, with skip_if_unavailable=1. dnf install -y caddy therefore exited No match for argument: caddy, cloud-init logged a failed scripts-user module and carried on, and the box booted serving the app on its own port with NOTHING on 80/443 — while beet-deploy reported success and every hostname returned 521. Caddy is now installed from the pinned upstream static release (LightsailBlock::CADDY_VERSION) with its own systemd unit, and the script verifies the binary (caddy version || exit 1) so a failed install stops the boot instead of silently producing a TLS-less box. Covered by installs_caddy_from_static_release_not_rpm in crates/beet_infra/src/blocks/lightsail.rs.

AN INTERRUPTED APPLY LEAVES TWO MESSES, AND BOTH BLOCK THE RETRY. A deploy killed mid-apply (SIGTERM, a lost terminal, a timeout) leaves (1) a stale state lock and (2) resources that exist in AWS but not in state, because tofu creates them before it persists. The retry then fails twice over: first Error acquiring the state lock naming a lock id and the time it was Created, then a wall of EntityAlreadyExists / ResourceAlreadyExistsException. Neither is a code fault and neither self-heals. Recovery, in order:

1. Confirm nothing is actually running (pgrep -af tofu), then tofu -chdir=target/infra/<app> force-unlock -force <LOCK_ID> using the id from the error. The lock is an object in the state bucket (<key>.tflock); deleting it by hand works but skips the id check.
2. Diff tofu -chdir=target/infra/<app> state list against what AWS actually holds, and tofu import <address> <id> each orphan. Prefer import to deleting and recreating whenever the resource is live (a schedule that has fired, a log group holding evidence). The ids are per-type: a log group is its name, an IAM role its role name, an inline role policy <role>:<policy>, a policy attachment <role>/<policy-arn>, a lambda its function name, a scheduler schedule <group>/<name> (ie default/<name>).
3. Re-run the deploy; it should now converge to no-op plus whatever genuinely changed.

CHECK THE REAL EXIT CODE. just beet-deploy ... | sed ...; echo $? reports the exit of sed, so a deploy killed by a signal reads as success. Redirect instead of piping (just beet-deploy --stage=prod > log 2>&1; echo "EXIT=$?") and strip the ANSI afterwards, or set -o pipefail. just prints recipe ... was terminated ... by signal 15 on the line above, which is the real tell.

AN APPLY OUTLIVES MOST TOOL DEADLINES; RUN IT DETACHED. A full deploy takes 10-15 minutes, and a deploy driven from a tool with a kill deadline (an agent harness, a CI timeout) can SIGTERM the mid-flight apply, leaving exactly the two messes above. Run it detached and watch the log instead: setsid nohup sh -c 'just beet-deploy --stage=prod > .agents/tmp/deploy.log 2>&1; echo "EXIT=$?" >> .agents/tmp/deploy.log' &, then tail the log until the EXIT= line appears.

REGRESSION GUARD (fixed): the release step passed a box that never served. <LightsailRelease/> confirmed only that the unit was active and that /proc/<MainPID>/environ carried this deploy's id — but under Restart=always/RestartSec=3 a crash-looping unit ALWAYS has a live MainPID carrying the right id, so a deploy exited 0 while the box 502'd and restarted ~340 times. The converge loop is now the gate: an attempt passes only when the unit is active, the running process reports the expected id, curl -fsS --max-time 5 http://127.0.0.1:<app_port>/ succeeds, AND NRestarts is unchanged across the whole attempt (a crash loop that answers one lucky request between restarts still fails). The probe is loopback deliberately — with a domain the firewall opens 80/443 only, so the app port is reachable from the box alone, which is exactly "the app itself serves". Covered by release_proves_the_running_process and release_probes_the_declared_port in crates/beet_infra/src/blocks/lightsail.rs. A release step reporting success on a box whose NRestarts is nonzero is this bug back.

Do NOT read a single curl: (22) The requested URL returned error: 500 in that step's narration as a failure. The script logs the stderr of EVERY attempt, so one failed probe followed by a pass is a converge loop doing its job, and on a fresh box the first request routinely is that failure: the box's very first / is served while the instance is still at zero burst capacity, and it 500s on an AWS SDK dispatch failure reaching the repo store, the same clamp described under Topology. The tell that it was transient rather than a crash loop is the box log — request #1 -> 500, #2 -> 200 OK — plus NRestarts=0 afterwards. A 500 that repeats across attempts fails the gate on its own and the deploy never reaches this line.

Commands

Creds load from .env (AWS, CLOUDFLARE_API_TOKEN/CLOUDFLARE_ZONE_ID, BEET_SSH_HOST_KEY). Always use the beet-* recipes: they build with --features infra,extra (without which the deploy routes load as inert tags and the verb does nothing) and clear AWS_PROFILE.

intent	command
local serve (http+ssh)	cargo run -p beet-cli -- --main=site serve --server=http,ssh
pre-apply safety check	just beet-validate then just beet-plan (eyeball the plan)
dev deploy	just beet-deploy
re-publish site only (no redeploy)	just beet-sync
tail the instance logs	just beet-watch
dev destroy	just beet-destroy
prod deploy	just beet-deploy --stage=prod

just beet-plan --stage=prod shows the prod plan without applying.

Verification (IDENTICAL for every step)

Each step verifies its environment with the same five checks. Parameters: a BASE_URL (http://localhost:<port> for local, https://dev.beet.org for dev, https://beet.org for prod) and an SSH target (127.0.0.1 + the local ssh port for local; the DNS-only app. subdomain + port 22 for dev/prod, ie app.dev.beet.org / app.beet.org -- the bare dev.beet.org/beet.org are Cloudflare-proxied web-only and do NOT forward ssh). All five must pass.

a. curl (raw http)

GET each key page; assert HTTP 200 and the expected marker:

• / (home) -> 200, renders the landing page
• /docs -> 200
• /docs/design -> 200
• /docs/design/counter -> 200, body contains Counter and You have clicked
• /docs/design/color_schemes -> 200 (the styles page)
• /blog -> 200, and the body carries a /blog/<slug> link per post. The listing is generated by <RouteIndex reverse="true"/> in site/routes/blog/index.bsx rather than maintained by hand, so a scan that found no posts renders an EMPTY page that still answers 200: assert the count, not the status.
• /blog/folk-technology -> 200 (a post reached by its slug, the segment <RoutesDir/> derives from the post's own slug frontmatter rather than its <number>- filename)
• /blog/post-6 -> 301 to /blog/folk-technology. Every post was renamed from post-<n> to a slug, and site/main.bsx keeps the old urls alive with a block of <Redirect path="post-<n>" redirect="<slug>"/>. That block is part of the entry document the box loads out of the app bucket, so it is exactly as deployable-and-breakable as a route: sweep all fifteen (for n in $(seq 1 15); do curl -o /dev/null -w '%{http_code} %{redirect_url}\n' "$BASE/blog/post-$n"; done) and assert every one is a 301 to a slug that itself answers 200. A <Redirect/> carries no PageRoute, so it is absent from the sidebar and from RouteIndex, and nothing but this check covers it.

Also fetch /docs/design/counter?color-scheme=light and ?color-scheme=dark and confirm 200 (the scheme is applied server-side for the screenshot check below).

/deploy -> 404. The Router is a url space, so a served request can never reach the sibling deploy commands; a 200 here means the site's routes and the stack's have been merged into one namespace.

b. browser verification (navigability, the counter, client errors, mobile layout)

The counter (site/routes/docs/design/counter.bsx) renders a document field as "You have clicked N times.", with a "More" and a "Less" button whose bx:click scripts mutate it. The browser check asserts first paint only: there is no client interactivity tier until the wasm one lands, so a click in the browser does nothing yet. The whole check is one committed test driving the in-house webdriver (chromedriver + a chromium on PATH are the only deps):

BEET_BASE_URL=<BASE_URL> cargo test --test site_browser \
  --features router,json,testing,webdriver -- --include-ignored

tests/site_browser.rs runs everything below and exits non-zero on any client error or overflow. The checks:

Client errors (fail on any). Before navigating, two collectors attach and the run fails if either fires, so a broken client script cannot ship silently -- exactly the miss that let crypto.randomUUID is not a function reach the analytics beacon in production:

• page.console() -- console.error plus uncaught exceptions and failed-request console messages (BiDi log.entryAdded).
• page.responses() -- any 4xx/5xx subresource. A favicon that fails to load raises no console error, so without this a site whose every asset 403s still reports green.

Asset sweep. Load a page that actually carries an image (/blog/folk-technology), collect every /assets/ reference on it (img[src], link[href], script[src], source[src]), fetch each following redirects, and assert 200. This is the store-topology check: the app serves assets from its own bucket, and a private bucket handing out a public url turns every asset into a redirect to a 403. Both this and the response collector were added after exactly that shipped past a green run (S3Store::public_url claimed a virtual-hosted url for a private bucket; it now returns None unless the store is explicitly with_public(true)).

Ignore nothing by default; if a message is genuinely benign, match it exactly and log that it was skipped. CAVEAT: some faults only surface in an insecure context -- crypto.randomUUID/crypto.subtle are gated to secure contexts, and localhost + https:// are both secure, so this check does NOT reproduce that specific bug. The durable fix is keeping secure-context-only APIs out of the client (the beacon now derives its id from crypto.getRandomValues, available on plain http); the collectors still catch the broad class of client JS errors on every env.

Counter + navigability. Headless chromium: goto BASE_URL/docs/design/counter and assert its first paint ("You have clicked 0 times", plus the "More"/"Less" buttons). Then navigate / -> /docs -> /docs/design -> the counter via in-page links at a desktop viewport (the collapsed-nav links are zero-size and unclickable, exactly like a real user; proves the site is navigable, not just direct loads), and load /blog + a post (/blog/bevys-five-and-beets-alive) so the beacon runs on a content page -- the pages the client error was reported on.

Mobile layout (no horizontal overflow). At viewports 375x812 and 320x812, for /, /blog, and /blog/bevys-five-and-beets-alive assert document.documentElement.scrollWidth <= document.documentElement.clientWidth + 1, printing the offending elements on failure so the culprit is obvious. Regression guard: a <pre> code block or a wide embed used to blow <main> past the viewport (<main> is a flex item, fixed with min-width: 0), and the header nav overflowed at 320px (fixed with an @media screen app-bar flex-wrap).

The test asserts ZERO collected client errors across every navigation above; extend tests/site_browser.rs if a deploy needs a check it does not cover, rather than scripting around it.

c. screenshot (styles + color schemes)