Understanding Durable Execution

The worker executor runs WASM agents whose progress survives the loss of the process running them. Everything below is verified against golem-worker-executor/src/ and tests/; when code and guide disagree, the code wins and the guide needs a fix. The scoped AGENTS.md files under golem-worker-executor/ state the mandatory rules; this skill explains the mechanisms.

Deeper material lives in reference/: timelines.md (worked oplog timelines), crash-matrix.md (what recovery does for each crash window), testing-patterns.md (tests that fail under a wrong model), streams.md (durable streams and streaming invocations), tools.md (tool invocations and entity bodies) and retries.md (in-function versus trap-based retries).

Three axioms

1. Replay is deterministic by construction. The guest is a deterministic function of its guest-observable host inputs. The executor records every nondeterministic host interaction in the oplog and feeds the recorded result back during replay. If a replaying guest asks for a different host call than the one recorded next, the executor has a bug (missing recording, wrong reconstruction, wrong completion association). It is never "normal nondeterminism". The host side is not deterministic — completions are tokio futures and finish in varying order — but replay reproduces the guest-visible completion order from the recorded CompletionDelivered markers, so the guest sees the same inputs in the same order. Owner: durable_host/replay_state/claims.rs (a missing Start match while replay is active is a strict divergence error, except for entries deleted by a Jump). The axiom is about replaying the same execution contract; replaying old history against a changed component during an Automatic update may legitimately diverge, and that is reported as FailedUpdate, not tolerated.

2. The resident runtime is disposable. The Wasmtime Store, the worker task, the executor process, sockets, channels and subscriptions can vanish while work is pending. Suspension, eviction, resharding, restart and crash must all be recoverable the same way: throw the instance away, build a new Store, replay the oplog, continue. Lifecycle hints (Suspend, Interrupted, Restart) change status and scheduling policy, not the recovery mechanism. Owner: worker/invocation_loop.rs::run (outer loop: create instance → recover → run → suspend/retry) and durable_host/mod.rs::prepare_instance.

3. Durable agent RPC is exactly-once at the logical callee-invocation level. Dispatch attempts (replay, transport retry, atomic rollback) share a durable key. An executing target persists one invocation and result; read-only cache hits/followers persist neither. Attempts are not executions. Owner: durable_host/wasm_rpc/mod.rs (derive_idempotency_key(begin_index)), worker/mod.rs::enqueue_worker_invocation_with_effect (lookup_invocation_result dedupe before appending PendingAgentInvocation).

┌──────────────┐ host call ┌──────────────────┐ append ┌──────────────┐
│  live guest  │──────────▶│ durable_host fn  │───────▶│    oplog     │
│  (Store #1)  │◀──────────│ Start…End…Deliver│        │ (persistent) │
└──────────────┘  result   └──────────────────┘        └──────┬───────┘
        ✕ Store #1 disappears (suspend / evict / crash / restart)  │
┌──────────────┐ host call ┌──────────────────┐  read   │
│ replay guest │──────────▶│ claim Start,     │◀────────┘
│  (Store #2)  │◀──────────│ resolve End,     │
└──────────────┘ recorded  │ deliver at marker│
                           └────────┬─────────┘
                    cursor exhausted│ + reconstruction validated
                                    ▼
                           publish Live → new work appends again

Durable versus resident state

Durable (authoritative)	Resident (disposable, derived)
Oplog entries and their payloads (golem-common/src/base_model/oplog/mod.rs)	Wasmtime Store, instance, linear memory
PendingAgentInvocation / AgentInvocationStarted / AgentInvocationFinished	Worker.queue: VecDeque<QueuedWorkerInvocation>, event subscriptions
Idempotency keys and recorded invocation results	hydrated_invocation_results cache, read-only cache
Durable-call Start/End/Cancelled and CompletionDelivered/CompletionDiscarded markers	DurableCallSession, ReplayableOneshot, spawned store tasks
PendingUpdate / SuccessfulUpdate / FailedUpdate, Snapshot hints, snapshot blobs	In-flight update decision, loaded snapshot bytes
BeginAtomicRegion/EndAtomicRegion/Jump entries (they mark skipped history; existing entries keep their indices)	Atomic-region logical idempotency counter (rebuilt from those entries during replay)
Agent status folded from the oplog (worker/status.rs)	Status cache / checkpoints (baselines only)
Persisted promises and scheduler actions (WakeupScheduler), RunningWorkers recovery index	Tokio timers, in-memory wake channels

Anything in the right column may be rebuilt from the left column at any time; a design that needs something from the right column to survive a restart is wrong. Sockets and other OS resources are recreated, not preserved (durable_host/sockets, durable_host/http); the application protocol must tolerate reconnect. Durable liveness also needs rediscovery: a timed wait schedules its wakeup as a persisted scheduler action first (durable_host/suspendable_wait.rs, WakeupScheduler::sleep_until), and the shard-keyed RunningWorkers index is updated synchronously so a crash/reshard can enumerate workers with pending work (worker/status_flusher.rs). The status blob cache is an asynchronously flushed baseline only.

Two persistence steps matter for every crash window: append puts an entry in the oplog buffer; commit makes it recoverable (commit_oplog_and_update_state(CommitLevel)). The buffer is a performance trade-off (no storage round trip per append); where an entry must be recoverable before the executor proceeds — accepting remote work, AgentInvocationFinished before notifying waiters — the code commits explicitly. CommitLevel (services/oplog/mod.rs) says how strict that commit is: Always waits for durable storage; DurableOnly does so only for durable agents (PrimaryOplog::commit flushes everything; EphemeralOplog honours the level). Guarantees such as "accepted only after commit" refer to the commit, not the append.

worker/state_actor.rs::commit_and_update_state samples the appended tip before its explicit commit and ignores receipt entries already folded into the published status. Primary/ephemeral threshold flushes and replica waits can commit outside the status actor, so even an empty receipt may hide a committed suffix. Unless the remaining receipt is exactly the contiguous suffix after the last published index, the status actor catches up with status::try_fold_status_from: committed storage is read in bounded chunks, external StreamSession payloads are hydrated, and the result is published once. This avoids retaining an unbounded auto-flushed tail and adds neither oplog entries nor a protocol change. Gap recovery conservatively invalidates authority snapshots after the fold. Ephemeral DurableOnly intentionally remains non-flushing and keeps its no-I/O fast path. This is status reconstruction, not replay tolerance.

Component map

Area	Files	Responsibility
Oplog model	golem-common/src/base_model/oplog/{mod.rs,oplog_macro.rs}	Entry kinds, is_hint(), Start/End pairing rules
Replay cursor	durable_host/replay_state/{mod.rs,cursor.rs,claims.rs}	Single cursor lock, speculative read vs commit, identity claims, Replaying→Settling→Live
Durable calls	durable_host/concurrent/{mod.rs,call.rs,delivery.rs,replay.rs}	DurableCallSession state machine, drop policies, completion delivery tokens
Durability guard	durable_host/durability.rs	begin_durable_function/end_durable_function, DurableFunctionType, in-function retry
Host context	durable_host/mod.rs	prepare_instance, resume_replay, switch_to_live, PendingReplayToLive, idempotency-key derivation, snapshot boundary checks
Tail work	durable_host/tail_work.rs	Keeps store tasks running until no durable work is active before AgentInvocationFinished
RPC	durable_host/wasm_rpc/mod.rs	Key derivation, first dispatch vs MayExist, replay claims, ephemeral phantom identity
Worker	worker/{mod.rs,invocation_loop.rs,lifecycle.rs,instance.rs,status.rs}	Queue persistence, dedupe, interrupt/resume/update decisions, eviction, retry decisions
Cut points	worker/cut_point.rs	Rejects revert/fork cuts that split a paired durable construct
Durable streams	durable_host/{durable_stream/mod.rs,durable_session.rs,stream_session.rs,stream_bus.rs,stream_transport.rs,schema_value_stream.rs}	Producer-oplog stream records, consumer session journal, exactly-once item delivery, protocol terminals
Tool invocations	durable_host/tool/{mod.rs,operation.rs,attachment.rs}, durable_host/entity.rs, worker/{entity_slot.rs,owner_lane.rs,entity_invocation.rs,instance.rs}	Discovery/authorization, owner-oplog boundary for entity bodies, shared replay cursor, lane serialization, attachment memory admission

Worker lifecycle and reconstruction

Worker (worker/mod.rs) receives invocations, persists them, and processes them in order. The run loop (worker/invocation_loop.rs::run) is:

┌─────────────────────────────────────────────────────────────────────┐
│ outer loop (one iteration = one resident instance)                  │
│  create Store + instance ──▶ prepare_instance                       │
│    durable agent:  handle PendingUpdate ▸ try snapshot ▸ resume_replay│
│    ephemeral:      replay first invocation only, append Restart     │
│  ──▶ inner loop: pop durable queue, run invocation, persist Finished │
│  ──▶ instance ends (idle / interrupt / trap / suspend)               │
│  ──▶ RetryDecision: Immediate | Delayed | ReacquirePermits | None | TryStop │
└─────────────────────────────────────────────────────────────────────┘

resume_replay (durable_host/mod.rs) loops get_oplog_entry_agent_invocation_started, replays each recorded invocation in InvocationMode::Replay, and when there is no further AgentInvocationStarted it switches to live. Replay starts from the chosen snapshot baseline (see Snapshots and updates), not necessarily from OplogIndex::INITIAL. Interruption kinds (Worker::set_interrupting): Interrupt stays interrupted, Restart is a simulated crash with automatic recovery, Suspend unloads and resumes on demand; all three end in the same reconstruction path. Eviction (EvictionClass::{LoadedIdle, WarmRunnable}) never unloads a worker that is executing or holds non-durable in-memory work. Ephemeral agents are fail-stop: reconstructed_ephemeral rebuilds only for observation and result lookup, "but the instance must never be started again" (worker/mod.rs, INACTIVE_EPHEMERAL_AGENT_ERROR).

Explicit interruption retires the cached owner and fences its replacement startup. Automatic shard-assignment recovery leaves Interrupted workers stopped, even with queued invocations or updates; an executor restart or shard move is not a request to resume them. If the worker has already unloaded (for example during OOM backoff), the retiring owner must commit an unclaimed terminal interrupt and notify invocation waiters before removal; no Store remains to do it. A terminal interrupt already claimed by the invocation loop is not recorded again, and a completed or failed invocation is not overwritten. Test: tests/scalability.rs::interrupt_during_oom_backoff_is_durable_before_restart.

recover_immediately selects Restart for Running, Suspended and Retrying workers. It never turns a simulated crash of a parked worker into a permanent interruption. If no invocation loop remains, the existing promise, scheduler or permit wakeup starts reconstruction; the queued restart does not fail the invocation waiter or append Interrupted.

Environment and application deletion invalidate component metadata, environment state and agent type caches before awaiting owner retirement. New metadata lookups then observe deletion instead of admitting requests against a retiring cached owner.

Ephemeral response leases delay only normal archival, not Store unloading or explicit retirement. The shared gRPC owner lookup acquires the lease before reading session metadata or accepting work. If normal archival already fenced the owner, lookup joins archival through cache removal, then resolves an observation-only owner from storage. Archive failure or cancellation rejects the lookup; it never grants access to the old poisoned producer or restarts the ephemeral invocation.

Normal archival and explicit interruption share Worker::quiesce_for_owner_retirement under the owner-cleanup lock: stop execution, drain stream retirement and lifecycle/forwarding work, commit, then stop status writers. Only explicit interruption records a pending terminal interrupt. Archival moves the oplog before removing the cached worker. The open oplog generation and its forwarding wrapper may be reused by the next worker, so ordinary retirement does not close their task admission or forwarding. Deletion claims ownership under the same owner-cleanup lock, then drains the resident stream producer before running maintenance on a private producer. Failed maintenance is drained before a retry; only deletion closes the oplog generation permanently.

Cold acquisition reserves one unresolved Worker in ActiveAgents. initialize_with owns one shared attempt independently of request cancellation. finish_construction prepares resolved data privately; failure drains and joins attempt-owned work before returning to Unresolved, without deleting persisted data. Existing waiters receive that attempt's error; later explicit demand retries by reloading persisted identity and pending initialization. Local success publishes the resolved data and Unloaded state. Remote topology recovery and dependent finished-session recovery run together in the post-publication reconciler, preserving the deletion gate: attachment RPCs can acquire mutually referring cold workers on different executors, so awaiting them before publication would create a cycle. Local readiness does not authorize a merely prepared stream attachment. Tests: tests/worker_initialization.rs exercises shared failure, real actor completion, cancellation, existing-only acquisition, and reciprocal cold topologies.

Lifecycle operations acquire the cached or persisted Worker through an existing-only path, so interrupt, delete, resume, update, revert, and plugin changes never create an absent agent. Delete is owned by that worker: concurrent callers share its retained attempt result, a later call retries only unfinished cleanup stages after failure, and successful cleanup retires active-worker and open-oplog cache entries only for the generation being deleted. A stale Arc<Worker> therefore cannot continue deletion against, or evict cache state belonging to, a replacement with the same AgentId.

The bounded unload result and final cleanup completion are separate facts. An unload timeout permanently fails that deletion attempt, while module-owned cleanup continues. A later explicit delete joins the retained completion, or retries a failed filesystem deletion through the owning filesystem generation. Durable storage and cache authority remain fenced until verified cleanup succeeds; old attempt handles retain their original error. Cleanup with no verifiable owning-component repair remains a failure: successful filesystem deletion cannot erase an unverified metering settlement, including ObserverLost during startup rollback.

Create, open, archival, fork-source reads, and deletion share the logical oplog's exclusive cold lifecycle guard. Fork reads persisted source history without constructing an absent source; target construction and rollback are not atomic and are not protected across executor ownership changes. Archival is routed through the existing worker owner. A scheduled archive releases the guard once its transfer is queued, while the oplog sweep holds it until the transfer finishes. No lifecycle lock is taken for individual stream items, oplog reads, or replay steps.

Oplog::stop_and_wait closes admission and joins work associated with the actual open oplog generation, including tasks belonging to older worker shells removed from the active cache. Transport roots are cancelled and their children joined without waiting for client IO. Invocation loops are joined, not cancelled: their final commits, state destruction, and panic cleanup must finish. Already-spawned metadata loads and attachment queries finish independently, so a suspended Store cannot retain their locks; registration occurs once per spawned task, never on cached no-spawn queries. Attachment queries may spawn every time. Worker-state actors register once at construction and drain lifecycle jobs before status jobs and the status flusher, including on ordinary worker drop. Final retirement also joins oplog actors, payload uploads, archive transfers, and monitors. An error is reported only after all owned work finishes. Deletion records that completion separately so an explicit retry can remove storage without reusing a retained stop error; the original attempt keeps its error. A later cold acquisition reloads persisted state rather than inheriting a stopped generation's error.

A failure while creating or preparing the instance is durable health state, not only a resident-worker error. The invocation loop commits Error { kind: Recovery, .. } before unloading and preserves the underlying classification. Infrastructure failures do not advance the agent's semantic retry policy: they remain Retrying without a limit and retry on the next demand (invoke, resume, scheduler activation, or shard reassignment), rather than keeping an executor resident for a scheduled retry. Invalid components, exports, snapshot baselines, replay divergence, and other permanent failures are terminal. An authoritative manual-update snapshot that cannot be loaded is terminal even when the immediate cause is a payload download failure, because recovery has no compatible replay fallback. The ordinary invocation trap path commits Error { kind: Invocation, .. }. The status fold exposes the kind with the failed/retrying status, so metadata and invocation admission agree after unload or reassignment. A later startup appends RecoverySucceeded only when it fully completes prepare_instance and an unresolved recovery error exists. Routine suspend/recovery writes no success marker. Structured metadata reports the underlying Failed/Retrying status and last_error_kind: Recovery; the human CLI table labels terminal recovery failures Unavailable. A queued update still starts a terminally failed worker but does not cosmetically change that durable health status until recovery succeeds.

Resuming an interrupted active durable invocation appends and commits the timestamp-only Resumed hint while the instance lock still proves the worker is unloaded. This happens only after reading the worker's memory requirement succeeds and before changing the resident state to WaitingForPermit. The status fold therefore changes Interrupted back to Running immediately, even if permit admission is still blocked (for example while the interrupted invocation is parked in a pending p3 wait). Resumed does not enqueue an invocation: the existing current_idempotency_key identifies the invocation that reconstruction continues. A Restart does not use this marker and retains its normal Idle/automatic-recovery semantics; ephemeral agents retain their clean fail-stop lifecycle and never append it.

Oplog model

Entries are positional or hints (OplogEntry::is_hint()). Replay consumes positional entries in order and skips hints. Key kinds:

• Start { parent_start_index, observational_owner, request } paired with End { start_index, response } or Cancelled { start_index, partial }. A call is identified by its Start index; End/Cancelled is its terminal. A terminal is visible when it lies below the replay target and outside a Jump/Revert-skipped region (has_visible_terminal / visible_terminal_record, replay_state/cursor.rs). Request-less Start/End pairs are scopes (batched writes, transactions), named <scope:batched-write> or, when the caller supplies a discriminator, <scope:batched-write:DISCRIMINATOR>; a discriminated claim matches only its exact name and never falls back to a plain sibling scope (execute_access_scope_start, concurrent/call.rs).
• AgentInvocationStarted / AgentInvocationFinished bracket one invocation; PendingAgentInvocation (hint) is the durable queue entry.
• CompletionDelivered / CompletionDiscarded (hints, always after their End) record the guest-observation boundary of an accessor completion.
• HostStreamFrame (hint): a frame of a host-owned stream (e.g. a p3 HTTP request body) attached to its owning call by parent_start_index; consumers find frames by scanning, interrupted recordings need no closing entry.
• BeginAtomicRegion / EndAtomicRegion, Jump, Revert, NoOp.
• PendingUpdate, SuccessfulUpdate, FailedUpdate, Snapshot (hint).
• Lifecycle hints: Suspend, Error, RecoverySucceeded, Interrupted, Resumed, Exited, Restart.

Hints are skipped by skip_forward (the physical cursor moves past them, but last_replayed_non_hint_index does not), take part in no Start/terminal pairing, and never satisfy a claim. Jump/Revert do not relocate entries: they mark a region as skipped or deleted, and the atomic-region logical counter is rebuilt from them (see RPC section).

Durable host call lifecycle

Two-step callers use DurableCallSession::begin (returning BegunCall) → BegunCall::resolve → ResolvedCall::{Live, Replay} (concurrent/call.rs). Only the Live branch performs live-only authorization and request preparation before start_live; BegunCall::is_live is private so callers cannot choose a branch before resolution. Replay consumes the recorded result or repairs an admitted incomplete call without re-authorizing. Resolution may finish a guarded replay-tail transition and refresh authority capture. Snapshot calls remain unpersisted; resolving one as Live does not publish Store liveness or lift snapshot restrictions.

Every nondeterministic host function goes through begin_durable_function / end_durable_function (durability.rs). Concurrent (p3 accessor) calls run inside a DurableCallSession (concurrent/call.rs); the serialized p2 path uses persist_durable_function_invocation / read_persisted_durable_function_invocation with the same Start/End shape.

Live (accessor path): append Start eagerly → run the live action → append End (or Cancelled) → hand the result to the guest → append CompletionDelivered (or CompletionDiscarded if the guest dropped the completion unread). The serialized direct path has no markers: its result is delivered when the host function returns. A trap while a call is in flight leaves Start incomplete (abandon_for_trap); a trap never writes Cancelled.

Replay: claim the matching Start (StartClaim, identity + optional request payload match), resolve its terminal through ConcurrentReplayResolver, then classify with classify_replay_resolution (concurrent/call.rs), which is total and shared by every path: