Fastly Compute Async Request Reliability

Problem

Fire-and-forget HTTP requests from Fastly Compute@Edge using send_async silently fail because the worker process can terminate before the async request reaches the backend. Additionally, backend names referenced in code but not configured in the Fastly dashboard cause silent failures that are easy to miss.

Context / Trigger Conditions

• Background migration, webhook, or notification triggered via req.send_async(backend)
• The PendingRequest returned by send_async is immediately dropped (not awaited)
• The main response is sent to the client, causing the Compute worker to terminate
• Error is swallowed with let _ = ... or match ... { Err(_) => ... }
• Works in local testing (fastly compute serve) but fails in production
• Backend name in code doesn't match any backend in fastly backend list --service-id

Solution

1. Use synchronous send() instead of send_async() for critical operations

// BAD: Fire-and-forget — worker terminates before request completes
match req.send_async(BACKEND) {
    Ok(_pending) => {
        // PendingRequest dropped here — request likely never completes!
        Ok(())
    }
    Err(e) => { /* ... */ }
}

// GOOD: Synchronous send — waits for response
match req.send(BACKEND) {
    Ok(resp) => {
        let status = resp.get_status();
        if status.is_success() {
            eprintln!("[MIGRATE] Success for {}", hash);
        } else {
            eprintln!("[MIGRATE] Backend returned {}", status);
        }
        Ok(())
    }
    Err(e) => {
        eprintln!("[MIGRATE] Failed: {}", e);
        Ok(()) // Don't fail the main request
    }
}

2. When synchronous is too slow, use a caching layer

If a VCL caching layer fronts Compute (service chaining), the extra latency from synchronous send only affects cache misses. Subsequent requests hit the cache. This makes synchronous send acceptable for operations like migration triggers.

3. Always verify backend existence

# List backends configured on the service
fastly backend list --service-id YOUR_SERVICE_ID --version latest

# Add missing backend
fastly backend create --service-id YOUR_SERVICE_ID --version latest --autoclone \
  --name cdn_divine --address cdn.divine.video --port 443 \
  --use-ssl --ssl-sni-hostname cdn.divine.video --override-host cdn.divine.video

Check that every backend name in your Rust code (req.send("backend_name")) has a corresponding entry in the Fastly dashboard. The fastly.toml [local_server.backends] section is for local dev only — it does NOT create production backends.

4. Beware: the URL hostname is cosmetic — the backend name decides routing

In Fastly Compute, req.send(backend_name) / req.send_async(backend_name) ignores the URL's hostname for routing purposes. The backend's dashboard configuration (address, override_host, TLS SNI) determines where the request actually lands. This means two different Cloud Run services that happen to share a backend constant will both deliver to whichever address that one backend is wired to — and the call site that looked correct because its URL said service-A.run.app will actually hit service-B.

This bites hardest after a global rename. Example: a refactor replaces CLOUD_RUN_BACKEND with UPLOAD_SERVICE_BACKEND across the whole file via sed-style search-and-replace. Most call sites were legitimately targeting the upload service, so they still work. But one call site was doing POST https://divine-transcoder-XXXX.run.app/transcode and the rename points it at the upload service backend — the request gets a nginx 404 from the upload service, send_async returns Ok (backend accepted the connection), and the eprintln!("[HLS] Triggered on-demand transcoding for {}", hash) log line is a false positive. The transcoder never sees the request. Videos pile up in an "in progress" state for days.

How to catch this class of bug:

1. When you see a rename of a backend constant, grep for every call site and verify the URL hostname matches what the new backend is configured to route to:

   grep -n "send(\|send_async(" src/*.rs
   grep -n "BACKEND: &str" src/*.rs  # find constant definitions
   

   Any call site where format!("https://{}/...", SOMETHING_ELSE) doesn't match the backend's dashboard address/override_host is a silent misroute.

2. Probe the "wrong" destination directly from the outside with curl using a synthetic payload. If you get a 404 or 405 from nginx/the wrong service, you've found a misroute. Example:

   curl -X POST https://upload.divine.video/transcode -H 'Content-Type: application/json' -d '{"hash":"0"*64}'
   # HTTP 404 <- wrong service, backend misrouted
   

3. If you own more than one Cloud Run service, define one backend per service (transcoder_backend, upload_service, transcriber_backend) and never collapse them just because both are *.run.app. The override_host on each backend pins its destination regardless of URL.

4. Switch fire-and-forget triggers that you care about to synchronous send() at least during investigation — a real non-2xx response from the wrong service makes the bug visible immediately instead of hiding behind send_async.

5. Never silently swallow backend errors for important operations

// BAD: Silent failure
let _ = trigger_migration(&hash, &source);

// GOOD: Log the error even if you don't fail the request
if let Err(e) = trigger_migration(&hash, &source) {
    eprintln!("[MIGRATE] Failed for {}: {}", hash, e);
}

Verification

1. Check Compute logs for the migration/webhook success messages
2. Verify the side effect actually happened (e.g., blob exists in GCS after migration)
3. fastly backend list --service-id ID --version latest shows all expected backends

Example

In Divine Blossom, the on-demand migration from Bunny CDN to GCS never worked because:

1. cdn_divine backend was never configured in the Fastly dashboard (only in fastly.toml)
2. send_async for the Cloud Run migration request was dropped before completion
3. Both errors were silently swallowed with let _ = ...

Fix: Added cdn_divine backend via CLI, changed send_async to send, added logging.

Notes

• fastly.toml [local_server.backends] only configures backends for fastly compute serve
• Production backends must be configured via dashboard or CLI (fastly backend create)
• In Fastly Compute, once the main response body starts streaming to the client, the worker can be terminated at any time — async requests in flight may be cancelled
• If you need true fire-and-forget, consider calling an external queue (Cloud Tasks, PubSub) instead of direct HTTP