GKE Stall Detection

Use this skill when a controller has gone quiet rather than red: a Gateway that never programs, a rollout that never finishes, a custom resource whose status stopped following its spec, an HTTPRoute nobody accepted. The symptom-driven skills (gke-workload-troubleshooting, gke-reliability) start from a failing pod or a bad event; this one starts from the absence of progress, which those states never show. It is read-only: it lists objects, reads their status and events, resolves referenced names, and proposes nothing beyond naming what is missing.

When to run it

• A user reports something "stuck", "still pending", "never came up", or "not reconciling" and nothing is CrashLooping.
• A Gateway, HTTPRoute, Certificate, Config Connector or Argo-managed object has been in the same state for longer than its controller should need.
• Before declaring a namespace healthy in a reliability review: a silent stall is what that review otherwise misses.

Heuristics and thresholds

/opt/data/scripts/stall_report.py applies four heuristics to every object it reads. Each is gated by an age threshold, so a controller that is merely slow is not reported.

Heuristic	Fires when	Age measured from
generation-lag	metadata.generation is ahead of status.observedGeneration (objects without observedGeneration are skipped)	newest managedFields entry that wrote f:spec
stale-condition	A Progressing, Accepted, Programmed, ResolvedRefs, Ready or Available condition is not True, including conditions nested under status.listeners[] or status.parents[]	the condition's lastTransitionTime
repeating-warnings	A Warning event on the object has a count of at least 3, spans at least the threshold, and was last seen within the threshold	the event's first to last occurrence
dangling-reference	The spec names an object that does not exist: certificateRefs, configMapRef, configMapKeyRef, secretRef, secretKeyRef, parentRefs, backendRefs, and volume configMap/secret/persistentVolumeClaim sources; optional: true references are skipped	newest managedFields entry that wrote f:spec

Thresholds live at the top of the script: DEFAULT_STALL_MINUTES = 15 for every kind, and STALL_MINUTES_BY_KIND for kinds whose API names its own horizon — Deployment is 10, matching the default progressDeadlineSeconds. --threshold-minutes replaces both for one run. Pods in Succeeded or Failed phase are skipped by every heuristic, and so are finished objects: a Job whose Complete or Failed condition is True, and anything whose spec asks for zero replicas (a Deployment's retired ReplicaSets, a scaled-down StatefulSet). A Deployment with spec.paused set is not read for stale conditions. References are resolved from kubectl get <kind> -o name, so Secret contents are never read — and a kind the agent's identity cannot list is never reported missing. On the default read-only permission set the Cluster Agent reaches its cluster as a GSA that cannot list Secrets, so Secret-typed references (certificateRefs, secretRef, secretKeyRef, volume secret) go unchecked there; the signal is a warning: cannot list secrets line on stderr, and the object's own conditions and warning events still carry the case. An identity that can list Secrets (a custom permission set, or the script run from an operator's workstation) gets the dangling-reference row too.

A single run reads one snapshot, so repeating-warnings cannot watch a count move; the span and recency conditions are its stand-in. Run the script twice a few minutes apart when a rising count is the evidence you need.

Running it

gcloud container clusters get-credentials <cluster_name> --region <cluster_location>

# Every namespaced kind in one namespace (kubectl api-resources decides which):
python3 /opt/data/scripts/stall_report.py --namespace <namespace>

# One or more kinds, as kubectl names them:
python3 /opt/data/scripts/stall_report.py --namespace <namespace> --kind gateways.gateway.networking.k8s.io,httproutes.gateway.networking.k8s.io

# A different horizon, or machine-readable output:
python3 /opt/data/scripts/stall_report.py --namespace <namespace> --threshold-minutes 60 --json

The table has one row per object and heuristic — OBJECT, HEURISTIC, DETAIL, STALLED_FOR — and always ends with stalled resources: <count>, the number of distinct objects with at least one row. A healthy namespace prints the header and stalled resources: 0. The scan reads one kind per kubectl get, so a kind whose listing is too large for the shell's output cap or that the identity cannot list is left out on its own: a warning: line on stderr names it and says its objects are missing from the count. A warning: line also names a referent kind the identity cannot list; references to it are then never reported missing. Only a scan that could read no kind at all exits non-zero.

Worked example: a Gateway waiting for a Secret that will never exist

A Gateway whose HTTPS listener names a TLS Secret that certificate automation was never going to create stays Accepted=True forever. The GKE Gateway controller sets Programmed=False Invalid on the Gateway and ResolvedRefs=False InvalidCertificateRef on the listener, emits a Warning SYNC event naming the Secret every few minutes, and never assigns an address; the HTTPRoutes behind it are accepted and never programmed, and nothing goes red. (Rows abridged: the Ready=False NotReady conditions on the Gateway and the listener, and the listener's own Programmed=False, appear too.)

$ python3 /opt/data/scripts/stall_report.py --namespace edge --kind gateways.gateway.networking.k8s.io
OBJECT        HEURISTIC           DETAIL                                                                                          STALLED_FOR
Gateway/edge  dangling-reference  listeners[0].tls.certificateRefs -> Secret/edge-tls not found                                   6h12m
Gateway/edge  stale-condition     Programmed=False Invalid                                                                        6h11m
Gateway/edge  stale-condition     listeners[https] ResolvedRefs=False InvalidCertificateRef                                       6h11m
Gateway/edge  repeating-warnings  SYNC x743: failed to translate Gateway "edge/edge": Error GWCER102: Secret edge/edge-tls not found.  6h10m
stalled resources: 1

The report names the object, the missing Secret and the stall duration from warning events alone. The first row is the one the identity decides: under the default read-only permission set it is absent, stderr carries warning: cannot list secrets in edge; references to secrets are not checked, and the listener's ResolvedRefs condition and the repeating SYNC events are what name the Secret. The HTTPRoutes behind the listener carry no row of their own — they are accepted, and the stall is the Gateway's — so run against the namespace that holds the Gateway, not only the routes. Confirm the reference with kubectl get gateway edge -n edge -o jsonpath='{.spec.listeners[*].tls.certificateRefs}' and check whether the Secret is expected from automation (a Certificate or ManagedCertificate in the namespace, itself possibly a row in the same report) or was simply never created.

The same four heuristics apply without Gateway-specific code: a Deployment whose pods cannot start because envFrom names a missing ConfigMap shows dangling-reference on the Deployment and stale-condition Ready=False on its Pods; a Config Connector or other custom resource whose controller is wedged shows generation-lag.

Recording the finding

You are a Cluster Agent under a strict read-only boundary: no patches applied, no Pull Requests opened, nothing passed back through the chat reply. Complete the kanban task you were spawned on, following the shape in gke-workload-troubleshooting:

kanban_complete(
  result="<which objects are stalled, for how long, on which heuristic, and what each is waiting for (the missing Secret, the unmet condition), grounded in the report rows and the confirming kubectl reads; or the statement that the namespace has no stalled resources>",
  summary="<one line: N stalled resources in <namespace>, or none>",
  metadata={
    "root_cause": "...",
    "evidence": ["<report rows>", "<quoted event or condition>"],
    "proposed_patch": "<YAML creating or correcting the missing reference, when one is identifiable; otherwise omit>"
  }
)

A report with no rows is a result, not a failure to find something: say the namespace has no stalled resources and what threshold that was measured at.