Microservice API Endpoint Routing 404 Debugging

Problem

API endpoint returns 404 even though the code implementing it has been deployed. This commonly happens in microservice architectures where multiple services share the same hostname and HTTPRoutes/Ingress rules determine which service handles which paths.

Context / Trigger Conditions

• New endpoint returns 404 but code is definitely deployed
• Multiple services (e.g., relay + api) share the same external hostname
• Service logs show "Health server listening" or similar health-only messages
• HTTPRoute or Ingress rules split traffic by path prefix
• Monorepo builds multiple Docker images from same commit

Solution

Step 1: Identify which service actually handles the path

Check HTTPRoute/Ingress rules to see routing:

kubectl get httproute <name> -n <namespace> -o yaml | grep -A 50 "rules:"

Look for path matching rules and which backend service they point to:

• /api/* might go to api-service:8080
• / might go to websocket-service:7777

Step 2: Test the endpoint directly on each service from inside cluster

kubectl run curl-test -n <namespace> --image=curlimages/curl --restart=Never \
  -- curl -v "http://<service-name>:<port>/<endpoint>" \
  && sleep 5 \
  && kubectl logs -n <namespace> curl-test \
  && kubectl delete pod -n <namespace> curl-test

This bypasses external routing and tests directly against each service.

Step 3: Check service logs for endpoint registration

kubectl logs -l app=<service-name> -n <namespace> --tail=50 | grep -i -E "(endpoint|route|api|listening)"

Look for messages like:

• "Health server listening on 0.0.0.0:8080" = health only, no API
• "API server listening" or "Registered /api/..." = serves API endpoints

Step 4: For monorepos, check if matching image exists for correct service

If feature was added to service A but endpoint is served by service B, check if service B has the same commit tag:

gcloud artifacts docker images list <registry>/<service-b> --include-tags --limit=10

Step 5: Deploy the correct service with feature + required env vars

Update the service that actually handles the endpoint path:

• New image tag with the feature
• Any environment variables (API keys, URLs) the feature needs
• Any secrets referenced by the feature

Verification

1. Service logs show endpoint registered: "Recommendations API enabled at /api/users/:pubkey/recommendations"
2. Direct curl from inside cluster returns 200
3. External request through HTTPRoute returns expected data

Example

Architecture:

Hostname: relay.example.com
  /api/* → funnelcake-api:8080 (REST API)
  /      → funnelcake-relay:7777 (WebSocket)

Problem: /api/users/x/recommendations returns 404

Investigation:

1. HTTPRoute shows /api/* → funnelcake-api (not relay)
2. Direct test to relay:8080 returns 404 (health server only)
3. Direct test to api:8080 also returns 404 (old image)
4. Registry shows both api and relay have image tag 3c0696a
5. API was running old tag 487178e

Fix: Deploy funnelcake-api with tag 3c0696a + GORSE_URL + GORSE_API_KEY env vars

Notes

• Services in same namespace can share Kubernetes Secrets
• Health endpoints (/livez, /readyz) are often on separate port from API
• HTTPRoute path matching: more specific paths take precedence
• Always check which service handles which paths before assuming endpoint location