GPU Live Metric Validation

Purpose

Run GPU metric validation against Kubernetes clusters that are exclusively running the Agent version under test during the selected validation window, then investigate any findings.

Required input

Ask the user which Datadog orgs to validate before running any queries. The supported values match tasks/gpu.py:

• prod (app.datadoghq.com)
• staging (ddstaging.datadoghq.com)

Validation workflow

1. Ask the user which supported orgs to validate.

2. Run the validator for each selected org:

   dda inv gpu.validate-metrics \
     --org <prod-or-staging> \
     --lookback-seconds <window-seconds>
   

   By default, the task derives the Agent image-tag wildcard from the current release branch and its latest release-candidate tag. It passes that wildcard to the validator, which queries datadog.agent.running grouped by kube_cluster_name,image_tag, selects clusters whose nonempty image tags all match the wildcard, and ANDs that cluster selection with the GPU configuration filters.

   Use --agent-version <wildcard> to override the derived version; it is required outside release branches (N.N.x), where derivation fails. Use --metric-filter <filter> only for an additional scope; it is ANDed with the version-derived cluster filter.

3. Record the selected org, Agent version wildcard, lookback window, and validation output before interpreting any findings.

Investigating findings

For follow-up Datadog queries, use dd-auth to select the target. pup consumes the injected DD_API_KEY, DD_APP_KEY, and DD_SITE; do not combine this workflow with pup --org.

1. Start from a failing GPU metric and group it by the smallest useful set of dimensions, normally gpu_uuid, host, and kube_cluster_name. Add the failed tag as a group-by dimension when investigating a tag failure.

   dd-auth --domain <target-domain> -- \
     pup --no-agent metrics query \
     --query='count:gpu.<metric>{<validation-filter>} by {gpu_uuid,host,kube_cluster_name,<failed-tag>}' \
     --from=<validation-window> --to=now
   

2. Identify patterns before drawing conclusions: whether failures are limited to a GPU architecture, device mode, GPU model, host, cluster, workload, or Agent image tag. Confirm the affected cluster's Agent image tag with:

   dd-auth --domain <target-domain> -- \
     pup --no-agent metrics query \
     --query='sum:datadog.agent.running{<cluster-filter>} by {kube_cluster_name,image_tag}' \
     --from=<validation-window> --to=now
   

3. Preserve the exact queries and affected GPU/host/cluster identifiers in the summary.

Candidate investigations by failure type

• Missing metric: compare the affected GPU configuration with the metric's expected support and query gpu.device.total for the same scope to confirm that devices were present.
• Missing required tag: group the failing metric by the required tag and affected GPU/host/cluster. Compare a related GPU metric to determine whether the absence is metric-specific.
• Invalid tag value: group by the invalid tag and affected GPU/host/cluster. Check whether a series has multiple values for the same tag key before treating a comma-separated value as a single emitted tag value.
• Unknown or extra tag: query a workload/container metric such as container.cpu.usage for the same pod and container. Compare its tags with the GPU metric to determine whether the tag originates from the workload.
• If a workload tag points to a Kubernetes resource, inspect the corresponding Kubernetes object and its labels and annotations before assigning a source.

Constraints

• Use an explicit, bounded validation window.
• Use dd-auth to select the target for pup queries.
• Do not validate clusters that reported a different nonempty Agent image tag in the validation window.
• Do not use pup --org with dd-auth.