Splunk Agent Observability Setup

Purpose and routing

Use this skill for the post-rebrand Splunk Agent Observability product: an observability, evaluation, and guardrail platform for generative AI and agentic applications. It has separate SaaS and customer-managed on-premises deployment contracts. It is a distinct product workflow from:

• Splunk AI Agent Monitoring in Splunk Observability Cloud APM, which the current product docs mark for deprecation. Use splunk-observability-ai-agent-monitoring-setup only when the user explicitly needs that legacy experience or its migration path.
• Legacy Galileo tenants. Route an existing Galileo application lifecycle or explicitly identified pre-rebrand API contract to galileo-platform-setup. Confirm the actual tenant/product contract with the customer or Splunk; do not treat the August 7, 2026 on-premises announcement date as a tenant migration or contract cutoff.
• Galileo on-premises deployment or legacy packaged components. Route to the applicable galileo-on-prem-* deployment skill only when the target is the legacy product, not Splunk Agent Observability on-premises.
• Splunk Observability Cloud AI telemetry/APM collection or AI infrastructure monitoring. Delegate those components to splunk-observability-ai-agent-monitoring-setup or the named collector, dashboard, detector, and Cisco integration child skills; do not confuse those APM destinations with the Agent Observability application itself.

When the requested product epoch is unclear, establish whether the tenant is legacy Galileo, Splunk Agent Observability SaaS, or Splunk Agent Observability on-premises from the tenant URL and product UI. Do not infer the tenant's onboarding date from the current date. Route unknown or boundary cases to a read-only compatibility review until the customer confirms the product and deployment.

When to Activate

Use this skill when the request names Splunk Agent Observability or asks to instrument, evaluate, secure, investigate, or administer its SaaS or on-premises application. Use the routing rules above for adjacent products.

Prerequisites

Identify the target deployment and project/Agent Stream, obtain authorized product access, and inspect the user's application repository if code changes are requested. Credentials must be available through protected local files, a secret manager, or an approved product credential flow; request paths and non-secret configuration values only.

Workflow Overview

┌──────────┐   ┌──────────────┐   ┌────────────────┐   ┌──────────┐
│ Discover │ → │ Plan & stage │ → │ Apply requested│ → │ Validate │
│ product  │   │ exact scope  │   │ action         │   │ evidence │
└──────────┘   └──────────────┘   └────────────────┘   └──────────┘

The bundled setup.sh is an offline task planner and validate.sh checks the local coverage contract. references/product-feature-matrix.json tracks product topics; references/object-action-matrix.json tracks distinct managed objects, policies, settings, or read-only outputs and their create/update/delete paths. Matrix direct_apply means the agent carries out a documented code, MCP, API, or UI action when that surface and authorization are available; it does not mean setup.sh changes a tenant. A public API reference is not proof that the route is enabled in a particular SaaS or on-premises tenant.

Operating contract

This is an action-oriented workflow. Inspect the user's repository, deployment, connected MCP tools, and target entitlement; perform the requested setup or configuration using the strongest documented interface available; and validate the result. Do not stop after giving generic instructions when a supported MCP, SDK, API, local file, or child-skill action is available.

1. Discover before acting. Read reference.md, references/product-feature-matrix.json, and references/object-action-matrix.json. Inspect the current official docs and release notes for the requested feature. For this fast-moving product, verify version and deployment-specific availability instead of reusing old Galileo or AI Agent Monitoring instructions.
2. Identify the deployment. Record SaaS versus on-premises, exact console/API base URL or Observability Cloud realm, product/release entitlement, and target project/Agent Stream. For SaaS, distinguish Free Edition from a separately entitled subscription. Free Edition is documented as available without a fixed trial and supports up to 15 hosts; paid access/entitlement steps can require the Splunk account team. Do not assume every SaaS feature is enabled for every organization. The on-premises release is announced, but the public docs do not provide a complete install/upgrade runbook; use a support-assisted deployment handoff unless current official release-specific instructions and an approved customer procedure are available.
3. Inspect available tools and permissions. Prefer an already connected, organization-approved Splunk Agent Observability MCP server for supported account actions. Treat its documented server as under construction until its current docs declare general availability. Inspect its actual tool list and schemas before acting; do not infer tool names, API routes, or CRUD support. Use the official splunk-ao SDK for application instrumentation and documented APIs for management operations. Use UI runbooks only where no documented machine interface exists.
4. Plan and stage. Inventory what exists first. Reuse the requested project, Agent Stream, evaluators, and integrations when possible. Stage application code/configuration changes for review. List each live create/update/delete, data export, integration credential, evaluator cost, and privacy consequence. For each requested object or policy, select its --object plan, read the linked source and live schema/UI, and record exact create, update, bind, verify, and cleanup operations. A not-documented entry is a gap to resolve from the actual tenant interface, not permission to invent an endpoint.
5. Apply the requested scope. Carry out read-only checks and specifically requested, reversible setup actions. A render_runbook matrix status means the interface must be checked first; when the documented interface is available and the task is authorized, perform the action and validate it. Before deleting data/resources, sharing access, changing production controls, enabling broad content capture, or generating material billable evaluation traffic, show the exact target and effect when those actions are not already authorized in the user's request. Do not use approval for one operation to authorize unrelated changes.
6. Validate with evidence. Send a small synthetic or approved test trace without real customer secrets or personal data; confirm it appears in the intended project and Agent Stream; check expected span/session/message relationships, metadata, evaluator results, experiment state, and controls according to the requested scope. Capture status, identifiers, links, and sanitized errors; never copy prompt/response bodies or credentials into the report unless explicitly requested and approved. For explicitly requested live end-to-end validation of SaaS telemetry and Agent Control in Kubernetes, follow references/live-validation-runbook.md; it covers secret staging, the deterministic allow/block proof, and exact-scope cleanup.
7. Close the loop. Report completed actions, created/changed identifiers, validation evidence, costs/limits relevant to the run, and remaining UI, account-team, or deployment-owner handoffs. Mark incomplete features as gaps.

Secret and data handling

• Never ask for or display API keys, ingest tokens, LLM provider keys, SSO secrets, passwords, or client secrets in conversation.
• Never put a literal secret in a command argument, shell history, generated example, source control, or shell command's environment-variable prefix. The SDK does require runtime token variables; load them from a protected, gitignored .env file or secret manager without printing values, and verify the exact variable name against the installed SDK and current deployment guide. Use approved secret files or the product's secure credential entry. Keep key material out of MCP server configuration files committed to a repository; prefer the IDE's secret/input facility or its documented environment-variable reference.
• Treat user prompts, system prompts, tool inputs/outputs, traces, evaluation data, datasets, and multimodal content as potentially sensitive. Collect the minimum fields needed. Mask, anonymize, or exclude PII before logging as the official FAQ directs. Confirm before enabling content capture or exporting data. Explain retention, access, sampling, and any destination change from current documentation; do not claim redaction unless the selected component documents and validates it.
• Do not send synthetic test traces to a production project without the user's direction. Do not call paid LLMs/evaluators or create large datasets without including expected usage in the plan.

Supported task families

Use the matrix in references/product-feature-matrix.json as the source of truth for topic coverage and references/object-action-matrix.json for object actions and documented gaps. It includes, at minimum:

• entitlement, edition, SaaS/on-prem deployment, release, console/API URL, and first-run readiness;
• API key/ingest authentication, user/role/capability and project-sharing readiness, SSO handoffs, and organization/project boundaries;
• Python SDK instrumentation, supported framework/model integrations (including NVIDIA NIM), custom logging, MCP tool-call spans, Agent Streams, sessions, traces/spans, metadata/tags, multimodal content, distributed tracing, saved views, search/export, and data maintenance;
• built-in, LLM-as-a-judge, Luna, custom code, composite, and local experiment evaluators; assignment, sampling/feedback/autotune behavior, test cases, Luna Studio training/deployment, and evaluation-cost review;
• projects, datasets, prompts, runs, experiment groups/comparison, and code, playground, or unit-test experiment paths;
• Agent Control policy/runtime setup and monitoring, annotations and queues, Signals, on-premises AI Assistant, Tokenomics, model pricing, integration costs, alerts, rankings, Trends dashboards, and Run Insights where documented;
• MCP server setup and current maturity, supported API/SDK surfaces, SaaS billing and limits, Cisco Cloud Control entry points, release notes, and troubleshooting/error catalog.

Deployment-specific boundaries include the current SaaS feature differences: SaaS has limited preset Luna evaluator support (Prompt Injection, Toxicity, Sexism, and PII only), does not offer Luna fine-tuning, custom code evaluators, or AI Assistant, and does not surface Agent Observability inputs/outputs in APM pages or Observability Cloud global search. Recheck the official comparison before each plan because this product is changing quickly.

Use the first-party Splunk Agent Observability MCP server for its documented capabilities when present: dataset creation/status, prompt templates, experiment guidance/actions, Signals, integrations, and documentation lookup. This server is currently documented as under construction. Check its live tools and current documentation before invoking it; use feature-specific SDK/API or UI paths for unsupported operations. Its presence does not mean it can administer every product feature.

Agent Control's managed guardrail object is a control. Its policy consists of scope (step type/name and Pre/Post stage), a selector and evaluator condition (including and/or/not combinations where supported), and a deny, steer, or observe action. Inventory the exact policy type and action exposed by the tenant before creation. Create the global control, bind it to the intended Agent Stream, and test both a safe allowed case and the expected policy action. Binding creates an independent stream clone; later global edits do not update it. The generic Agent Control server reference documents create, update, delete, attach, and detach endpoints, but its default local-server routes and auth are not automatically the AO SaaS contract. Verify the AO deployment's live route and clone behavior before using those endpoints for update or cleanup. AO SaaS capabilities name separate control CRUD, binding, and runtime permissions. The runtime SDK can register an Agent Control agent name. For disposable work, check whether the target tenant offers a supported delete-agent action before registering a new name; deleting an AO project alone may leave that agent record behind.

Quick start

Resolve paths from this skill's directory. For a scoped local plan, run bash scripts/setup.sh --plan --feature first-trace-python-sdk --deployment saas from that directory and select the relevant feature ID for the actual request. The plan lists the official source and validation evidence; the agent must then carry out the requested action through the available product or application interface. For an exact resource action, run for example bash scripts/setup.sh --plan --object agent-control-global-policy --operation create --deployment saas. If an operation is marked not-documented, check the live tenant and report the gap if no supported action exists; do not claim complete CRUD from topic count.

Check whether the product MCP server is already configured and inspect its available tools without exposing credentials. If the application is in this workspace, inspect its dependency manager, framework, and tests before editing. Follow the official first-trace quickstart for the exact current SDK calls and environment names. Do not copy sample secrets into the repository; use file-backed local settings and .gitignore.

Delegation

Keep this skill as the owner of Splunk Agent Observability product orchestration and final validation. Delegate only a clearly bounded component when needed:

Work	Canonical skill
Legacy Splunk AI Agent Monitoring or OTel AI telemetry	splunk-observability-ai-agent-monitoring-setup
Splunk Observability Cloud AI Infrastructure Monitoring	splunk-observability-ai-agent-monitoring-setup and splunk-observability-otel-collector-setup
Splunk OTel Collector installation/configuration	splunk-observability-otel-collector-setup
Splunk Observability dashboards	splunk-observability-dashboard-builder
Observability Cloud detectors, native alerts, or operations	splunk-observability-native-ops
Cisco AI Defense integration	cisco-security-cloud-setup or the specific current Cisco AI Defense skill in the catalog
Legacy Galileo tenant lifecycle	galileo-platform-setup
Legacy Galileo Agent Control only	galileo-agent-control-setup

Do not delegate new Splunk Agent Observability objects to a legacy Galileo script or use a legacy Galileo token/API endpoint against the new product.

Examples

• “Instrument this Python agent with Splunk Agent Observability and verify one safe trace in the staging Agent Stream.”
• “Create a test dataset, run the evaluator experiment against this prompt, and summarize failures without exposing the trace content.”
• “Audit which evaluators are enabled on this Agent Stream and recommend safe production thresholds.”
• “Review Signals and Tokenomics for this team; don't change any live controls.”
• “Help me enable Agent Control for prompt injection and PII, then validate its behavior in staging before we consider production.”

Troubleshooting

Use bash scripts/validate.sh --help from this skill's directory for local, offline repository validation. For a live deployment, validate the application behavior and target state with the actual approved interface and sanitized test data. A valid repository matrix does not prove tenant entitlement or live data ingest.

Symptom	Check
Access denied or missing product menu	Verify edition/entitlement, organization, user role, resource collaborator role, API key capabilities, and SaaS role-assignment propagation.
No traces or wrong project	Verify deployment-specific SDK configuration, console URL or SaaS realm/token, project and Agent Stream selection, network access, and SDK error output.
Trace appears but evaluators are empty	Verify evaluator assignment/enabled state, supported node types, provider integration, access, and that the input data needed by the evaluator is present.
Evaluator results vary or cost unexpectedly	Check model/provider, evaluator type, sampling, run size, Luna availability, and current billing docs before rerunning.
On-premises feature or endpoint mismatch	Verify exact product release and deployment docs; do not copy SaaS realms/endpoints into self-managed API calls.
MCP setup or action fails	Reinspect current MCP docs, server maturity, endpoint and API key scope; use the supported SDK/API/UI fallback and record the unsupported action.

References

reference.md links the current official documentation by task family and records product-routing and feature-availability boundaries. The feature matrix is the tracked topic coverage contract and feeds the repository product-feature coverage audit. The object matrix is the action contract. Review both whenever a Splunk release or Agent Observability doc changes.