Splunk Observability Isovalent Integration

Prerequisites

Tool or access	Purpose	Verify
Bash and Python 3	Run bundled setup and validation helpers	bash --version && python3 --version
Required product/platform access	Inspect or configure the selected target	Complete the documented preflight
Credential files for live modes	Keep secrets out of chat	Verify paths only

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Wiring Cilium, Tetragon, or Hubble metrics into Splunk Observability Cloud, shipping Tetragon logs to Splunk Platform, or validating Isovalent telemetry after platform install.
• Preview and review the splunk observability isovalent integration workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-isovalent-integration/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-isovalent-integration/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Issue	Cause	Resolution
Preflight fails	A required tool or access path is missing	Resolve it before rendering or applying
Rendered assets are incomplete	Required non-secret inputs are absent	Complete intake and render again
Apply is blocked	Review, credentials, or explicit acceptance is missing	Use the documented handoff
Validation is incomplete	Live evidence is unavailable	Record the gap and keep completion open

This skill wires an installed Isovalent stack to Splunk Observability Cloud and Splunk Platform. It depends on the platform install completed by cisco-isovalent-platform-setup. Run that first, then this.

What it renders

• splunk-otel-overlay/values.overlay.yaml — Splunk OTel collector agent.config overlay with seven Prometheus scrape jobs and the filter/includemetrics allow-list. Designed to merge with the base values produced by splunk-observability-otel-collector-setup via yq deep-merge.
• Splunk Platform logs path (DEFAULT — file-based via OTel filelog receiver):
  • agent.extraVolumes + agent.extraVolumeMounts hostPath mount of /var/run/cilium/tetragon.
  • logsCollection.extraFileLogs.filelog/tetragon block with sourcetype cisco:isovalent, index cisco_isovalent.
  • splunkPlatform.logsEnabled: true.
  • Coordinates with cisco-isovalent-platform-setup's default Tetragon export.mode: file + exportDirectory: /var/run/cilium/tetragon.
• Alternative paths (behind explicit flags):
  • --export-mode stdout — Tetragon stdout + container log collection (no hostPath mount; useful when SCC/PSP blocks).
  • --legacy-fluentd-hec — fluentd splunk_hec block. DEPRECATED (fluent-plugin-splunk-hec archived 2025-06-24).
• dashboards/cilium-by-isovalent.json and dashboards/hubble-by-isovalent.json — token-scrubbed re-exports (sourced from the Isovalent_Splunk_o11y reference repo's examples/*.json only after scripts/scrub-tokens.py confirms zero accessToken material).
• detectors/*.yaml — starter detectors for cilium_*, hubble_*, tetragon_* series.
• scripts/setup.sh, render_assets.py, validate.sh, handoff-base-collector.sh, handoff-hec-token.sh, handoff-cisco-security-cloud.sh, handoff-dashboards.sh, handoff-detectors.sh, scrub-tokens.py.
• metadata.json.

Safety Rules

• Never ask for the Splunk Observability ingest token or Splunk Platform HEC token in conversation.
• File-backed token flags only:
  • --o11y-token-file (Splunk Observability Org access token; passed through to base collector).
  • --platform-hec-token-file (Splunk Platform HEC token; or --render-platform-hec-helper to delegate to splunk-hec-service-setup).
• Reject direct token flags (--access-token, --token, --bearer-token, --api-token, --o11y-token, --sf-token, --platform-hec-token, --hec-token).
• Token files must be chmod 600; the wrapper aborts otherwise (override with --allow-loose-token-perms, emits WARN). The actual apply happens via the rendered handoff-*.sh scripts, which inherit the same chmod 600 enforcement when they re-invoke the base collector / HEC / dashboard / native-ops setups.
• The renderer scrubs every dashboard JSON it ships against an accessToken regex before writing. Render aborts if the source JSON contains plaintext token material.

Primary Workflow

1. Confirm the Isovalent stack is installed (run cisco-isovalent-platform-setup first).

2. Render:

   bash skills/splunk-observability-isovalent-integration/scripts/setup.sh \
     --render \
     --validate \
     --realm us0 \
     --cluster-name lab-cluster \
     --output-dir splunk-observability-isovalent-rendered
   

3. Review splunk-observability-isovalent-rendered/:

   • splunk-otel-overlay/values.overlay.yaml — the overlay; merge into the base collector values.
   • dashboards/*.json — token-scrubbed dashboard exports.
   • detectors/*.yaml — starter detectors.
   • scripts/handoff-*.sh — hand-off drivers for the four downstream skills.

4. Apply the overlay onto the existing Splunk OTel collector helm release (recommended one-shot path). Merges this overlay onto current release values via yq, normalizes legacy otlphttp exporter names to otlp_http, preserves the live OBI eBPF ConfigMap data when OBI is installed, runs helm upgrade --atomic --force-conflicts, and rolls the agent + cluster-receiver. The helper preserves existing gateway/operator topology unless collector.disable_gateway or collector.disable_operator is set, auto-discovers the collector namespace, and pins the installed chart version unless collector.namespace or collector.chart_version is set in the spec. Refuses without --accept-k8s-apply, refuses if no Cilium/Tetragon install is detected, and prints the active kube-context first:

   bash skills/splunk-observability-isovalent-integration/scripts/setup.sh \
     --apply --accept-k8s-apply
   

   --apply --accept-k8s-apply --dry-run runs helm upgrade --dry-run without mutating the cluster.

   Then provision HEC and route Tetragon logs to Splunk Platform via Cisco Security Cloud:

   bash splunk-observability-isovalent-rendered/scripts/handoff-hec-token.sh
   bash splunk-observability-isovalent-rendered/scripts/handoff-cisco-security-cloud.sh
   bash splunk-observability-isovalent-rendered/scripts/handoff-dashboards.sh
   bash splunk-observability-isovalent-rendered/scripts/handoff-detectors.sh
   

Hand-offs

• Splunk OTel Collector base install: splunk-observability-otel-collector-setup. Render the base values, then merge our overlay via yq deep-merge.
• Splunk Platform HEC token: splunk-hec-service-setup.
• Splunk Platform Tetragon log ingestion: cisco-security-cloud-setup with PRODUCT=isovalent (sourcetype cisco:isovalent:processExec, index cisco_isovalent); confirmed in skills/cisco-security-cloud-setup/products.json lines 200-219.
• Dashboards: splunk-observability-dashboard-builder.
• Detectors: splunk-observability-native-ops.

Out of scope

• Cilium / Tetragon / Hubble install lifecycle — handled by cisco-isovalent-platform-setup.
• Splunk OTel collector base install — handled by splunk-observability-otel-collector-setup.

Validation

bash skills/splunk-observability-isovalent-integration/scripts/validate.sh

Static checks: overlay shape, token-scrub assertion, dashboard JSON validity, sourcetype/index match. With --live:

• helm status for the OTel collector release.
• Pod-IP scrape probes for the seven Prometheus ports (uses kubectl get --raw for Tetragon, NOT kubectl exec).
• Optional SignalFlow probe for cilium_*, hubble_*, tetragon_* series presence.
• Optional Splunk Platform search check: index=cisco_isovalent sourcetype=cisco:isovalent returns events.

See reference.md and the references/ annexes for collector-overlay details, Splunk Platform paths, Tetragon hostPath coordination, sourcetype reference, dashboards catalog, and troubleshooting.