Splunk Observability Cloud — AWS Lambda APM Setup

Prerequisites

Tool or access	Purpose	Verify
Bash and Python 3	Run bundled setup and validation helpers	bash --version && python3 --version
Required product/platform access	Inspect or configure the selected target	Complete the documented preflight
Credential files for live modes	Keep secrets out of chat	Verify paths only

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Instrument Lambda functions for APM/tracing, attach the Splunk OTel Lambda layer, wire SPLUNK_ACCESS_TOKEN safely, set up Lambda APM dashboards or detectors, or migrate from Datadog, New Relic, or ADOT to Splunk OTel.
• Preview and review the splunk observability aws lambda apm setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Issue	Cause	Resolution
Preflight fails	A required tool or access path is missing	Resolve it before rendering or applying
Rendered assets are incomplete	Required non-secret inputs are absent	Complete intake and render again
Apply is blocked	Review, credentials, or explicit acceptance is missing	Use the documented handoff
Validation is incomplete	Live evidence is unavailable	Record the gap and keep completion open

Render-first skill that owns the complete lifecycle of Splunk OTel Lambda layer attachment. The workflow is render-first by default. AWS Lambda function changes only happen when the operator explicitly runs --apply after reviewing the rendered plan.

The skill is the live fulfillment of the handoffs.lambda_apm stub emitted by splunk-observability-aws-integration (previously marked "future companion").

Coverage Model

Section	Coverage status
Beta acknowledgement	api_validate
Target function validation	api_validate
Layer ARN resolution (snapshot)	api_validate
Exec-wrapper wiring	api_validate
Token delivery (Secrets Manager / SSM)	api_apply
X-Ray coexistence flag	api_validate
IAM ingest-egress policy (local collector disabled)	api_validate / not_applicable
Terraform variant	handoff
CloudFormation variant	handoff
SAM template variant	handoff
CDK TypeScript + Python variant	handoff
SAR advisory	handoff
Container-image Dockerfile snippet	handoff (image targets only)
AWS CLI apply plan	api_apply
Span attribute validation	api_validate
Vendor-coexistence check	api_validate
SnapStart guidance (Java)	api_validate
Lambda@Edge	not_applicable (refused)
Provisioned Concurrency (SAM/CFN emit)	api_validate
splunk-lambda-metrics extension layer	api_validate (opt-in)
GovCloud / China	not_applicable (refused)
Cross-skill handoffs	handoff / not_applicable

Safety Rules

• Never ask for the Splunk O11y access token in conversation.
• Never pass SPLUNK_ACCESS_TOKEN as a command-line value. Use --token-file. The apply helper rotates the selected AWS secret, fetches it into a private temporary file, and merges it into the Lambda environment without exposing it in rendered files, shell history, or process arguments.
• Token files must be regular, single-hardlink files containing one non-empty printable-ASCII value with at most one trailing LF or CRLF, and no larger than the backend limit (4 KiB for default-tier SSM Parameter Store; 64 KiB for Secrets Manager) with mode 600. The generated writer opens the file once with O_NOFOLLOW, verifies stable inode, metadata, and content fingerprints, and builds the AWS payload from that descriptor. Use write_secret_file.sh to create one without shell-history exposure.
• --allow-loose-token-perms is an explicit development-only escape that relaxes only the source mode check. The setup wrapper propagates it to the generated writer; regular-file, non-symlink, single-hardlink, size, UTF-8, and fingerprint checks remain mandatory.
• Reject direct-secret flags: --token, --access-token, --api-token, --o11y-token, --sf-token, --password.
• signalfx/splunk-otel-lambda is BETA. Gate all operations behind accept_beta: true in the spec or --accept-beta on the CLI.
• GovCloud (us-gov-*) and China (cn-*) regions have no published layers. The renderer fails with not_applicable coverage; do not fabricate ARNs.

Five-mode UX

Mode	Flag	Purpose
render	--render (default)	Produces the plan tree. No AWS calls.
apply	--apply [SECTIONS]	Runs the rendered aws-cli plan. Sections: layer,env,iam,validation.
validate	--validate [--live]	Static checks plus an optional unauthenticated ingest-endpoint reachability probe; --live does not validate configured state or telemetry arrival.
doctor	--doctor	Vendor conflict, ADOT, X-Ray, snapshot-freshness checks.
quickstart	--quickstart	Render + print exact --apply command.

--quickstart-from-live --target FUNCTION writes only an allowlisted subset of the Lambda configuration to a mode-600 snapshot. It deliberately excludes Environment (and therefore environment secrets), revision IDs, and other unneeded response fields.

Primary Workflow

1. Copy and edit the spec

cp skills/splunk-observability-aws-lambda-apm-setup/template.example my-lambda-spec.yaml
# fill in realm, targets (function names, regions, runtimes, arches)

2. Render

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --render \
  --spec my-lambda-spec.yaml \
  --realm us1

3. Review the plan

splunk-observability-aws-lambda-apm-rendered/
  01-overview.md               # plan summary + apply command
  02-targets.md                # per-function targets + resolved layer ARNs
  03-layers.md                 # layer attachment plan
  04-env.md                    # environment variable plan
  05-validation.md             # validation steps
  aws-cli/apply-plan.sh        # guarded AWS CLI apply helper
  terraform/main.tf            # Terraform resource snippets
  cloudformation/snippets.yaml # CloudFormation snippets
  sam/template.yaml            # AWS SAM template snippet
  cdk/lambda-apm-stack.ts      # CDK TypeScript snippet
  cdk/lambda_apm_stack.py      # CDK Python snippet
  sar/README.md                # SAR advisory (Splunk does not publish SAR)
  container-image/             # Dockerfile.<runtime> (image-package targets only)
  iam/                         # IAM policy (local collector disabled only)
  scripts/write-splunk-token.sh # one-time token write to secret backend
  scripts/handoffs.sh           # cross-skill handoff drivers
  coverage-report.json          # per-section coverage status

4. Write the access token to the secret backend (once)

# Prepare the token file (never put the value in shell history):
bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token

# Write token to Secrets Manager (or SSM if secret_backend: ssm):
TOKEN_FILE=/tmp/splunk_o11y_token \
  bash splunk-observability-aws-lambda-apm-rendered/scripts/write-splunk-token.sh

# Keep the local file until the apply step below completes, then delete it.

5. Apply

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --apply \
  --spec my-lambda-spec.yaml \
  --realm us1 \
  --token-file /tmp/splunk_o11y_token

rm /tmp/splunk_o11y_token

Or apply only specific sections:

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --apply layer,env \
  --spec my-lambda-spec.yaml

Quickstart (single command)

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --quickstart \
  --spec my-lambda-spec.yaml \
  --realm us1 \
  --accept-beta

The quickstart renders the plan and prints the exact --apply command to run next. No AWS calls happen until you explicitly run --apply.

GitOps mode (Terraform / CloudFormation only)

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --render \
  --gitops-mode \
  --spec my-lambda-spec.yaml

--gitops-mode suppresses the aws-cli/ directory and produces only terraform/ and cloudformation/ artifacts.

The Terraform variant necessarily reads the token as plaintext and stores it in Terraform state as part of the Lambda environment. Use only an encrypted, access-controlled remote backend with locking and never commit local state; prefer the CloudFormation dynamic-reference or guarded AWS CLI variant when state cannot be protected to the same standard as the token.

Doctor

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --doctor \
  --target my-function \
  --realm us1

Doctor checks: beta status, snapshot freshness, vendor conflicts (Datadog / New Relic / AppDynamics / Dynatrace), ADOT conflict, X-Ray coexistence flag, and VPC egress reminder when local_collector_enabled: false.

Rollback

# Detach layer only:
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback layer \
  --target my-function

# Remove env vars only:
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback env \
  --target my-function

# Detach IAM ingest-egress policy (only when local_collector_enabled=false):
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback iam \
  --target my-function

# Full rollback (layer → env → iam in sequence):
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback all \
  --target my-function

Each rollback section renders idempotent AWS CLI commands for review; the skill never mutates AWS without operator review.

Hand-offs

• CloudWatch Lambda metrics → splunk-observability-aws-integration
• Lambda APM dashboards → splunk-observability-dashboard-builder
• Cold-start / error / latency detectors → splunk-observability-native-ops
• Lambda log ingestion → splunk-connect-for-otlp-setup
• Gateway OTel Collector path → splunk-observability-otel-collector-setup (set handoffs.gateway_otel_collector: true and local_collector_enabled: false)
• FROM splunk-observability-aws-integration → this skill is the fulfillment of handoffs.lambda_apm: true

Out of Scope

• Lambda log ingestion (handed off to splunk-connect-for-otlp-setup)
• CloudWatch metrics for AWS/Lambda namespace (handed off to splunk-observability-aws-integration)
• AppDynamics serverless instrumentation (handled by splunk-appdynamics-apm-setup)
• AlwaysOn Profiling for Lambda (not officially supported in signalfx/splunk-otel-lambda)
• Go, Ruby, .NET Lambda runtimes (no published layers from publisher 254067382080)
• GovCloud / China regions (no published layers; renderer refuses with not_applicable)
• Lambda@Edge (no env-var injection support in the Lambda@Edge execution model; renderer refuses)

Validation

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/validate.sh \
  --output-dir splunk-observability-aws-lambda-apm-rendered

Static checks: required files, IAM JSON shape, secret-leak scan. With --live: runs an unauthenticated reachability-only probe of the Splunk ingest endpoint. It does not validate Lambda configuration, token authorization, span export, or APM data arrival. Invoke the function and verify its expected span attributes in APM for configured-state acceptance.

See references/splunk-doc-feature-matrix.md for the full feature matrix and limitations.