Deploy OTEL Observability Stack

Deploy a complete OpenTelemetry observability stack to a Kind cluster for testing ToolHives telemetry capabilities.

Steps

1. Verify Prerequisites

Check that required tools are installed:

echo "Checking prerequisites..."
command -v kind >/dev/null 2>&1 || { echo "ERROR: kind is not installed"; exit 1; }
command -v helm >/dev/null 2>&1 || { echo "ERROR: helm is not installed"; exit 1; }
command -v kubectl >/dev/null 2>&1 || { echo "ERROR: kubectl is not installed"; exit 1; }
echo "All prerequisites met."

2. Create Kind Cluster

Create the Kind cluster if it doesn't exist:

CLUSTER_NAME="toolhive"

if kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$"; then
  echo "Kind cluster '${CLUSTER_NAME}' already exists"
else
  echo "Creating Kind cluster '${CLUSTER_NAME}'..."
  kind create cluster --name ${CLUSTER_NAME}
fi

# Export kubeconfig
kind get kubeconfig --name ${CLUSTER_NAME} > kconfig.yaml
echo "Kubeconfig written to kconfig.yaml"

3. Add Helm Repositories

echo "Adding Helm repositories..."
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
echo "Helm repositories updated."

4. Install Prometheus/Grafana Stack

echo "Installing kube-prometheus-stack..."
helm upgrade -i kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -f examples/otel/prometheus-stack-values.yaml \
  -n monitoring --create-namespace \
  --kubeconfig kconfig.yaml \
  --wait --timeout 5m

echo "Prometheus/Grafana stack installed."

5. Install Tempo for Distributed Tracing

echo "Installing Grafana Tempo..."
helm upgrade -i tempo grafana/tempo \
  -f examples/otel/tempo-values.yaml \
  -n monitoring \
  --kubeconfig kconfig.yaml \
  --wait --timeout 3m

echo "Grafana Tempo installed."

6. Install OpenTelemetry Collector

echo "Installing OpenTelemetry Collector..."
helm upgrade -i otel-collector open-telemetry/opentelemetry-collector \
  -f examples/otel/otel-values.yaml \
  -n monitoring \
  --kubeconfig kconfig.yaml \
  --wait --timeout 3m

echo "OpenTelemetry Collector installed."

7. Verify Deployment

echo "Verifying deployment..."
kubectl get pods -n monitoring --kubeconfig kconfig.yaml

8. Display Access Instructions

cat <<'EOF'

=== OTEL Stack Deployment Complete ===

To access the UIs, run these port-forward commands:

  # Grafana (admin / admin)
  kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:3000 --kubeconfig kconfig.yaml

  # Prometheus
  kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 --kubeconfig kconfig.yaml

EOF

Troubleshooting

If Helm installations fail due to incompatible values, it may be because the Helm charts have been updated and our values.yaml files are no longer compatible.

Chart Documentation:

• OpenTelemetry Collector: https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector
• Prometheus Stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
• Tempo: https://github.com/grafana/helm-charts/tree/main/charts/tempo

If you encounter issues:

1. Check the chart's values.yaml for schema changes in the versions of the Charts we are using
2. Compare with our values files in examples/otel/
3. Create an issue at: https://github.com/stacklok/toolhive/issues describing what the issue is and recommend a fix

What This Deploys

Component	Description
Prometheus	Metrics storage, scrapes OTEL collector on port 8889
Grafana	Visualization dashboards (admin/admin)
Tempo	Distributed tracing backend, receives traces from OTEL Collector
OTEL Collector	Receives OTLP metrics/traces, exports to Prometheus and Tempo

Cleanup

To remove everything:

task kind-destroy

Or manually:

kind delete cluster --name toolhive
rm -f kconfig.yaml