Add llama.cpp

This skill installs llama.cpp, runs llama-server as a local host service, and wires the local endpoint into Deus only when the current checkout already supports the optional llama_cpp provider.

Use this when the user wants a faster or cheaper local text-generation path for experiments, benchmarks, or future backend work.

Slash command: /add-llama-cpp.

Important boundaries:

• This does not replace Ollama for memory embeddings or the default judge. Ollama remains required unless the repo deliberately changes those surfaces.
• This skill is macOS-first for installation and service management. On Linux or Windows, continue only if llama-server is already installed or the user explicitly wants a manual install path.
• If the current checkout does not yet contain the optional Deus-side llama_cpp integration, complete the host install anyway and tell the user the runtime wiring is a separate source task.

Phase 1: Pre-flight

Check current state

command -v llama-server >/dev/null 2>&1 && llama-server --version || echo "llama.cpp not installed"
curl -fsS http://127.0.0.1:8080/health 2>/dev/null || echo "llama-server not responding on 127.0.0.1:8080"
test -f evolution/generative/providers/llama_cpp.py && echo "DEUS_LLAMA_CPP_PROVIDER=true" || echo "DEUS_LLAMA_CPP_PROVIDER=false"
test -f setup/llama-cpp.ts && echo "DEUS_LLAMA_CPP_SETUP=true" || echo "DEUS_LLAMA_CPP_SETUP=false"

Ask scope

AskUserQuestion: Do you want host install only, or host install plus optional Deus wiring if this checkout supports it?

1. Host install only - install llama.cpp, run llama-server, and verify the local endpoint
2. Host install + Deus wiring - also configure repo env vars and run Deus-side verification when the checkout supports it

If the user chooses option 2 but either provider/setup file is missing, say clearly that the host install can proceed now and the checkout wiring remains a separate code task.

Phase 2: Install llama.cpp

macOS

If llama-server is not already installed:

brew install llama.cpp

Verify:

llama-server --version

Linux or Windows

If llama-server is already available in PATH, continue.

If it is missing, stop and tell the user this skill currently automates installation on macOS only. Offer to continue once llama-server is installed manually, or handle platform-specific installation as a separate task.

Phase 3: Configure the Local Service

Create a local env file outside git:

mkdir -p "$HOME/.config/deus" "$HOME/.config/deus/scripts"
cat > "$HOME/.config/deus/llama-cpp.env" <<'EOF'
LLAMA_CPP_BIND_HOST=127.0.0.1
LLAMA_CPP_PORT=8080
LLAMA_CPP_MODEL=ggml-org/gemma-3-1b-it-GGUF:Q4_K_M
LLAMA_CPP_ALIAS=ggml-org/gemma-3-1b-it-GGUF:Q4_K_M
LLAMA_CPP_CTX_SIZE=8192
EOF

Create the launcher script:

cat > "$HOME/.config/deus/scripts/start-llama-cpp.sh" <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

ENV_FILE="$HOME/.config/deus/llama-cpp.env"
if [ -f "$ENV_FILE" ]; then
  set -a
  . "$ENV_FILE"
  set +a
fi

: "${LLAMA_CPP_BIND_HOST:=127.0.0.1}"
: "${LLAMA_CPP_PORT:=8080}"
: "${LLAMA_CPP_MODEL:=ggml-org/gemma-3-1b-it-GGUF:Q4_K_M}"
: "${LLAMA_CPP_ALIAS:=$LLAMA_CPP_MODEL}"
: "${LLAMA_CPP_CTX_SIZE:=8192}"

exec llama-server \
  --host "$LLAMA_CPP_BIND_HOST" \
  --port "$LLAMA_CPP_PORT" \
  -hf "$LLAMA_CPP_MODEL" \
  --alias "$LLAMA_CPP_ALIAS" \
  -c "$LLAMA_CPP_CTX_SIZE" \
  --jinja
EOF
chmod +x "$HOME/.config/deus/scripts/start-llama-cpp.sh"

Phase 4: Run as a Host Service

macOS LaunchAgent

Write the LaunchAgent:

PLIST="$HOME/Library/LaunchAgents/com.deus.llama-cpp.plist"
mkdir -p "$HOME/Library/LaunchAgents" "$HOME/Library/Logs"
cat > "$PLIST" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.deus.llama-cpp</string>
    <key>ProgramArguments</key>
    <array>
      <string>$HOME/.config/deus/scripts/start-llama-cpp.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>$HOME/Library/Logs/deus-llama-cpp.log</string>
    <key>StandardErrorPath</key>
    <string>$HOME/Library/Logs/deus-llama-cpp.error.log</string>
  </dict>
</plist>
EOF
launchctl unload "$PLIST" 2>/dev/null || true
launchctl load "$PLIST"
launchctl kickstart -k "gui/$(id -u)/com.deus.llama-cpp"

Linux or Windows

Do not invent a service wrapper if the platform path is unclear. Prefer a foreground verification run:

"$HOME/.config/deus/scripts/start-llama-cpp.sh"

If the user wants persistent background service management on Linux or Windows, treat that as a follow-up task after the endpoint is verified.

Phase 5: Verify the Local Endpoint

Check health:

curl -fsS http://127.0.0.1:8080/health
curl -fsS http://127.0.0.1:8080/v1/models

Run a chat-completions smoke test:

curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ggml-org/gemma-3-1b-it-GGUF:Q4_K_M",
    "messages": [{"role": "user", "content": "Reply with exactly: OK"}]
  }'

If the first model download takes time, inspect the log instead of assuming failure:

tail -50 "$HOME/Library/Logs/deus-llama-cpp.log" 2>/dev/null || true
tail -50 "$HOME/Library/Logs/deus-llama-cpp.error.log" 2>/dev/null || true

Phase 6: Optional Deus Wiring

Run this phase only if:

• the user chose host install plus Deus wiring, and
• evolution/generative/providers/llama_cpp.py exists in the current checkout, and
• setup/llama-cpp.ts exists in the current checkout.

Rebuild the agent container

If upgrading from a build predating llama.cpp support, rebuild the container so the new backend module is available:

./container/build.sh

Configure repo env vars

Write or update the repo env:

if grep -q '^LLAMA_CPP_BASE_URL=' .env 2>/dev/null; then
  tmpf=$(mktemp) && sed 's#^LLAMA_CPP_BASE_URL=.*#LLAMA_CPP_BASE_URL=http://127.0.0.1:8080#' .env > "$tmpf" && mv "$tmpf" .env
else
  echo 'LLAMA_CPP_BASE_URL=http://127.0.0.1:8080' >> .env
fi

if grep -q '^LLAMA_CPP_MODEL=' .env 2>/dev/null; then
  tmpf=$(mktemp) && sed 's#^LLAMA_CPP_MODEL=.*#LLAMA_CPP_MODEL=ggml-org/gemma-3-1b-it-GGUF:Q4_K_M#' .env > "$tmpf" && mv "$tmpf" .env
else
  echo 'LLAMA_CPP_MODEL=ggml-org/gemma-3-1b-it-GGUF:Q4_K_M' >> .env
fi

mkdir -p data/env && cp .env data/env/env

Activate the backend: Deus wiring is complete but the active backend is unchanged. To switch all future sessions to llama.cpp, run:

deus backend set llama-cpp

Ollama remains the default for embeddings and judge scoring regardless of this setting.

Verify setup surface

LLAMA_CPP_BASE_URL=http://127.0.0.1:8080 npx tsx setup/index.ts --step llama-cpp

Verify provider path

If the benchmark harness exists:

LLAMA_CPP_BASE_URL=http://127.0.0.1:8080 python3 -m evolution.benchmark_generative \
  --providers llama_cpp \
  --model llama_cpp=ggml-org/gemma-3-1b-it-GGUF:Q4_K_M \
  --json

If the benchmark file is missing, use a minimal availability check instead:

python3 - <<'PY'
from evolution.generative.providers.llama_cpp import LlamaCppGenerativeProvider
provider = LlamaCppGenerativeProvider()
print({"available": provider.is_available(), "model": provider.get_default_model()})
PY

Troubleshooting

llama-server exits immediately

Check whether the model slug is valid and whether the first download returned 404 or auth errors:

tail -100 "$HOME/Library/Logs/deus-llama-cpp.error.log" 2>/dev/null || true

If the chosen Hugging Face preset is invalid, switch LLAMA_CPP_MODEL in ~/.config/deus/llama-cpp.env to a known-good preset and restart the service.

Health works but /v1/chat/completions fails

The endpoint is up, but the model did not finish loading or prompt templating is wrong. Check logs first. For instruction-tuned GGUFs like Gemma, keep --jinja enabled in the launcher script.

Deus wiring files are missing

Host installation is still complete. Tell the user clearly:

llama.cpp is running on the host, but this checkout does not yet include the optional Deus-side llama_cpp provider wiring. That remains a separate source change.

Revert

Stop the service and remove the local files:

launchctl bootout "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.deus.llama-cpp.plist" 2>/dev/null || true
rm -f "$HOME/Library/LaunchAgents/com.deus.llama-cpp.plist"
rm -f "$HOME/.config/deus/scripts/start-llama-cpp.sh"
rm -f "$HOME/.config/deus/llama-cpp.env"