End-to-End Testing

Run structured smoke tests against the live Mycelium stack. Tests are cumulative — each phase depends on the previous one passing.

Arguments

• --quick — Stack health + memory CRUD + search only (< 1 min)
• --full — Quick + aligner-mediated negotiation to consensus (~ 3 min)
• No argument — defaults to --full

Phase 1: Stack Health

Verify all services are running and healthy.

# 1. Backend health
curl -sf http://localhost:8000/health | python3 -m json.tool
# Expect: status=ok, database.status=ok, embedding.status=ok, llm.status=ok

# 2. Container status
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -E "mycelium|ioc"
# Expect: all containers healthy

# 3. CFN mgmt plane (if IoC enabled)
curl -sf http://localhost:9000/health
# Expect: {"status":"healthy"}

# 4. CFN node (if IoC enabled)
docker inspect ioc-cfn-svc --format '{{.State.Health.Status}}'
# Expect: healthy

Fail criteria: Any service unhealthy → stop and diagnose. Do not proceed.

Phase 2: Memory CRUD + Search

Test the core memory pipeline: write, read, list, search, delete.

# Setup
mycelium room create e2e-test-room --trigger threshold:10
mycelium room use e2e-test-room

# Write memories (with embeddings)
mycelium memory set decisions/test-db "Chose Postgres for reliability" -H e2e-agent
mycelium memory set decisions/test-cache "Redis for session caching" -H e2e-agent
mycelium memory set failed/test-sqlite "SQLite can't handle concurrent writes" -H e2e-agent
mycelium memory set status/test-deploy "Staging deploy in progress" -H e2e-agent

# Read back
mycelium memory get decisions/test-db
# Expect: content matches what was written

# List
mycelium memory ls
# Expect: 4 memories listed

# List by prefix
mycelium memory ls decisions/
# Expect: 2 decisions shown in table

# Semantic search
mycelium memory search "what database did we pick"
# Expect: decisions/test-db appears with high similarity

mycelium memory search "what failed"
# Expect: failed/test-sqlite appears

# Delete
mycelium memory rm decisions/test-cache --force
mycelium memory ls
# Expect: 3 memories (test-cache gone)

# Filesystem verification
ls ~/.mycelium/rooms/e2e-test-room/decisions/
# Expect: test-db.md exists, test-cache.md gone
cat ~/.mycelium/rooms/e2e-test-room/decisions/test-db.md
# Expect: YAML frontmatter + content

Fail criteria: Any write/read/search fails → embedding or DB issue.

Phase 3: CLI Negotiation

Test the full coordination pipeline: post positions → summon the aligner → await → respond → consensus → plan.

Coordination is the resident-runtime protocol: each participant is a live caller that loops await → reason → respond. The aligner (a backend engine) runs a real NEGMAS negotiation, @-addressing one agent at a time, and owns termination — it stops the instant the agents agree, then compiles the consensus into plan/tasks.md. There is no daemon and no cold-spawn: an @-mention to a non-resident handle just waits on the durable transcript cursor until someone awaits. For this smoke test, the operator plays each agent's turn by hand.

# Register the aligner once in the room
mycelium engine create aligner --kind aligner --room e2e-test-room

# Each participant posts an opening position
mycelium respond --room e2e-test-room --handle agent-alpha "Prioritize performance"
mycelium respond --room e2e-test-room --handle agent-beta  "Prioritize developer experience"

# Summon the aligner to converge
mycelium engine invoke aligner "converge on the priority tradeoff" -r e2e-test-room

# Loop each agent: await the aligner's address, then reply. Repeat until the
# plan lands. (In production the runtime does this via `mycelium await --loop
# --exec <cmd>`; here we drive it by hand.)
mycelium await   --room e2e-test-room --handle agent-alpha --json   # read the prompt
mycelium respond --room e2e-test-room --handle agent-alpha "I can accept perf caps if DX tooling ships too"
mycelium await   --room e2e-test-room --handle agent-beta  --json
mycelium respond --room e2e-test-room --handle agent-beta  "works if we keep the fast path"

# On agreement the aligner records the episode and compiles the plan BEFORE the
# consensus is announced (so the plan exists when `await` returns).
mycelium plan tasks --room e2e-test-room
# Expect: a shared - [ ] checklist with @handle owners

Fail criteria:

• await never returns after the summon → aligner not registered, or LLM unavailable (mycelium status → llm)
• Aligner loops to a step cap instead of stopping on agreement → NEGMAS termination regression (it must stop at unanimity, never run out the cap)
• No plan/tasks.md after convergence → plan compiler outage; check backend logs (fail-soft should still emit the raw issue=value agreement)
• An unreadable reply produces phantom convergence → interpretation regression (an unreadable proposer must hold its own last line, never the standing offer)

Phase 4: Second episode (same room)

Verify a second negotiation can run in a room after the first converges. A room is persistent; each summon opens a fresh, independent episode.

# Post fresh positions and summon again — same room, new episode
mycelium respond --room e2e-test-room --handle agent-gamma "Ship fast"
mycelium respond --room e2e-test-room --handle agent-delta "Ship safe"
mycelium engine invoke aligner "converge on the ship-speed tradeoff" -r e2e-test-room

# Drive the await → respond loop for both agents as in Phase 3, then:
mycelium plan tasks --room e2e-test-room
# Expect: convergence with a distinct episode id and no stale-participant errors

Fail criteria:

• Second summon reuses the first episode's transcript slice → episode isolation regression
• Aligner sees the prior episode's positions → episode scoping leaked across summons

---

Cleanup

# Delete test room
curl -s -X DELETE http://localhost:8000/api/rooms/e2e-test-room
# Also clean up any episode sub-rooms

Interpreting Failures

Symptom	Likely cause	Check
Backend returns 500 on memory write	Embedding model not loaded	docker logs mycelium-backend | grep embed
Search returns empty	Embeddings are null (wrote with --no-embed)	Reindex: mycelium memory reindex
await never returns after a summon	aligner not registered or LLM down	mycelium engine ls -r <room>; mycelium status → llm
Aligner never stops (runs to the cap)	NEGMAS termination regression	it must stop at unanimity, never run out the step cap
No plan/tasks.md after convergence	plan compiler outage	backend logs; fail-soft emits the raw issue=value agreement
Phantom convergence on an unreadable reply	interpretation regression	proposer must hold its own last line, never the standing offer