Agent Eval Tests

Overview

Evals verify assistant behavior by running a real prompt against the full agent stack, live, then grading the outcome with a Scorer — code that checks the real DB/tool-invocation effect (deterministic, "dimension G") or an LLM judge for open-ended quality ("dimensions A/B/H"). This supersedes trusting the agent's own self-reported completedCriteria.

Package: packages/core/compute/assistant-evals. Library: src/runner.ts (createEvalRunner), src/assertions.ts (deterministic helpers), src/judge.ts (LLM-judge helper). Evals live in src/evals/*.eval.ts. See packages/core/compute/ai/TESTING.md for how this package is scoped (cross-plugin scenarios live here; single-plugin scenarios belong in their own plugin package, importing this library). The older memoized/live gated agent-e2e harness is a separate, deprecated package, @dxos/assistant-e2e — not covered by this skill; see its own README.

Eval File Structure

import * as Effect from 'effect/Effect';
import * as Schema from 'effect/Schema';
import { evalite } from 'evalite';

import { objectExists } from '../assertions';
import { createEvalRunner } from '../runner';

const task = createEvalRunner({
  instructions: trim`
    Create a new organization called "{{name}}".
  `,
  input: Schema.Struct({ name: Schema.String }),
  output: Schema.Unknown,
  dbQuery: ({ name }) => objectExists(Organization.Organization, (org) => org.name === name),
});

evalite('Descriptive scenario name', {
  data: [{ input: { name: 'Cyberdyne Systems' } }],
  task,
  scorers: [
    {
      name: 'organization-created',
      description: 'The named Organization object exists in the DB after the run.',
      scorer: ({ output }) => (output.dbQuery ? 1 : 0),
    },
  ],
});

createEvalRunner boots a full Composer test harness, invokes the prompt, and — when dbQuery is passed — runs a deterministic assertion while the space is still open, returning { agentOutput, dbQuery } instead of the bare agent output. Model precedence: variant.model → options.model → com.anthropic.model.claude-opus-4-8.default.

createEvalRunner options

• instructions / input / output — the prompt (supports {{field}} templating from input) and its Effect Schemas.
• skills — defaults to getDefaultSkills() (SkillManagerSkill + DatabaseSkill); pass [] for scenarios that need no tools (e.g. smoke), or a custom Ref.make(SomeSkill.make())[].
• plugins — extra plugins beyond the default ClientPlugin/AssistantPlugin/RoutinePlugin/ InboxPlugin set (e.g. MarkdownPlugin(), CrmPlugin()).
• sessionChat: true — provisions a Chat on the session feed. Required whenever a skill's tool resolves context via Chat.getFromContext (e.g. planning's update-tasks — its plan lives at Chat.plan); omitting it when needed fails with a context-resolution error, not a clear one.
• expect: 'failure' — inverts success semantics for scenarios that assert the agent correctly fails. The task resolves { failed: boolean } instead of throwing, so a scorer can grade "failed as instructed" as a pass. Internally runs via Effect.runPromiseExit instead of EffectEx.runAndForwardErrors. A timeout is never treated as this kind of failure — it always throws, even here.
• timeout — milliseconds before the run is aborted, default 60_000. evalite has no per-scenario timeout of its own; this is what actually bounds each eval (vitest.config.ts's testTimeout is just the outer safety net). Raise it only for scenarios with more tool round-trips than a typical eval — e.g. crm-mailbox.eval.ts/planning.eval.ts use 150_000.
• dbQuery: (input, spaceId) => Effect<D, unknown, Database.Service> — see Assertions below.

Assertions (../assertions.ts)

All are Effect<_, _, Database.Service> — compose freely inside a dbQuery's Effect.gen:

• objectExists(type, predicate) / findObject(type, predicate) — query the DB for a matching entity (object or relation). findObject returns the match itself (e.g. to load a Ref field off it, or inspect a relation's fields); objectExists just a boolean.
• For relations, resolve the endpoints with Relation.getSource(rel) / Relation.getTarget(rel) (from @dxos/echo) — synchronous, no load needed.
• completedBlocks() — every CompleteBlock event off the space's trace feed, in order, as { role, block }. This is how you check the assistant's actual chat text (filter block._tag === 'text' && role === 'assistant') without trusting the agent's self-report.
• toolInvocations() — built on completedBlocks(); pairs toolCall/toolResult blocks by toolCallId into { name, operationKey?, input, result?, error? }. Use operationKey (a stable dxn:org.dxos.function.* string) to match a specific Operation-backed tool — not name, which is a display/toolkit name that varies (see Gotchas). Absent operationKey means the tool isn't Operation-backed (provider-defined tools like Anthropic's web search, MCP tools).

LLM-judge scorer (../judge.ts)

For criteria a deterministic check can't grade (open-ended quality, e.g. "is this a well-formed haiku about X"):

import { judge } from '../judge';

const rubric = 'Does the text contain a well-formed 3-line poem about spring rain? Pass only if...';

function* example() {
  const verdict = yield* judge(rubric, assistantText);
  // verdict: { pass: boolean, reasoning: string }
}

judge() calls @dxos/ai's LanguageModel.generateObject directly (Anthropic, via the same DX_ANTHROPIC_API_KEY-backed access runner.ts uses) with a schema-typed { pass, reasoning } response — no free-text/regex JSON parsing. Uses claude-haiku-4-5 (grading is classification, not generation; a fast/cheap model is enough).

Deliberately does not use autoevals's built-in LLM-judge classifiers (Factuality, ClosedQA, Battle, etc., already a dependency, used for Levenshtein in basic.eval.ts) — those are hardcoded to an OpenAI-shaped client; using them here would need a separate OpenAI API key or routing through Braintrust's proxy, neither of which this repo has wired up.

Use narrowly. A judge is non-deterministic and costs a real model call every run. Reach for it only for the specific criterion that needs a content judgment, never as a blanket replacement for a deterministic check that already exists — and when you add one, also demonstrate it can fail (a judge that only ever passes is worthless as a scorer). See planning.eval.ts for the pattern: one dbQuery-embedded judge call for the real scenario's haiku-quality criterion, plus a second evalite() case in the same file feeding the same rubric a hand-crafted bad transcript, asserting pass === false. Don't build a separate meta-test file for the judge mechanism itself, and don't convert every eval's checks to judges just because one exists — most criteria in this package should stay deterministic.

Running Evals

Requires a real DX_ANTHROPIC_API_KEY — never run in CI (evalite isn't in any CI workflow; see .github/workflows/check.yml), manual/on-demand only for now.

# Whole suite
export DX_ANTHROPIC_API_KEY=...
moon run assistant-evals:evals

# Single file — the moon task hardcodes `args: [src/evals]`, so passing another arg through
# `moon run ... -- <file>` errors ("Too many arguments"). Bypass moon:
cd packages/core/compute/assistant-evals
npx evalite run src/evals/database.eval.ts

In this repo, pull the key from the 1Password CI vault rather than exporting it manually:

eval "$(pnpm -ws 1p-credentials)"
npx evalite run src/evals/planning.eval.ts

Gotchas (found the hard way — real debugging sessions, not speculation)

Symptom	Cause	Fix
TypeError: Cannot read properties of undefined (reading 'meta') before any model call	evalite's flat vitest.config.ts must include '#*' in PluginImportSource's include list, or Node subpath imports resolve to a stale compiled dist/ bundle instead of src/.	Keep PluginImportSource({ include: ['@dxos/**', '#*'] }) in vitest.config.ts. Don't remove it.
NOT NULL constraint failed: results.output	The task returned (or resolved to) undefined — e.g. completeJob called with no success payload when no output schema was requested. evalite's SQLite storage rejects it.	Coerce in the eval file ((await runner(...)) ?? {}), not in runner.ts's general contract.
A multi-tool scenario times out around 60s	That's createEvalRunner's per-eval default (timeout option), not evalite's own 30s default or vitest.config.ts's testTimeout — those are a fallback and an outer safety net, respectively.	Pass an explicit timeout (ms) to createEvalRunner for that scenario, not a global config bump — see options below.
A tool-name/operationKey check that should obviously match doesn't	Recorded names aren't always what you'd guess: web-search's toolkit name is 'AnthropicWebSearch', not 'web_search'; planning's operationKey has a 'dxn:' prefix.	Don't guess twice — inspect node_modules/.evalite/cache.sqlite's results table directly (SELECT output FROM results ORDER BY id DESC LIMIT 1) to see the actual recorded value.

Common Mistakes

Mistake	Fix
Adding a judge for a criterion a dbQuery check could grade	Reach for judge() only when the criterion is a genuine content/quality judgment.
A judge with no demonstrated failure case	Add a case (in the same eval file) proving it can fail, using a hand-crafted bad input.
Matching a tool by name instead of operationKey	name is a display/toolkit name and varies; operationKey is the stable match target.
Guessing a tool name/operationKey string instead of checking	Add a temp debug field to the dbQuery output, run once, inspect cache.sqlite, fix, remove the debug field.
Forgetting sessionChat: true for a chat-scoped skill's tool	Symptom is a context-resolution error, not "no chat found" — check the skill's operation for Chat.getFromContext.
Assuming pre-seeded data without saying so in the prompt	State the DB starts empty; seed via the database skill's tools at the start of the prompt.
Pasting entire eval files in chat when structure is standard	Point at the file + line range instead.