Arize evaluator

Design evaluator prompts, create or update Arize evaluators, map task columns, run evaluations, and troubleshoot ax failures without fabricating results or reading local credential files.

When to invoke

• "Create an Arize LLM judge evaluator."
• "Run a hallucination eval on spans."
• "Score this experiment for correctness."
• "Fix evaluator column mapping."
• "Set up continuous monitoring with trigger-run."

Prerequisites and context

Proceed directly with the needed ax command; do not check versions, environment variables, or profiles up front. If an ax command fails, react to the error.

Symptom	Resolution
command not found or version error	Read references/ax-setup.md.
401 Unauthorized or missing API key	Run ax profiles show. If the profile is missing or wrong, use references/ax-profiles.md; if the user lacks a key, direct them to https://app.arize.com/admin > API Keys.
Space unknown	Run ax spaces list and select by name, or ask the user.
LLM provider call fails because OPENAI_API_KEY or ANTHROPIC_API_KEY is missing	Run ax ai-integrations list --space SPACE to check platform-managed credentials. If none exist, ask for a key or use the arize-ai-provider-integration skill.

Security rule: never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. If credentials are unavailable through those channels, ask the user.

SPACE, --space, and ARIZE_SPACE accept either a space name such as my-workspace or a base64 space ID such as U3BhY2U6...; find values with ax spaces list.

Evaluator model

Field	Meaning
Template	Judge prompt with {variable} placeholders such as {input}, {output}, {context}, or {conversation}.
Classification choices	Allowed labels such as factual / hallucinated, correct / incorrect, or pass / fail; each may carry a numeric score.
AI Integration	Stored LLM provider credentials used by the evaluator.
Model	Judge model such as gpt-4o or claude-sonnet-4-5.
Invocation params	JSON model settings such as {"temperature": 0}.
Optimization direction	maximize when higher scores are better, minimize when lower scores are better.
Data granularity	span, trace, or session; most evaluators default to span.

Evaluators are versioned. Every prompt or model change creates a new immutable version, and the newest version is active.

Task model and granularity

Task field	Meaning
Evaluators	One or more evaluators to run.
Column mappings	Maps template variables to span or run fields, such as input → attributes.input.value.
Query filter	SQL-style expression such as span_kind = 'LLM' to choose spans or runs.
Continuous	Project task option that scores new spans as they arrive.
Sampling rate	Continuous task fraction from 0 to 1.

Granularity	What it evaluates	Use for	Result column prefix
span	Individual spans	Q&A correctness, hallucination, relevance	eval.{name}.label, eval.{name}.score, eval.{name}.explanation
trace	Spans grouped by context.trace_id	Agent trajectory and full call-chain task correctness	trace_eval.{name}.label, trace_eval.{name}.score, trace_eval.{name}.explanation
session	Traces grouped by attributes.session.id and ordered by start_time	Multi-turn coherence, tone, and conversation quality	session_eval.{name}.label, session_eval.{name}.score, session_eval.{name}.explanation

For trace granularity, values are grouped by context.trace_id and comma-joined, with each value truncated to 100K characters. For session granularity, trace-level grouping happens first, then traces are ordered by start_time and grouped by attributes.session.id; session-level values are capped at 100K characters total. At session granularity, {conversation} renders as a JSON array of {input, output} turns from attributes.input.value / attributes.llm.input_messages and attributes.output.value / attributes.llm.output_messages. At span or trace granularity, {conversation} is resolved like any other mapped variable.

Multi-evaluator tasks may contain different granularities. Runtime uses the highest granularity, session > trace > span, and splits into one child run per evaluator. Per-evaluator query_filter in the task evaluators JSON narrows included spans, such as only tool-call spans within a session.

Template design rules

Rule	Requirement
Portable variables	Use {input}, {output}, and {context}, not project-specific names such as {attributes_input_value}. Wire actual paths in column_mappings.
Binary first	Prefer two labels such as hallucinated / factual because more labels increase ambiguity and lower inter-rater reliability.
Exact label output	Prompt the judge to respond with only one label string, and ensure labels exactly match --classification-choices by spelling and casing.
Low temperature	Use --invocation-params '{"temperature": 0}' for reproducible scoring.
Explanations during setup	Use --include-explanations while debugging judge behavior.
Shell quoting	Pass templates in single quotes, for example --template 'Judge this: {input} → {output}'; double quotes can cause shell interpolation.
Classification choices	Always set --classification-choices; omitting it can fail with "missing rails and classification choices."

Limits

Never fabricate evaluation results. If a task fails, is cancelled, or produces no scores, report the failure and explain what happened. Do not perform a manual evaluation, invent quality scores, estimate percentages, or present agent analysis as Arize evaluation output. Recommend fixing the issue and retrying, trying the Arize UI, verifying credentials with ax ai-integrations list, or contacting https://arize.com/support.

Progressive disclosure and bundled resources

• references/evaluator-crud-workflows.md: GraphQL CRUD calls, project and experiment evaluator setup, trigger-run operations, task management, column mapping, continuous monitoring, and troubleshooting.
• references/ax-setup.md: ax install and version remediation.
• references/ax-profiles.md: profile creation and update workflow.

Related primitives

Name	Type	Use it when
arize-ai-provider-integration	skill	Creating, updating, or deleting LLM provider credentials.
arize-trace	skill	Exporting spans to discover column paths and time ranges.
arize-experiment	skill	Creating experiments and exporting runs for experiment column mappings.
arize-dataset	skill	Exporting dataset examples to find input fields when runs omit them.
arize-link	skill	Creating deep links to evaluators and tasks in the Arize UI.

Output template

### Arize evaluator result

**Status:** created | updated | run complete | failed | blocked
**Space:** `<SPACE or ARIZE_SPACE>`
**Evaluator:** `<name/id/version>`
**Task:** `<task id or n/a>`
**Granularity:** span | trace | session

| Step | Command | Result |
| --- | --- | --- |
| <step> | `ax ...` | <output summary> |

### Column mappings
| Template variable | Data field |
| --- | --- |
| `{input}` | `<field path>` |
| `{output}` | `<field path>` |

### Evaluation results
- <actual Arize result, task status, or failure reason; never fabricated>

Quality gate

• The task proceeded with the needed ax command before speculative prechecks.
• SPACE / --space / ARIZE_SPACE was resolved by name or base64 ID.
• Credentials were checked only through ax profiles or ax ai-integrations; no .env files were read.
• Template labels exactly match --classification-choices.
• --invocation-params '{"temperature": 0}' is used unless a different temperature is justified.
• Column mappings connect every template variable to real span, trace, session, project, dataset, or experiment fields.
• Results reported are actual Arize outputs; failures are not converted into manual scores.

References

• Arize admin API keys
• Arize support