Annotate and review traces

Automated metrics are downstream of one thing: a human deciding what "good" means. For specialized or high-stakes domains (finance, health, legal), and for early-stage apps, structured human review of real traces is the single highest-ROI activity. It produces the golden labels every other eval depends on, and it surfaces failure modes you did not know to look for.

Set up the review loop

1. Pull a sample of traces to review. Stratify (by topic, difficulty, low online-eval score, thumbs-down) so reviewers see the interesting cases, not 100 easy ones.
2. Give reviewers the full context the model had: input, retrieved docs, tool results, output. Redact PII first for regulated data (redact-pii-for-tracing).
3. Use a simple, consistent schema: pass/fail (or a small rubric score) + a free-text failure reason + a category tag. The free-text is where you discover new failure modes; the categories let you count them.
4. Use the tooling rather than spreadsheets where possible: annotation queues exist in Langfuse, Phoenix, Opik, LangSmith and let annotations attach to the trace.

Do error analysis (not just labeling)

The point is not a score, it is understanding. After a review pass:

• Read the free-text reasons and cluster them into failure categories (retrieval miss, hallucination, formatting, refusal, tone, ...). Count each.
• Fix the biggest category first. A few categories usually explain most failures.
• This is the loop that turns "the app is kind of bad" into "34% of failures are retrieval misses, here is the fix."

Turn reviews into durable assets

• Reviewed pass/fail labels become your golden eval dataset (build-eval-dataset) and regression cases (trace-based-testing).
• Reviewed labels also calibrate your LLM-as-judge: check the automated judge agrees with the humans; fix the rubric until it does (add-llm-evals). Then the judge can scale what humans validated.

Verify

• Reviewers see full context and use a consistent schema.
• Failure reasons are clustered into categories with counts, not just an average score.
• Reviewed items feed the eval set and calibrate the automated judge.

Anti-patterns

• Never looking at your actual data, only at aggregate metrics (you miss the failure modes).
• Reviewing without the retrieved context/tool results (you cannot tell why it failed).
• Pass/fail with no reason text (you get a number, not an insight).
• Labels that never become an eval set or judge calibration (wasted expert time).

Grounding

"Look at your data" and error analysis are the core of practitioner eval methodology: Hamel Husain, Your AI Product Needs Evals; human labels are the ground truth that automated LLM-as-a-judge is calibrated against (Zheng et al. 2023, arXiv:2306.05685).