Agent evaluation
SkillProductivityThis skill lets your AI evaluate how well agents and tool-using workflows actually perform. It measures task success, tool accuracy, latency, cost, and safety, and can run regression suites. It is useful when shipping agent features, comparing prompts or models, or debugging why an agent failed.
Available today. Use it from your connected AI after setup.
No other account needed.
Add the skill, then ask your AI to evaluate an agent workflow. Good moments to use it are before shipping a feature, while comparing prompts or models, or when investigating a failure.
Then ask your AI: use the Agent evaluation skill
What your AI can do with it
- Measure how often agent tasks succeed
- Check whether an agent uses its tools correctly
- Track the latency and cost of agent runs
- Evaluate agent safety
- Run regression suites to catch problems before release
- Compare prompts or models
What this skill tells your AI
The instructions your AI receives, as published by charlieviettq/awesome-agent-skill in .claude/skills/agent-evaluation/SKILL.md and read by ahel’s review.
What to measure
| Dimension | Examples |
|---|---|
| Task success | End state matches spec (binary or rubric) |
| Tool use | Correct tool, valid args, no spurious calls |
| Safety | No policy violations, no secret leakage |
| Efficiency | Tokens, latency, tool call count |
| Stability | Same input -> consistent outcome across runs |
Workflow
- Define tasks — realistic user intents with clear pass/fail or scored rubric.
- Build dataset — golden set + edge cases (errors, ambiguous input, empty context).
- Run baseline — fixed model/settings; log traces (inputs, tools, outputs).
- Score — automated checks first; human review for ambiguous cases.
- Compare — A/B prompts, models, or tool schemas; report deltas with confidence notes.
- Gate — block release on regression in must-pass tasks.
Automated checks
- Schema validation on tool arguments.
- Assert final answer contains required fields or avoids forbidden content.
- Snapshot tests for deterministic sub-steps where possible.
Human rubric (when needed)
Score 1-5 on: correctness, completeness, tone, safety. Document disagreements.
Anti-patterns
- Eval only on cherry-picked happy paths.
- Changing task and model simultaneously without isolation.
- No trace logs when debugging tool failures.
Output
Summary table: variant | success rate | avg tools | avg latency | notes.
Signals
- GitHub stars
- 26
- Forks
- 9
- Last commit
- Jul 2026
Advanced
- Catalog kind
- skill
- Gateway key
agent-evaluation-charlieviettq- Source
- github.com/charlieviettq/awesome-agent-skill