What does ciagent do?

It adds pytest-native regression testing for the AI agent being built: it records golden baselines, generates a test spec, and runs trace-diff regression, multi-run stability checks, and LLM-judge audits after every agent change.

When should the check skill be run?

After every agent change, so behavior changes are caught before they ship.

What is verdict-flip attribution?

Part of the multi-run stability testing: it attributes flips in verdicts across runs, helping identify unstable agent behavior.

Does it require pytest?

Yes, the tests are pytest-native, so a pytest setup is needed.

ciagent

PackDev tools

ciagent is a plugin that adds pytest-native regression testing for the AI agent being built. An onboard skill records golden baselines and generates a test spec, while a check skill runs trace-diff regression, multi-run stability checks with verdict-flip attribution, and LLM-judge audits after every agent change. This catches behavior changes before they ship.

Unavailable. Delivery for this kind is on the roadmap — not serving yet.

Have a pytest setup for the project where the agent is being built.

What your AI can do with it

Onboard skill records golden baselines and generates a test spec
Check skill runs trace-diff regression testing
Multi-run stability testing with verdict-flip attribution
LLM-judge audits after every agent change
Tests run natively under pytest

Getting started

Have a pytest setup for the project where the agent is being built.
Install the ciagent plugin.
Run the onboard skill to record golden baselines and generate a test spec.
Run the check skill after each agent change to run regression, stability, and LLM-judge audits.

Signals

GitHub stars: 40k
Forks: 4k
Last commit: Sep 2026

Questions

What does ciagent do?: It adds pytest-native regression testing for the AI agent being built: it records golden baselines, generates a test spec, and runs trace-diff regression, multi-run stability checks, and LLM-judge audits after every agent change.
When should the check skill be run?: After every agent change, so behavior changes are caught before they ship.
What is verdict-flip attribution?: Part of the multi-run stability testing: it attributes flips in verdicts across runs, helping identify unstable agent behavior.
Does it require pytest?: Yes, the tests are pytest-native, so a pytest setup is needed.

Advanced

Item type: plugin
Key: wshobson-agents-ciagent
Source: github.com/wshobson/agents

github.com/wshobson/agents