Hugging Face Community Evaluations
SkillAI & modelsRun reproducible local or remote model evaluations with inspect-ai or LightEval, bounded smoke tests, captured configurations, and comparable results.
Available today. Use it from your connected AI after setup.
No other account needed.
Connect ahel once, and every AI you use reads what you have installed.
Then ask your AI: use the Hugging Face Community Evaluations skill
What this skill tells your AI
The instructions your AI receives, as published by metaspartan/cybara in plugins/huggingface-workflows/skills/huggingface-community-evals/SKILL.md and read by ahel’s review.
Use Inspect or LightEval for models hosted on the Hub. Keep evaluation execution separate from publishing results.
Workflow
- Record the model ID, immutable revision, inference backend, tokenizer, chat template, dtype, quantization, generation parameters, task revision, and seed.
- Check gated-model authentication without printing credentials.
- Choose a backend supported by the model and hardware. Prefer vLLM for supported throughput workloads and Transformers or Accelerate as compatibility fallbacks.
- Start with a bounded smoke run such as 10 examples.
- Inspect failures and sample outputs before scaling.
- Save raw results, aggregate metrics, environment metadata, and the exact command.
- Compare only runs with compatible task versions, prompts, few-shot settings, and inference parameters.
Use uv run for Python evaluation environments. Run remote evaluation through the huggingface-jobs workflow only after confirming paid hardware and timeout.
Do not cherry-pick favorable tasks, silently discard failures, or present incomparable scores in one ranking. Publishing model-card results, opening a pull request, or modifying a leaderboard is a separate external action that requires confirmation.
Signals
- GitHub stars
- 28
- Forks
- 7
- Last commit
- Sep 2026
Advanced
- Catalog kind
- skill
- Gateway key
huggingface-community-evals- Source
- github.com/metaspartan/cybara