Implement AI Bot Rules in robots.txt
SkillCloud & infraSub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt
Available today. Use it from your connected AI after setup.
No other account needed.
Connect ahel once, and every AI you use reads what you have installed.
Then ask your AI: use the Implement AI Bot Rules in robots.txt skill
What this skill tells your AI
The instructions your AI receives, as published by fabricioctelles/skills in skills/agent-ready-cloudflare/ai-rules/SKILL.md and read by ahel’s review.
Add explicit User-agent entries for AI crawlers in your robots.txt per
RFC 9309.
Requirements
- Add
User-agentblocks for AI-specific bots:GPTBot,OAI-SearchBot,Claude-Web,Google-Extended,Amazonbot,anthropic-ai,Bytespider,CCBot,Applebot-Extended - Set
Allowand/orDisallowrules matching your content policy - A wildcard
User-agent: *block alone is not sufficient — explicit AI bot entries are required
Cloudflare
AI Crawl Control lets you manage AI bot rules from the dashboard without editing robots.txt manually.
Validate
POST https://isitagentready.com/api/scan
Content-Type: application/json
{"url": "https://YOUR-SITE.com"}
Check that checks.botAccessControl.robotsTxtAiRules.status is "pass".
Signals
- GitHub stars
- 77
- Forks
- 7
- Last commit
- Sep 2026
Advanced
- Catalog kind
- skill
- Gateway key
agent-ready-ai-rules- Source
- github.com/fabricioctelles/skills