Publish a robots.txt file
SkillWeb & browsingrobots-txt is a skill for auditing the robots.txt file of any public website. It guides an agent through checking that the file is served correctly, that its directives are valid, and that nothing accidentally blocks search engines from crawling the site. It also covers fixing problems and explaining how crawler access rules affect indexing.
Available today. Use it from your connected AI after setup.
No other account needed.
Have an agent environment that can load skills and fetch public web pages.
Then ask your AI: use the Publish a robots.txt file skill
What your AI can do with it
- Fetch /robots.txt on a live domain and verify it returns HTTP 200
- Check User-agent, Disallow, and Allow directives for correct syntax
- Verify a Sitemap: directive points to the XML sitemap
- Detect accidental Disallow: / rules that block the whole site
- Flag blocked CSS/JS assets needed for rendering-based indexing
- Review metadata, rendered HTML, and headers for robots.txt violations
Getting started
- Have an agent environment that can load skills and fetch public web pages.
- Add the robots-txt skill to the agent's available skills.
- Give the agent the production domain of the site to audit.
- Ask it to check the robots.txt file, report any blocking directives, and propose fixes.
What this skill tells your AI
The instructions your AI receives, as published by thedaviddias/front-end-checklist in skills/robots-txt/SKILL.md and read by ahel’s review.
robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.
Quick Reference
- Serve a valid
robots.txtat/robots.txton the production domain, returning HTTP 200 - Include a
Sitemap:directive pointing to your XML sitemap - Never disallow crawling of CSS/JS assets that render your pages
- Avoid blocking all crawlers with
Disallow: /on a live site
Check
Fetch /robots.txt on the live domain and verify it returns HTTP 200, uses correct User-agent / Disallow / Allow syntax, and includes a Sitemap: directive pointing to the XML sitemap. Check for accidental Disallow: / directives.
Fix
Create or update robots.txt at the web root with valid directives. Add a live Sitemap: line for the production sitemap URL. Remove any Disallow: / rules that block the whole site or resources needed for rendering.
Explain
Explain how robots.txt controls crawler access, why an accidental Disallow: / can delist a site, and why CSS/JS must remain accessible for rendering-based indexing.
Code Review
Review metadata generation, rendered HTML, structured data, and response headers related to Publish a robots.txt file. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.
For full implementation details, code examples, and framework-specific guidance,
see references/rule.md.
Rule page: https://frontendchecklist.io/en/rules/seo/robots-txt
Signals
- GitHub stars
- 74k
- Forks
- 7k
- Last commit
- Aug 2026
Questions
- What does this skill apply to?
- It applies to any public website, and is meant for auditing technical SEO foundations or diagnosing crawl coverage gaps.
- How long does it take?
- The estimated time is about 10 minutes, and the difficulty is beginner level.
- Why is robots.txt important?
- It is the first file crawlers fetch. A misconfigured directive can silently block search engines from crawling the entire site and remove its organic visibility.
- What are the most common mistakes it catches?
- A missing or non-200 robots.txt, invalid directive syntax, a missing Sitemap: line, accidental Disallow: / rules, and blocked CSS/JS assets needed for rendering.
Advanced
- Item type
- skill
- Key
robots-txt-thedaviddias- Source
- github.com/thedaviddias/front-end-checklist