Robots Meta Conflict
SkillSearchrobots-meta-conflict is a skill for diagnosing search indexing problems on sites that use both robots.txt disallow rules and meta robots noindex tags. It helps explain why pages that were deindexed still appear in search results, so the underlying conflict between blocking and noindexing can be identified and fixed.
Available today. Use it from your connected AI after setup.
No other account needed.
Have an AI agent that can load skills.
Then ask your AI: use the Robots Meta Conflict skill
What your AI can do with it
- Diagnose why pages blocked by robots.txt but tagged noindex still show up in Google search
- Investigate why deindexed pages still appear in search results
- Identify conflicts between robots.txt disallow rules and meta robots noindex tags
- Apply to sites that use both robots.txt and meta robots controls
Getting started
- Have an AI agent that can load skills.
- Add the robots-meta-conflict skill to the agent's available skills.
- Ask the agent to investigate why deindexed pages still appear in search results on the affected site.
- Provide the relevant robots.txt rules and meta robots tags so the agent can analyze the conflict.
What this skill tells your AI
The instructions your AI receives, as published by thedaviddias/front-end-checklist in skills/robots-meta-conflict/SKILL.md and read by ahel’s review.
Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt — Google cannot read the noindex directive it cannot fetch.
Quick Reference
- If a page is blocked in robots.txt, crawlers never fetch it and therefore never read its
noindexmeta tag - To properly deindex a page, use
<meta name="robots" content="noindex">without blocking it in robots.txt - If you block a URL in robots.txt, Google may still show it in results using anchor text from backlinks
- Audit pages that need to be deindexed by confirming they are crawlable but carry
noindex
Check
Cross-reference the site's robots.txt disallow paths against pages carrying <meta name="robots" content="noindex">. Flag any page where the URL matches a Disallow rule AND has a noindex tag — the noindex will never be seen by crawlers.
Fix
For pages that must be deindexed: remove the robots.txt Disallow rule for that URL and keep the noindex meta tag. For pages that must simply be uncrawled without appearing: keep the Disallow rule and remove the noindex tag (understanding Google may still show the URL).
Explain
Explain why a page blocked by robots.txt can still appear in Google search results, why the noindex meta tag inside a blocked page is never processed, and how to correctly deindex a page.
Code Review
Review metadata generation, rendered HTML, structured data, and response headers related to Robots Meta Conflict. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.
For full implementation details, code examples, and framework-specific guidance,
see references/rule.md.
Rule page: https://frontendchecklist.io/en/rules/seo/robots-meta-conflict
Signals
- GitHub stars
- 74k
- Forks
- 7k
- Last commit
- Aug 2026
Questions
- When should this skill be used?
- Use it on sites that use both robots.txt disallow rules and meta robots noindex tags, and when investigating why deindexed pages still appear in search results.
- What problem does it solve?
- It helps diagnose why pages blocked by robots.txt but tagged noindex still show up in Google search results.
- Does it fix the indexing problem automatically?
- The item describes diagnosis only; it helps identify the conflict so it can be addressed, but no automatic fixing is stated.
Advanced
- Item type
- skill
- Key
robots-meta-conflict- Source
- github.com/thedaviddias/front-end-checklist