crawl-census
IntegrationWeb & browsingYour AI can find out how a website treats crawlers before it fetches anything. crawl-census answers, for any domain, whether the site will serve its requests, refuse them, or charge for access. The result is fewer blocked fetches and no guesswork when your AI reads the web.
Serves today. ahel can serve it to every connected agent through your gateway link.
After adding it, have your AI check a domain before it fetches any page from that site. Details are in the crawl-census-client repository on GitHub.
What your AI can do with it
- Check whether a domain serves, refuses, or charges crawlers before fetching
- Look up a site's stance on crawler access before visiting a page
- Skip fetches to domains that refuse crawler access
- Know in advance when a domain charges for access
Advanced
- Delivery
- crawl-census MCP server → your ahel gateway (mcp.ahel.ai) → every connected AI client.
- Catalog kind
- mcp-server
- Gateway key
io-github-taylorsmithgg-crawl-census- Source
- github.com/taylorsmithgg/crawl-census-client
- Hosted endpoint
https://crawlcensus.com/mcp