Blocked scrape triage

A scrape came back wrong. Find out why before changing anything, then escalate in cost order. Most "blocks" are not bot detection, some are not even a live server, and the browser is the most expensive answer rather than the first one.

Work the steps in order. Step 0 ends more investigations than the whole ladder does.

Prerequisites

• Apify account (sign up)
• Authentication via one of:
  • apify login (OAuth, if using the Apify CLI)
  • APIFY_TOKEN environment variable
  • Token from Apify Console → Settings → Integrations
• A one-time permission approval, before the first diagnostic run. apify/cheerio-scraper requires full account access and refuses to start until that is approved in the Console. Both paths refuse, with different wording: MCP call-actor answers requires full access to your account. You must approve its permissions before running it, while the CLI prints Error: Actor apify/cheerio-scraper requires full access to your Apify account and has not been approved yet. on stdout with exit code 0 — so an unattended CLI flow reads it as a normal result and stops without saying why. Approve it once at the URL the message prints, before you need a diagnosis. The same applies to apify/web-scraper and the browser Actors in Rung 5; apify/website-content-crawler does not require it.

What a diagnosis is allowed to cost

One DNS pair, one navigation per egress per host, then stop and conclude. The unit is the host, not the organisation: one county published its records across three hostnames that refused a request in three different ways, and a budget spent on "the target" would have bought one of those three answers. The budget counts requests to hosts the target owns; looking something up on a third-party platform, a data catalogue or your own IP check is free. If you are four requests into one host and still cannot name the cause, you are no longer diagnosing, you are probing, and you should say what you know and what it would cost to learn more. One navigation means one page: a browser pulling forty subresources for that page is still one navigation, and a burst of repeated probes is not. The one exception worth spending on is the residential probe in Step 3, a single run that separates "your network" from "actually defended".

Sample before you commit to a fix, change one variable at a time, and remember that your diagnosis is traffic the target counts. Residential traffic and browser Actors are the two line items that surprise people.

One variable includes time. Measurements taken from one egress at 05:44 and from another at 05:57 differ by network and by half an hour, and a refusal that expired on its own is indistinguishable from one your second egress defeated. When you compare two egresses, interleave them inside the same minute. This bit a real investigation: three hosts looked like a clean "the network is the cause" result until the blocked egress was re-measured later and two of the three answers had changed.

Know your own vantage point first

Two requests, both free, both about you rather than the target. Skipping them is how a problem with your own address becomes a proxy purchase.

1. What your egress looks like from outside. Not its class in the abstract, the actual address, network and country the site sees:

curl -s "http://ip-api.com/json/?fields=query,country,as,org,hosting"

That service is free for non-commercial use and rate limited, and it tells you where you stand in its own headers: a live call returned X-Rl: 44 (calls left in the window) with X-Ttl: 60 (seconds until reset). Read them rather than discovering the limit as a failure, and for anything regular use a source whose terms fit your use. When you only need the address, the country and which POP you land on, an endpoint with no such conditions does that much:

curl -s "https://www.cloudflare.com/cdn-cgi/trace"

Measured side by side from one egress: the trace returned ip=64.176.60.193, loc=JP and colo=KIX, while the first call added what the trace does not carry, AS20473 The Constant Company and "hosting": true. The ASN and the hosting flag are the parts that predict a refusal, so the cheaper endpoint is a supplement rather than a replacement.

"hosting": true means every WAF you meet has classified you before reading a single header of yours, and that a 403 from a public site is the expected answer rather than a surprise. Egress below means exactly this: the network your request leaves from, as the site sees it. Two egresses of the same class, a hosting IP and a datacenter proxy, are usually one data point to a WAF rather than two.

Run this from every machine you are about to call a separate vantage point, and compare the addresses rather than the machines. A laptop behind a VPN client and the VPN's own server are one egress, not two, and the check says so in one line. Measured: two machines a continent apart returned the same query address, so every "second opinion" taken from the laptop was the first opinion again. What did differ was the path: the same request that timed out after 30 seconds from the laptop reported connect=0.000000s from the server, which is a silent TCP drop rather than a slow host, and only the second form of the measurement says which one you have.

2. Whether your DNS answers mean anything. Resolve over DoH rather than UDP, against two providers:

curl -s -H "accept: application/dns-json" "https://dns.google/resolve?name=HOST&type=A"
curl -s -H "accept: application/dns-json" "https://cloudflare-dns.com/dns-query?name=HOST&type=A"

If UDP answers hand back a non-routable address while DoH returns a real one, your own machine is answering, and the UDP answer is not evidence of anything. The usual cause is not hostile: proxy clients in the Clash, mihomo and sing-box family default to fake-ip mode, where the client invents an address out of 198.18.0.0/15 so it can route by hostname rather than by IP. Addressing the query to 1.1.1.1 or 8.8.8.8 changes nothing, because the answer never leaves the machine. Observed on exactly such a setup: both UDP queries returned 198.18.5.101 for a host whose real addresses came back over DoH from those same two providers.

If you are on a VPN or proxy client at all, assume this until DoH says otherwise, and never report a host as dead on a UDP answer alone.

DoH is the better witness here, not an infallible one. On a corporate or container network with split-horizon DNS, an internal name resolves only through the local resolver, and DoH will answer NXDOMAIN for a host that exists for you; some networks also block DoH outright. The rule above is about public sites, where the public answer is the one that matters. If your target is internal, invert it: trust the local resolver and use DoH only to prove the name is not public.

Step 0: is this data published somewhere cheaper

These checks cost at most one request each and they end the investigation outright. In a sweep of twenty-one public data sites, this section was the right answer more often than the entire ladder below. That sample is public-sector and open-data heavy, where defences are rare and publication is common, so read the ratio as an argument for checking cheaply first, not as a claim about the web at large: on commercial marketplaces and listing sites the balance moves the other way, and this section will end fewer of your investigations.

• A deliberate publication of the same data. An open data portal, an ArcGIS REST service, a bulk download page, a .json twin of the HTML route, sitemap.xml. A published feed is cheaper, more stable and more complete than anything you will scrape, and it often answers from an egress the main site is refusing.
• A login you do not own, or a paywall. Stop and say so.
• Terms that forbid this collection. Read the target page itself, not only robots.txt: assessment portals in particular tend to print the prohibition in the page body while robots.txt says nothing. Many sites publish no robots.txt at all, and its absence decides nothing either way. Where the terms forbid automated collection and a portal publishes the same records, the portal is not a workaround, it is the supported route.

Find the portal rather than guessing at it. Two families cover most public data, and both have a discovery step people skip because the URL template is easy to remember and useless without an id.

# Socrata: ask the federated catalog what a domain publishes, then fetch by id
curl -s "https://api.us.socrata.com/api/catalog/v1?domains=DOMAIN&q=TERM&limit=5"
curl -s "https://DOMAIN/resource/DATASET_ID.json?\$limit=5"

# ArcGIS: walk the directory instead of guessing a layer number
curl -s "https://HOST/arcgis/rest/services?f=json"
curl -s "https://HOST/arcgis/rest/services/FOLDER/SERVICE/MapServer?f=json"
curl -s "https://HOST/arcgis/rest/services/FOLDER/SERVICE/MapServer/LAYER/query?where=1%3D1&outFields=*&resultRecordCount=5&returnGeometry=false&f=json"

A hostname that answers is still a guess, and the check is the content rather than the status. The catalog endpoint above returns the whole federated catalog on any host, so a guessed portal name answered 200 with ten thousand results belonging to three other states, and the real portal returned the same rows. Asking the platform whether it knows the domain separated them in one request: ?domains=GUESSED_HOST answered {"error":"Domain not found: ..."} for the guess and its own datasets for the real one. Generalised: ask whether the records name the host you asked. The same rule kills wildcard DNS, where every name in a zone resolves and none of them is a service.

A hostname with no owner in it is usually a vendor serving hundreds of clients, so the block belongs to the vendor rather than to the body whose data you want, and that body often publishes the records itself. Check the CNAME before assuming independence: four portals under four different government domains resolved to the same three addresses, each CNAME naming the same vendor, sharing one rate limit and one point of failure that the hostnames do not show.

Step 1: did anything come back, and is the target still there

Everything after this step assumes a server answered you. Three failure classes produce no HTTP conversation at all, they are the cheapest things that can be wrong, and they are the most expensive to miss, because every rung below will fail identically against them at increasing cost.

Symptom	What it usually is	Check
No status, no headers, no body; the client reports a DNS error	the hostname does not exist publicly	resolve it against two public resolvers you did not configure
TCP connects, then the TLS handshake fails	something in your own path is intercepting, or the name resolved to the wrong host	compare the resolved address across resolvers before touching the request
Connection refused or reset with no response	wrong port, dead host, or a network-level drop	as above, then check the site's current navigation
Your resolver returns an address in a private range, CGNAT (100.64/10) or 198.18.0.0/15	either the name is not public, or your path intercepts transparently	the two-answer test below. Do not stop on the address alone

Resolve the way the pre-flight section says, over DoH, against two providers. A local, corporate or captive resolver will happily answer for a name that does not exist publicly, and on an intercepted path it will do so in the name of whatever resolver you addressed. If two providers both return NXDOMAIN over DoH, the host does not exist and nothing below applies. One free query, and it can end the investigation.

A non-routable answer is a question, not a verdict. Transparent interception is common on corporate and VPN paths, and it works: your traffic still reaches the real origin.

DoH answers	Your traffic	Meaning	Do
NXDOMAIN from both	nothing arrives	the host does not exist	stop, re-target
a non-routable address, typically from 198.18.0.0/15	anything	your own proxy client answered (fake-ip mode), and the address says nothing about the host	re-resolve over DoH; if DoH gives a real address, use that and move on. If DoH agrees, treat DNS as unavailable and let a second egress decide
real address	nothing arrives, or the origin looks wrong	your path is breaking the connection	fix or bypass your path first, and do not blame the site
real address	real origin headers arrive (CDN POP ids, coherent ETags across repeats)	transparent interception that works	proceed, and record this vantage point as confounded: any verdict now needs a second egress

An answer proves less than absence does. NXDOMAIN proves the name is not public; a real address proves almost nothing, because behind a shared CDN a wildcard record answers for every name in the zone. Confirm a guessed hostname the way Step 0 describes, by asking whether the content names the host you asked.

A verdict belongs to a host and a path together. Measured on one vendor platform in the same minute from one egress: robots.txt answered 200 while / and /sitemap.xml answered 403. The 200 came from the CDN's own managed copy of robots.txt and never reached the origin. Before recording "this host blocks us", fetch a second path as a control, and prefer one the CDN cannot be answering for.

Then check the URL is still the one the site publishes. The most ordinary cause of "I pulled nothing useful" is not a defence, it is a stale address. Sites move search, catalogues and APIs onto vendor platforms and leave the old hostname to rot for years. Open the site's own current navigation and confirm the path you are using is the one it links to today.

Stop rule for this step. If the host does not resolve, or the site no longer publishes that path, you are finished. There is nothing to unblock.

Step 2: classify what came back

You are here only if a server answered. The single most expensive mistake now is reading every failure as "they detected me" and jumping to a browser with residential proxies. Match the symptom first, and read the body before you read the status.

What came back	Most likely cause	Do NOT	Go to
403, CDN error page, the edge identified as Cloudflare (server: cloudflare, cf-ray), no cf-mitigated header, and a blocked-page body rather than an interstitial	a rule that read your address or ASN. Nothing of yours was evaluated	change your client: a real browser can fail identically	Step 3, egress work
403 or 503, interstitial body (Just a moment...), cf-mitigated present, CSP naming a challenge host	a managed challenge. The edge will admit a good client, and the check clears itself given seconds	escalate straight to a browser Actor, see Rung 5 for what that actually did	Rung 1, then Rung 5
403 from an edge that is not Cloudflare, or one you have not identified yet	unclassified. The Cloudflare marker cannot appear here, so its absence carries no information	read "no cf-mitigated" as "my address was judged"	name the vendor first, see below
200, a full server-rendered page carrying a form with hidden state (__VIEWSTATE, __EVENTVALIDATION, a CSRF field), and no records	not a block and not a shell. The records are behind a POST you never issued	read the empty form as a defence, or send it to Rung 5 for a browser	the form test below
403 with an XML or JSON error body from a storage service (AccessDenied, RequestId, Server: AmazonS3 or equivalent)	an object that does not exist. Storage answers 403 rather than 404 for a missing key	shop for proxies	Step 1, the path is dead
403 only after N successful requests	rate, or a session that aged out	rotate IPs blindly	Rung 1
429, often with Retry-After	explicit rate limit, and they are telling you the number	ignore the header	Rung 1
A normal page plus an injected sensor script and edge cookies (incap_*, __cf*) set on the first hit	not a block. It is notice that the API behind this page is defended even though the page is not	assume the next layer is as open as this one	Rung 4, and carry those cookies (Rung 3)
200, small body, a mount point and a JS bundle (<div id="root">, /assets/index-*.js)	a single-page app shell. The records arrive after load	assume blocking; assume the URL matters	Rung 4
200, a rendered site whose record links point at another domain	a brochure. The data was never on this host	keep parsing this host	Step 1, re-target
200, tiny body, no scripts, a <meta http-equiv="refresh"> or a location assignment	a redirect stub. The real page is one hop away	parse this body for records	follow the target, then re-classify
200, a body of a few dozen bytes that is the refusal (Invalid Password, Not authorized, Missing parameter)	not a block and not a defence. An endpoint that wants arguments you did not send	treat the 200 as success, or send it to Rung 4: there is no JSON twin behind a CGI	the parameter test below
200, small body, a stock server welcome page (IIS Windows Server, Welcome to nginx)	a bare host or the wrong virtual host, not the application	read it as the site's answer	Step 1, re-target
A 2xx that is not 200, especially 202 with a near-empty body	not a success. Often an edge answering for an origin that did not, and frequently the site has moved	treat 2xx as "it worked"	compare loadedUrl with what you asked for, then Step 1
200, but fewer items than the browser shows	pagination cap, API limit, or a logged-out view	assume blocking	Rung 4
200 titled Page not found, with full site chrome	a soft 404	trust status, headers or size; only content shows it	Step 1, the path is dead
Worked yesterday, empty today, no error	the site changed markup or endpoints	escalate proxies	Rung 4, re-inspect

The first two rows are one status code and two different situations. A hard block and a managed challenge both arrive as 403 from the same CDN vendor, they route in opposite directions, and the header separates them. Both were measured on the same afternoon from one egress: a government portal answered a plain HTTP client with a bare 403 and no cf-mitigated, a vendor platform answered the same client with cf-mitigated: challenge and Just a moment.... Your diagnostic Actor cannot see either header until you unblind it, and until then both arrive as an empty item. The one-line fix is in "Making refusals visible" below; apply it before using this table.

Name the edge before you use its marker, because cf-mitigated is one vendor's header and absence is not evidence. On any edge that is not Cloudflare the header cannot appear, so "no cf-mitigated" is guaranteed and tells you nothing, while the rule that reads it happily returns "your address was judged" for every one of them. Identify the vendor from what did arrive:

Edge	What identifies it	What separates hard block from challenge there
Cloudflare	server: cloudflare, cf-ray. Measured on a refusal: 4,560-byte body, Sorry, you have been blocked	cf-mitigated first, then the body. Present means a managed challenge; absent plus a blocked-page body means your address. Absent with an interstitial body is still a challenge: custom rules and newer challenge widgets do not all set the header, so let the body overrule it
Akamai	Server-Timing: ak_p, X-Reference-Error, a body naming errors.edgesuite.net. Measured on a refusal: 384 bytes, Access Denied, and no Server and no Via header at all	not cf-mitigated, which never appears here. Treat the refusal as unclassified and let a second egress decide
anything else, or unidentified	whatever Server, Via, X-Cache and Server-Timing do carry	unknown. Say so, and route by the second egress rather than by an absent header

A refusal is a measurement, not a property of the site, and some refusals expire. The Akamai refusal above carried Server-Timing: cdn-cache; desc=HIT on a 403 whose own Cache-Control said max-age=0, meaning the error object was served from the edge cache and never reached the origin. Thirty-eight minutes later the same URL from the same address answered 302 and then 200 with 81,337 bytes, three times running. Nothing about the requester had changed. This is not a promise that refusals dissolve: an ASN or geo rule in a firewall is stable for as long as someone leaves it there, and re-measuring only confirms it. The point is the asymmetry of cost. One repeat request is free and it either upgrades a single observation into a stable fact or saves you a proxy purchase against a refusal that was never about you.

The parameter test, for an endpoint that answers 200 by refusing you. A missing argument and a credential you do not own look identical in the response, and they end in opposite places: one is Rung 3, the other stops the investigation at Step 0. Decide by where the value comes from, not by what it is called.

• The site itself hands it out to anyone who loads the page: a district code in a public dropdown, a build id in the HTML, a token the page fetches before its first search, a guest login printed in the UI. That is a parameter of a public route. Take it the way the page does, Rung 3.
• It is issued to a person, or it gates records the site does not otherwise publish. That is a credential you do not own, whatever the field is named. Stop and say so, and never guess, rotate or construct one.
• Unsure after one look at the page that issues it? Treat it as the second case and go to Step 0 to look for a published copy of the data. A source that requires an account almost always has a bulk or catalogue route that does not.

The form test, for a full page that carries a form instead of records. Server-rendered forms with hidden state are the default shape of government and utility record systems, and a GET-only crawler receives a valid 200 from one forever, with no error and no records. This is not the SPA shell row: there is no mount point and no bundle, the page is large, and the data is one POST away rather than one XHR away.

1. Issue the POST the form describes, carrying every hidden field it hands you (__VIEWSTATE, __VIEWSTATEGENERATOR, __EVENTVALIDATION and friends) plus the search field, with a Referer of the form page.
2. Read where the answer goes. If the response is a redirect to a URL carrying your query in its parameters, you have the record address rather than a session.
3. Then test that address cold, in a fresh client with no cookies and no Referer. This is the step that decides the whole job.

Measured on a county assessor's system: the form page answered 200 with 5,316 bytes and a __VIEWSTATE; the POST answered 302 to .../ParcelDetail.aspx?hdnParcel=<key>&hdnInstance=pcl7; and a cold curl of that URL, no cookies, no Referer, default user agent, answered 200 with 38,817 bytes of the actual record. One of the two redirect parameters turned out to be optional, so strip parameters one at a time before you build them into a crawl. A key the system does not know answered 200 with 747 bytes reading No record found, which is a third outcome to record rather than an empty row to drop.