Web Scraping

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Bulk fetching from hosts you do not own: how fast you may go, when a paid transport beats a direct fetch, and how to prove the fallback you are relying on has ever run.

Contents

• Iron Law: the rate is computed, not chosen
• Quota vs rate limit
• Verify the fallback before you rely on it
• Direct fetch or paid transport
• Vendor table
• Facts
• Red flags — STOP
• References

Iron Law: the rate is computed, not chosen

workers / sleep_seconds is the policy. A worker count with no arithmetic beside it is not a tuning decision — it is an unreviewed multiplication of load on somebody else's server, usually from an IP the institution owns and cannot swap. Raising 1 worker to 8 to "help the run finish faster" is not helpful if it is the run that gets the university's address throttled: the user then loses the whole corpus, not the hours you saved.

# ceiling: SEC EDGAR publishes 10 req/s and requires a declared User-Agent
# https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data (read 2026-09-08)
WORKERS, SLEEP_SECONDS = 8, 1.0
EFFECTIVE_RPS = WORKERS / SLEEP_SECONDS          # 8.0 req/s = 80% of the documented ceiling
assert EFFECTIVE_RPS <= 10.0

No published ceiling → say so with the URL and the date you looked, and hold at 1 req/s. "Not documented" is not "unlimited".

Quota vs rate limit

Name which one the target enforces, in a comment next to the worker count. The two have opposite corrections, so a client that has not named one has guessed.

	caps	more workers →
Rate limit	requests per unit time	hits the cap sooner per second; total budget untouched, throttled requests retry, job still completes
Quota	total requests per IP/key per period	does not reduce total requests — only exhausts the budget sooner

Under a quota, parallelising buys nothing on the binding constraint and brings the wall forward. Under a rate limit, parallelising up to the ceiling is exactly right.

A fact recorded in a module docstring but never connected to the worker count is a fact nobody applied. Put it where the decision is.

Verify the fallback before you rely on it

A declared fallback transport — proxy, unblocker, mirror, secondary API — is unverified until it has executed. Count it; do not read about it.

xan frequency -s transport data/output/manifest.csv
# direct,1538
# proxy,0        <- documented as required; has never once run

Zero uses forces one of three, stated explicitly:

1. Exercise it — force the fallback on a 5–10 row test, record the result, keep the dependency.
2. Downgrade the claim — mark it UNVERIFIED — 0 executions as of <date> everywhere it appears.
3. Delete it — dependency, credentials and docs together.

Leaving a never-run path documented as required is the worst option: it makes an untested branch look like a safety margin, so the run proceeds at a pressure the fallback was supposed to justify.

Direct fetch or paid transport

Situation	Use
Public data, documented ceiling, no anti-bot	Direct fetch at the computed rate. A proxy here buys nothing and bills per request.
Per-IP quota you will exhaust	Paid transport — it is the only thing that changes the binding constraint, because it changes the IP. More local workers cannot.
Anti-bot / CAPTCHA / fingerprinting	Paid unblocker. Rolling your own is a maintenance treadmill.
Geo-restricted content	Proxy with geo-targeting.
Data already collected by a vendor	Buy the dataset or archive dump; do not re-crawl.

Vendor table

Every figure below was read from the vendor's own page on 2026-09-08; URLs in References. Pricing and limits move — re-read before quoting them to anyone.

	Zyte API	Bright Data	ScrapingBee
Throughput model	Rate limit, RPM-based — 3,000 RPM standard, 10,000 RPM enterprise, per API key. Not a concurrency cap.	Per-zone concurrency + spend caps. Vendor claims no concurrency limit (SERP pricing FAQ); no canonical docs page states a number — treat as unverified.	Concurrency cap, per plan — 25 / 50 / 100 / 200 / 400 concurrent requests, Hobby → Business+.
Over-limit behaviour	HTTP 429, charged nothing, retry with backoff; official clients retry by default.	Per-zone auto-throttling on the target; spend limit is the real guardrail.	429/5xx from the target; failed attempts still burn credits unless mode=auto fails entirely (0 credits).
Client concurrency knob	AsyncZyteAPI(n_conn=...), CONCURRENT_REQUESTS in Scrapy; docs use ~15 as the worked example.	Zone config; raise via support.	Whatever your plan's cap is — cap per domain, not just globally.
Billing unit	Per successful response; tier depends on target + HTTP vs browser. Screenshots $0.002; auto-extraction $0.0004–$0.0016/type.	Per successful request: Unlocker/SERP/Crawl from $1/1k req; Browser API from $5/GB; Web Archive dump ~$0.001/page; datasets from $250/100k rec.	Credits: 1 plain, 5 JS render, 10 premium proxy no-JS, 25 premium+JS, 75 stealth. Plans $19–$599/mo for 75k–8M credits.
Entry price	PAYG on signup, $5 free credit, $100/mo spend limit.	Free tier on Unlocker/SERP/Scraper APIs.	1,000 free credits, no card.
Best at	Scrapy-native pipelines; explicit published rate limits you can compute against.	Breadth — archive corpus, prebuilt datasets, unlocker, all under one token. Already wired for FINRA/IAPD work.	Simple REST unblocking with a hard, legible concurrency number per plan.

Do not read the concurrency column as a target. It is the vendor's ceiling on their side; the target site's ceiling is a different number and is the one E7 measures against. ScrapingBee's own docs warn that running at the cap provokes 429/5xx from targets and burns credits on failures.

Facts

• SEC EDGAR publishes 10 requests/second and requires a declared User-Agent naming a contact address; it explicitly reserves the right to throttle and states it does not allow botnets or automated crawling outside that policy. A scraper that neither declares a UA nor computes its rate is not merely impolite — it is the case the policy names, and the block lands on the shared IP.
• Rate-limited Zyte requests cost nothing, so retrying with backoff is free; a hand-rolled client that treats 429 as a fatal error throws away a retry the vendor already priced at zero.
• ScrapingBee credits are not requests. JS rendering is 5×, premium proxy with JS is 25×, stealth is 75×. A 1M-credit plan is 1M pages only if nothing renders JS — sizing a job in "requests" against a credit allowance overstates capacity by up to 75×.
• Bright Data's "unlimited concurrency" appears in pricing/marketing copy, not in a docs page with a number. Quoting it as a verified limit is an unverified claim presented as fact; if a run depends on it, open a ticket and get the zone provisioned rather than assuming.
• A per-IP quota is the one case where local concurrency cannot help at all. Parallelising into a quota is the shape of work that looks like a 3× speedup in the log and is a 0× improvement on what the run can actually retrieve.

Red flags — STOP

About to	Why wrong	Do instead
Set max_workers=N with no EFFECTIVE_RPS beside it	The load change is unreviewed and unrecorded	Compute workers / sleep, name the ceiling and its URL
Raise workers because the job is slow	If the target enforces a quota this buys nothing and exhausts it sooner	Name quota vs rate limit first; that decides the fix
Treat "no documented rate limit" as permission	Absence of a published number is not consent	Hold at 1 req/s and record where you looked
Cite a fallback proxy as the answer to a quota	It may never have executed	xan frequency -s transport on the manifest; zero uses = untested
Quote a vendor concurrency or price figure from memory	These pages changed within the last year	Re-read the vendor page and date the citation
Size a ScrapingBee job in requests against a credit allowance	Credits are 1–75× per request	Multiply by the tier you will actually use
Call a paid unblocker for public, unprotected pages	Bills per request for something a direct fetch does free	Direct fetch at the computed rate
Run a bulk scrape without a 5–10 page test first	Repo rule: test before scaling	Fetch 5–10, inspect, then scale

References

Read on 2026-09-08:

• Zyte rate limits — https://docs.zyte.com/zyte-api/usage/rate-limit.html
• Zyte pricing (3,000/10,000 RPM, spend limits, per-feature costs) — https://docs.zyte.com/zyte-api/pricing.html
• Zyte parallelism guidance — https://docs.zyte.com/zyte-api/usage/optimize.html
• ScrapingBee pricing (plans, credits, concurrency) — https://www.scrapingbee.com/pricing/
• ScrapingBee credit multipliers and mode=auto — https://www.scrapingbee.com/documentation/
• Bright Data Web Unlocker overview — https://docs.brightdata.com/scraping-automation/web-unlocker/introduction
• Bright Data pricing — https://brightdata.com/pricing/web-unlocker
• SEC EDGAR fair-access policy — https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data

Local:

• ${CLAUDE_SKILL_DIR}/../../rules/ds-rate-limit-choice.md — constraint E7, normative.
• ${CLAUDE_SKILL_DIR}/references/bright-data-api.md — Bright Data API, auth, cost enforcement, endpoints.
• ${CLAUDE_SKILL_DIR}/references/bright-data-webarchive-api.md — Web Archive filters and parallel-poll harness.
• ${CLAUDE_SKILL_DIR}/references/bright-data-datasets-catalog.md — the 1,576-dataset marketplace.
• ${CLAUDE_SKILL_DIR}/references/bright-data-finra-sec-coverage.md — verified FINRA/IAPD coverage and dump costs.
• The sec-fetch skill owns SEC.gov specifically (declared-UA access that avoids the 403).