browser-data-scraper

Overview

Fetches static web pages, selects a supported traversal strategy (URL-parameter pagination, next-link following, recursive filtering, sitemap crawl, or a fixed URL list), extracts structured records incrementally to JSONL, deduplicates by content hash, and exports to the requested format. The JSONL file is written page by page so a partial or interrupted run keeps everything already collected. This skill reads static HTML only; it does not run a browser or execute JavaScript.

Workflow

<Definition - Mode Selection> When mode is auto, determine the mode from the input:

• Single URL -> paginate (reconnaissance picks the pagination strategy).
• Multiple URLs (comma or newline separated) -> url_list (extract the same fields from each).
• URL ending in sitemap.xml, or the user says "crawl" -> crawl (discover URLs from the sitemap, then scrape each). </Definition - Mode Selection>

<Definition - Incremental Storage> All extracted records are written to a JSONL file (one JSON object per line) in a single output directory.

Output directory resolution:

1. If output_path is provided, use it (create it if it does not exist).
2. Otherwise create scrapes/{domain}_{YYYYMMDD_HHMMSS}/ under the workspace directory and tell the user the full path.

All files for one job live in that directory: the raw JSONL, the cleaned export, and any log. JSONL filename pattern: {domain}_{YYYYMMDD_HHMMSS}.jsonl. Each line is a self-contained record. Source URL, page number, and extraction timestamp live under a _meta key on each record. This makes partial scrapes usable, lets deduplication run across pages, and enables resume: count existing lines to find the last completed page. </Definition - Incremental Storage>

<Definition - Content Hash> A SHA-256 hash of the record's sorted, serialized data fields, excluding _meta. Computed in run_python with hashlib. Used to deduplicate within and across pages, and to detect completion: when a page yields only known hashes, pagination has ended or looped. </Definition - Content Hash>

<Definition - Traversal Strategies> The supported strategies, their detection signals, and the priority order for falling back are catalogued in references/scraping-strategies.md. That file also lists the strategies that require a live browser and are out of scope for this skill. Read it during reconnaissance before choosing a strategy. </Definition - Traversal Strategies>

<Workflow - Intake description="Parse input, resolve mode and output location, and confirm the job." tools=[get_current_time, folder_create] triggers=["User provides URL(s) and a description of the data to extract"]

1. [Agent] Parse urls: split on commas and newlines, keep entries starting with http:// or https://, and note whether any is a sitemap. Validate: At least one valid URL extracted. If fails: Ask the user for a valid URL.

2. [Decide] Determine the mode. If mode is set explicitly, use it. If mode is auto, apply <Definition - Mode Selection>. Validate: Exactly one mode selected. If fails: Ask the user which mode they intend.

3. [Agent] Get the current time with get_current_time, then resolve the output directory and JSONL filename per <Definition - Incremental Storage>. Create the directory with folder_create if needed. Validate: Output directory exists and the JSONL filename is unique. If fails: Append a numeric suffix to the filename and retry.

4. [Ask user] Present the job summary using <Template - Job Summary> and ask whether to start reconnaissance. Validate: User confirms. If fails: Adjust parameters per feedback and re-present.

</Workflow - Intake>

<Workflow - Reconnaissance description="Fetch the first page, check robots.txt, analyze structure, and choose a supported strategy." tools=[url_fetch, run_python, file_read] triggers=["User confirms the job summary", "mode is paginate"]

1. [Agent] Fetch the first URL with url_fetch. Validate: A page body is returned with no 4xx or 5xx status. If fails: Report the HTTP error and suggest the user check the URL.

2. [Agent] Fetch {origin}/robots.txt with url_fetch and parse it in run_python. Check whether the target path is disallowed for a generic user-agent. Validate: Target path is not disallowed. If fails: Tell the user robots.txt disallows this path and ask whether to proceed (Rule 4).

3. [Agent] Parse the fetched HTML in run_python with beautifulsoup4. Identify the repeating elements (cards, rows, list items) that match target_data and the fields inside each. Validate: At least one repeating pattern with extractable fields is found. If fails: If the HTML is an empty shell, treat as JavaScript-rendered and stop per Rule 11. Otherwise ask the user where on the page the data appears.

4. [Agent] Read references/scraping-strategies.md and detect the pagination mechanism against its detection signals, in priority order. Validate: One supported strategy is detected, or the page is confirmed single-page. If fails: Ask the user how additional pages are reached.

5. [Agent] Estimate volume: count records on page 1, look for a total-count indicator (for example "1-20 of 450"), and compute estimated pages. Validate: An estimate is produced (approximate is fine).

6. [Ask user] Present findings with <Template - Site Analysis> and ask whether to proceed with the recommended strategy. Validate: User approves the strategy. If fails: Adjust the strategy per feedback.

</Workflow - Reconnaissance>

<Workflow - Preview Extraction description="Extract page 1, write it to JSONL, and confirm data quality with the user." tools=[url_fetch, run_python, run_python_with_write] triggers=["User approves the reconnaissance findings"]

1. [Agent] Extract records from the page-1 HTML in run_python using the approved strategy: read each repeating element, pull the target fields, and build a list of objects with consistent keys. Validate: At least one record with the expected fields is extracted. If fails: Adjust the selectors. If still empty, fall back to the next supported strategy (Rule 9).

2. [Agent] Write the page-1 records to the JSONL file with run_python_with_write, adding _meta (source_url, page, extracted_at, hash) to each record per <Definition - Incremental Storage> and <Definition - Content Hash>. Validate: The JSONL file exists and its line count equals the records extracted. If fails: Recompute and rewrite the file.

3. [Ask user] Show 3 to 5 sample rows with <Template - Sample Extraction> and ask whether to proceed with the full scrape. Validate: User confirms data quality and field mapping. If fails: Adjust extraction logic, re-extract page 1, and re-present.

</Workflow - Preview Extraction>

<Workflow - Full Extraction description="Walk the remaining pages with the approved strategy, deduplicating and checkpointing each page." tools=[url_fetch, run_python, run_python_with_write, file_read] triggers=["User approves the preview extraction"]

1. [Agent] Initialize state from the JSONL already on disk: read it with file_read (or count lines in run_python) to load the set of existing hashes and the last completed page. Set consecutive_empty = 0. Validate: The starting page and hash set reflect what is already in the file (resume-safe). If fails: Rebuild the hash set by re-reading the JSONL.

2. [Agent] For each page from the next page up to max_pages, one page per iteration: a. Build or find the next page's URL per the approved strategy: increment the page or offset parameter (URL manipulation), or read the next-link href from the current page's HTML (next-link following), or run the next sub-query (recursive filtering). b. Wait 1 to 2 seconds, then fetch the page with url_fetch. c. Parse and extract records in run_python using the same logic as page 1. d. Compute each record's hash; skip records whose hash is already known. e. Append the new records to the JSONL with run_python_with_write and update the hash set. f. Check termination: if the page yielded only known hashes or no records, increment consecutive_empty; 3 consecutive such pages ends the loop. If url_fetch returns 429 or a block page, stop and inform the user (Rule 6). Validate: After each page the JSONL grows, or a termination condition fires. If fails: Log the failing page and error, fall back to the next supported strategy (Rule 9); if all are exhausted, go to <Workflow - Error Recovery>.

3. [Agent] Log the extraction summary (pages scraped, records extracted, duplicates skipped, termination reason). Validate: The summary counts match the JSONL line count. If fails: Recount from the file.

</Workflow - Full Extraction>

<Workflow - Clean and Export description="Deduplicate, normalize, export to the chosen format, and present results." tools=[run_python, run_python_with_write, open_in_session_tab] triggers=["Full extraction completes or stops with partial results"]

1. [Agent] Load the JSONL in run_python and clean: run a full-file dedup pass, standardize dates to ISO 8601, strip whitespace from string fields, drop the _meta key from export records (leave the JSONL intact as source of truth), and sort by the first detected field or a user-specified column. Validate: Cleaned record count is less than or equal to the raw count, with no empty rows. If fails: Report the anomaly and keep the raw JSONL.

2. [Decide] Export per output_format with run_python_with_write:

   • jsonl -> copy the cleaned records (without _meta) to the final path.
   • csv -> write a header row plus data rows using the standard-library csv module.
   • xlsx -> write with XlsxWriter (header row, auto-sized columns). Validate: The output file exists and is non-empty. If fails: Fall back to CSV.

3. [Agent] Write the export as {output_dir}/{domain}_{YYYYMMDD_HHMMSS}_clean.{ext} and open both the export and the raw JSONL with open_in_session_tab. Validate: Files open without error. If fails: Report the file paths so the user can open them manually.

4. [Agent] Present the final summary with <Template - Final Summary>. Validate: The summary matches the actual file contents.

</Workflow - Clean and Export>

<Workflow - URL List Mode description="Extract the same fields from a fixed list of URLs, one fetch each." tools=[url_fetch, run_python, run_python_with_write] triggers=["mode is url_list", "Called from Crawl Mode with a filtered URL list"]

1. [Agent] Parse the URL list from urls (comma separated, newline separated, or an array). Validate: At least 2 valid URLs, unless called from Crawl Mode. If fails: Suggest paginate mode if only one URL was given.

2. [Agent] Fetch the first URL, parse the HTML, and identify the target fields. Validate: Fields matching target_data are identified. If fails: Ask the user to describe where the data appears, or stop per Rule 11 if the shell is empty.

3. [Ask user] Show the detected fields and a sample from URL 1 and ask whether this mapping fits all URLs. Validate: User confirms. If fails: Adjust and re-present.

4. [Agent] For each remaining URL: wait 1 to 2 seconds, fetch, extract the fields, and append to the JSONL with the source URL in _meta. Log URLs that yield no data and continue. Validate: Record count grows, or the URL is logged as skipped. If fails: Log the failed URL and continue; report all failures at the end.

5. [Agent] Proceed to <Workflow - Clean and Export>.

</Workflow - URL List Mode>

<Workflow - Crawl Mode description="Discover URLs from a sitemap, filter them, then scrape each as a URL list." tools=[url_fetch, run_python] triggers=["mode is crawl", "URL ends in sitemap.xml"]

1. [Agent] Fetch the sitemap URL with url_fetch and parse the URL entries in run_python with lxml or beautifulsoup4. If the input is a page rather than a sitemap, extract same-domain links from the fetched HTML instead. Validate: At least one URL discovered. If fails: Ask the user for a sitemap URL or which links to follow.

2. [Agent] Filter discovered URLs: remove duplicates and non-content paths (login, logout, privacy, terms), keep the same domain unless the user says otherwise, and limit to max_pages. Validate: The filtered list has at least 1 URL and is within max_pages. If fails: Relax or tighten filters and re-run.

3. [Ask user] Present the crawl plan with <Template - Crawl Plan> and ask whether to scrape the filtered pages. Validate: User approves. If fails: Adjust filters per feedback.

4. [Agent] Hand the filtered list to <Workflow - URL List Mode> starting at step 2.

</Workflow - Crawl Mode>

<Workflow - Error Recovery description="Recover partial results after a mid-scrape failure and offer next steps." tools=[run_python, file_read] triggers=["Extraction stops on a block, timeout, or strategy exhaustion"]

1. [Agent] Count the records already written to the JSONL with file_read or run_python. Validate: The JSONL exists and holds at least one record. If fails: Report that no data was recovered.

2. [Agent] Proceed to <Workflow - Clean and Export> with the partial data, noting the stop page and reason in the summary. Validate: A cleaned export is produced from the partial JSONL.

3. [Ask user] Report the recovered count and offer: keep as-is, resume from the next page, or try a different strategy. Validate: User selects an option. If fails: Re-summarize and re-ask. On resume, return to <Workflow - Full Extraction> step 1 (its resume logic picks up from the JSONL). On a different strategy, return to <Workflow - Reconnaissance> step 4.

</Workflow - Error Recovery>

<Template - Job Summary> Scraping Job

• Mode: [mode]
• URL(s): [list]
• Target data: [target_data]
• Max pages: [max_pages]
• Output format: [output_format]
• Output directory: [output_dir]

Ready to start reconnaissance? </Template - Job Summary>

<Template - Site Analysis> Site Analysis

• Data pattern: [description of repeating elements]
• Fields detected: [list]
• Pagination type: [detected strategy]
• Estimated items: [count] across about [pages] pages
• Recommended strategy: [strategy name]
• robots.txt: [allowed / disallowed / not found]

Does this look right? Should I proceed with [strategy]? </Template - Site Analysis>

<Template - Sample Extraction> Sample Extraction (Page 1)

Field1	Field2	Field3
...	...	...

Records on page 1: [N] Estimated total: [N x pages]

Does this data look correct? Should I proceed with the full scrape? </Template - Sample Extraction>

<Template - Crawl Plan> Crawl Plan

• URLs discovered: [total]
• After filtering: [filtered count]
• Sample: [first 10 URLs]

Proceed with scraping these [N] pages? </Template - Crawl Plan>

<Template - Final Summary> Scraping Complete

• Source: [URL(s)]
• Pages scraped: [N]
• Records extracted: [total clean]
• Duplicates removed: [count]
• Fields: [column list]
• Output directory: [output_dir]
• Export file: [filename]
• Raw JSONL: [filename] </Template - Final Summary>