Fitter — web data for AI agents

Fitter turns any website or API into structured JSON — declaratively. One JSON/YAML config describes where the data lives (HTTP request, headless browser, file, static value) and what to extract (JSON paths, CSS selectors, XPath). No code, no brittle scraping scripts.

🚀 Try it in your browser — the real engine compiled to WebAssembly: live examples, a visual config builder, no install.

Because configs are plain data, LLMs can author them. The built-in MCP server lets Claude Code, Claude Desktop, or any MCP client write and run scraping pipelines on your machine, on demand:

"Get the top 5 HackerNews stories with titles and scores" → the model authors a fitter config, validates it, runs it locally, and gets clean JSON back.

One engine, five ways to use it:

🤖 Fitter MCP	MCP server exposing fitter to Claude Code, Claude Desktop and any MCP client
🧠 Fitter Agent	AI-powered CLI: natural language → config → executed result
🖥 Fitter CLI	run configs locally for test/debug/home usage
📦 Fitter Lib	embed the engine in your own Go program
⚙️ Fitter	long-running service mode with scheduling & notifications

Why fitter for AI agents?

• Declarative & auditable — the agent produces a config you can read, save and re-run, not throwaway code
• Local-first — all fetching happens on your machine; no third-party scraping API, no keys, no per-request billing
• Batteries included — HTTP client, headless browser (Playwright/Chromium/Docker), JSON/HTML/XML/XPath/PDF parsing, pagination, cached references, host rate-limits — in a single static binary
• Reusable — what the agent authored today becomes tomorrow's cron job or service config

How to use Fitter_MCP

Fitter MCP is a Model Context Protocol server (stdio transport) which lets any MCP client — Claude Code, Claude Desktop, IDE assistants, custom agents — run Fitter configs and get structured JSON back.

Quick start (Claude Desktop — one click)

Download fitter-mcp-<os>-<arch>.mcpb from the release page and open it — Claude Desktop installs the server automatically.

Quick start (Claude Code)

# 1. get the binary: download fitter_mcp_<version>-<os>-<arch> from the release page
#    https://github.com/PxyUp/fitter/releases — or build it from source:
go build -o fitter_mcp ./cmd/mcp

# 2. register it once, available in every project
claude mcp add fitter -s user -- "$(pwd)/fitter_mcp"

Then just ask:

Get the top 5 HackerNews stories with titles and scores using fitter

The model calls fitter_config_reference, authors a config, optionally checks it with fitter_validate_config and executes it via fitter_run — all data fetching happens locally on your machine. For a ready-made pipeline try examples/config_morning_briefing.json:

Run examples/config_morning_briefing.json with fitter and give me the briefing

Register in Claude Desktop

{
  "mcpServers": {
    "fitter": {
      "command": "/path/to/fitter_mcp"
    }
  }
}

Browser support (Playwright)

The .mcpb bundle and native binary ship without browsers: HTTP, static and file connectors work out of the box, but browser configs (the playwright connector) need Playwright's browsers. A few ways to get them:

• On first use (native binary / .mcpb): set "install": true in the playwright connector — fitter downloads the driver + browser matching its built-in playwright-go version on first use (one-time, cached), so no separate install step is needed.
• Ahead of time (native, optional): to avoid the first-run download, install the browsers beforehand with the same playwright-go version fitter is built against (check go.mod, currently v0.6100.0):

  go run github.com/mxschmitt/playwright-go/cmd/playwright@v0.6100.0 install
  # Linux: append --with-deps to also install the required OS libraries
  

  The version must match go.mod exactly — playwright-go refuses to run against a mismatched driver. Then run configs without "install": true.
• Docker: use the ghcr.io/pxyup/fitter-mcp:playwright image, which bundles Chromium, Firefox and WebKit preinstalled (no "install": true needed).

Tools

Tool	Description
fitter_run	Run a Fitter config passed inline (JSON or YAML string) and return the extracted data as JSON. Accepts an optional input value available in the config via {{{FromInput=.}}} / {{{FromInput=json.path}}}
fitter_run_file	Same as fitter_run but reads the config from a local .json/.yaml file
fitter_run_url	Same as fitter_run but downloads the config from an HTTP(S) URL, e.g. a raw GitHub link
fitter_inspect_url	Fetch a URL and return a compact structure outline + candidate selectors/paths (gjson paths for JSON; repeated-element/list-row selectors for HTML) so the model authors a config first-try instead of guessing selectors and getting nulls. Detects client-rendered SPAs and can render them in a headless browser. Read-only — does not extract
fitter_validate_config	Validate a config without executing it (structure, response_type, connector data source, model). Useful while iterating on a config
fitter_config_reference	Return a condensed reference of the whole config format (connectors, parsers, model/field schema, placeholders, notifiers, references, limits) with working examples, so the model can author configs without external docs

The reference is also exposed as MCP resource fitter://config-reference for clients that support resources.

The config format is exactly the same as for Fitter_CLI: a top-level object with item (required), limits and references. Notifiers work too (the result is additionally pushed to http/telegram/redis/file/console); trigger_config and http_server are service-mode only and are ignored in MCP calls.

Remote / hosted mode (streamable HTTP)

By default fitter_mcp talks stdio. Pass --http to serve the streamable HTTP transport instead — for a shared team server, a container, or any remote deployment:

# serve MCP at http://<host>:8080/mcp (health probe at /healthz)
FITTER_MCP_AUTH_TOKEN=my-secret fitter_mcp --http :8080

# register the remote endpoint in Claude Code
claude mcp add --transport http fitter http://localhost:8080/mcp --header "Authorization: Bearer my-secret"

• --http <addr> (env FITTER_MCP_HTTP_ADDR) — listen address; stdio mode when empty
• FITTER_MCP_AUTH_TOKEN — when set, every /mcp request must send Authorization: Bearer <token>; without it the endpoint is unauthenticated, so bind to localhost or put it behind a proxy
• --stateless (env FITTER_MCP_STATELESS=true) — no per-session state, so replicas can sit behind a load balancer without sticky sessions
• --metrics-addr <addr> (env FITTER_MCP_METRICS_ADDR) — serve Prometheus metrics on a separate address (e.g. 127.0.0.1:9091); works in stdio mode too; empty disables. Exposes MCP request counts by method, tool call durations by tool and outcome, and outbound HTTP request durations by host, method and status. The endpoint is unauthenticated — bind it to localhost or a private interface

The server shuts down gracefully on SIGINT/SIGTERM.

Docker

A slim multi-arch image (linux/amd64 + linux/arm64) ships with every release:

# hosted HTTP mode
docker run --rm -p 8080:8080 \
  -e FITTER_MCP_HTTP_ADDR=:8080 \
  -e FITTER_MCP_AUTH_TOKEN=my-secret \
  ghcr.io/pxyup/fitter-mcp:latest

# or stdio mode, spawned by the MCP client
claude mcp add fitter -s user -- docker run --rm -i ghcr.io/pxyup/fitter-mcp:latest

The slim image contains only the fitter binary and CA certificates: server/static/file connectors work, browser connectors (chromium/docker/playwright) do not.

For browser-based configs use the playwright variant, which bundles Playwright with Chromium, Firefox and WebKit (matched to the playwright-go version fitter is built against, so no "install": true is needed in configs):

docker run --rm -i ghcr.io/pxyup/fitter-mcp:playwright        # stdio mode
# per-release tag: ghcr.io/pxyup/fitter-mcp:vX.Y.Z-playwright

It is built from Dockerfile.mcp-playwright; build with --build-arg PLAYWRIGHT_BROWSERS=chromium for a smaller Chromium-only image.

OAuth2 accounts in Docker

Both images ship fitter_cli, so the one-time OAuth2 login can run inside the container. Store the token on a volume mounted at /tokens (pre-created writable in the image) and share it with the MCP server:

# one-time login, device flow: no ports needed — open the printed url on any device
docker run --rm -it -v fitter-tokens:/tokens --entrypoint fitter_cli \
  ghcr.io/pxyup/fitter-mcp:latest \
  auth --provider github --client-id <ID> --client-secret <SECRET> --token-file /tokens/github.json

# or browser flow (device flow not enabled for the app): publish the callback port and
# bind on 0.0.0.0 so the published port reaches the listener; the browser still visits 127.0.0.1
docker run --rm -it -p 8988:8988 -e FITTER_AUTH_LISTEN=0.0.0.0 \
  -v fitter-tokens:/tokens --entrypoint fitter_cli ghcr.io/pxyup/fitter-mcp:latest \
  auth --provider github --client-id <ID> --client-secret <SECRET> --token-file /tokens/github.json

# then run the MCP server with the same volume; configs reference "token_file": "/tokens/github.json"
# stdio mode (spawned by the MCP client, no port):
docker run --rm -i -v fitter-tokens:/tokens ghcr.io/pxyup/fitter-mcp:latest
# hosted HTTP mode (MCP endpoint on 8080, like the run examples above):
docker run --rm -p 8080:8080 -v fitter-tokens:/tokens \
  -e FITTER_MCP_HTTP_ADDR=:8080 \
  -e FITTER_MCP_AUTH_TOKEN=my-secret \
  ghcr.io/pxyup/fitter-mcp:latest

Note: 8988 is only for the one-time browser-flow login; the MCP server itself needs no port in stdio mode and only 8080 in hosted HTTP mode.

Logged-in browser sessions in Docker

Browser sessions need the playwright image (the slim one has no browsers). The one-time headed login needs a display, so run it on the host, then bind-mount the session dir into the container (the image pre-creates a writable /sessions):

# on the host: log in once, save the session
fitter_cli browser-login --url https://example.com/login --storage-state ~/.fitter/sessions/example.json

# run the MCP server with the sessions dir mounted; configs reference "storage_state_file": "/sessions/example.json"
docker run --rm -i -v ~/.fitter/sessions:/sessions ghcr.io/pxyup/fitter-mcp:playwright

Use a bind mount (not a named volume): the container writes refreshed cookies back after every run, so the host copy stays current and can be re-extended with browser-login at any time.

The volume must stay writable for the server: rotated refresh tokens are written back on every refresh.

Environment variables

1. FITTER_PLUGINS - string[""] - path for plugins folder, same as the --plugins flag of Fitter/Fitter_CLI
2. FITTER_MCP_HTTP_ADDR - string[""] - listen address for remote mode, same as --http
3. FITTER_MCP_AUTH_TOKEN - string[""] - bearer token protecting the HTTP endpoint
4. FITTER_MCP_STATELESS - bool[false] - stateless HTTP transport, same as --stateless
5. FITTER_MCP_METRICS_ADDR - string[""] - Prometheus metrics listen address, same as --metrics-addr

Recipes

Complete, tested configs showing the main patterns. All of them run unchanged via Fitter_MCP (fitter_run_file), Fitter_CLI or the library — more in examples/.

Scrape a page that has no API, enrich from one that does

GitHub trending has no official API — scrape the HTML for repo slugs (html_attribute reads the href), then fan each one out into the GitHub REST API with {PL}:

examples/config_github_trending.json

{
  "item": {
    "connector_config": {
      "response_type": "HTML",
      "url": "https://github.com/trending",
      "server_config": { "method": "GET", "headers": { "User-Agent": "Mozilla/5.0 (fitter demo)" } }
    },
    "model": {
      "array_config": {
        "root_path": "article.Box-row h2 a",
        "length_limit": 5,
        "item_config": {
          "field": {
            "type": "string",
            "html_attribute": "href",
            "generated": { "model": {
              "type": "object",
              "connector_config": {
                "response_type": "json",
                "url": "https://api.github.com/repos{PL}",
                "server_config": { "method": "GET", "headers": { "User-Agent": "fitter-demo" } },
                "null_on_error": true
              },
              "model": { "object_config": { "fields": {
                "repo": { "base_field": { "type": "string", "path": "full_name" } },
                "stars": { "base_field": { "type": "int", "path": "stargazers_count" } },
                "language": { "base_field": { "type": "string", "path": "language" } }
              } } }
            } }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "api.github.com": 2 } }
}

[{"repo": "block/buzz", "stars": 6214, "language": "Rust"}, {"repo": "koala73/worldmonitor", "stars": 71179, "language": "TypeScript"}]

Join on a JSON field with an expression

When array items are objects, the join key lives inside them — pull it out with {{{FromExp=...}}} (expr-lang over fRes, the current item). Book search → author details, search query supplied via input:

examples/config_book_authors.json

"url": "https://openlibrary.org/authors/{{{FromExp=fromJSON(fRes).author_key[0]}}}.json"

./fitter_cli --path=examples/config_book_authors.json --input=dune

[{"title": "Dune", "year": 1965, "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}}]

Write results to a local file

The file_storage generated field turns fields into writes — top-5 crypto coins appended to a CSV, one row per item. Bare {{{json.path}}} placeholders read the current item; {HUMAN_INDEX} stamps the 1-based rank (items are processed in parallel, so appends land in completion order — sort by the rank column):

examples/config_crypto_csv.json

"file_storage": {
  "content": "{HUMAN_INDEX},{{{name}}},{{{current_price}}},{{{price_change_percentage_24h}}}\n",
  "file_name": "coins.csv",
  "path": "/tmp/fitter-report",
  "append": true
}

$ sort -n /tmp/fitter-report/coins.csv
1,Bitcoin,64778,-2.3
2,Ethereum,1881.01,-3.4
3,Tether,0.999265,0

Extract text from a PDF

response_type: "pdf" turns any fetched PDF into a JSON document — {"text": "...", "pages": ["..."], "total_pages": N} — so regular JSON paths (text, pages.0) and expressions work on it. The Bitcoin whitepaper, page count plus a trimmed intro:

examples/config_pdf.json

{
  "item": {
    "connector_config": {
      "response_type": "pdf",
      "url": "https://bitcoin.org/bitcoin.pdf",
      "server_config": { "method": "GET" }
    },
    "model": {
      "object_config": {
        "fields": {
          "total_pages": { "base_field": { "type": "int", "path": "total_pages" } },
          "intro": {
            "base_field": {
              "type": "string",
              "path": "pages.0",
              "generated": {
                "calculated": {
                  "type": "string",
                  "expression": "trim(fRes[:100]) + \"...\""
                }
              }
            }
          }
        }
      }
    }
  }
}

{"intro": "Bitcoin: A Peer-to-Peer Electronic Cash SystemSatoshi Nakamotosatoshin@gmx.comwww.bitcoin.orgAbstrac...", "total_pages": 9}

Way to collect information

1. Server - parsing response from some API's or http request(usage of http.Client)
2. Browser - emulate real browser using chromium + docker + playwright/cypress and get DOM information
3. Static - parsing static string as data

Format which can be parsed

1. JSON - parsing JSON to get specific information
2. XML - parsing xml tree to get specific information
3. HTML - parsing dom tree to get specific information
4. XPath - parsing dom tree to get specific information but by xpath
5. PDF - extracting text from PDF documents; the content is exposed as JSON {"text": "...", "pages": ["..."], "total_pages": N} so regular JSON paths like text or pages.0 work

Use like a library

go get github.com/PxyUp/fitter

package main

import (
	"fmt"
	"github.com/PxyUp/fitter/lib"
	"github.com/PxyUp/fitter/pkg/config"
	"log"
	"net/http"
)

func main() {
	res, err := lib.Parse(&config.Item{
		ConnectorConfig: &config.ConnectorConfig{
			ResponseType:  config.Json,
			Url:           "https://random-data-api.com/api/appliance/random_appliance",
			ServerConfig: &config.ServerConnectorConfig{
				Method: http.MethodGet,
			},
		},
		Model: &config.Model{
			ObjectConfig: &config.ObjectConfig{
				Fields: map[string]*config.Field{
					"my_id": {
						BaseField: &config.BaseField{
							Type: config.Int,
							Path: "id",
						},
					},
					"generated_id": {
						BaseField: &config.BaseField{
							Generated: &config.GeneratedFieldConfig{
								UUID: &config.UUIDGeneratedFieldConfig{},
							},
						},
					},
					"generated_array": {
						ArrayConfig: &config.ArrayConfig{
							RootPath: "@this|@keys",
							ItemConfig: &config.ObjectConfig{
								Field: &config.BaseField{
									Type: config.String,
								},
							},
						},
					},
				},
			},
		},
	}, nil, nil, nil, nil)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(res.ToJson())
}

Output:

{
  "generated_array": ["id","uid","brand","equipment"],
  "my_id": 6000,
  "generated_id": "26b08b73-2f2e-444d-bcf2-dac77ac3130e"
}

Use lib.ParseCtx(ctx, ...) to pass a context.Context: cancelling it aborts in-flight fetches (HTTP requests, headless browsers, docker containers) and applies deadlines end-to-end. lib.Parse is equivalent to lib.ParseCtx(context.Background(), ...).

How to use Fitter

Download latest version from the release page

or locally:

go run cmd/fitter/main.go --path=./examples/config_api.json

Arguments

1. --path - string[""] - path for the configuration of the Fitter
2. --url - string[""] - url for the configuration of the Fitter
3. --verbose - bool[false] - enable logging
4. --plugins - string[""] - path for plugins for Fitter
5. --log-level - enum["info", "error", "debug", "fatal"] - set log level(only if verbose set to true)

How to use Fitter_CLI

Download latest version from the release page

or locally:

go run cmd/cli/main.go --path=./examples/cli/config_cli.json

Arguments

1. --path - string[""] - path for the configuration of the Fitter_CLI
2. --url - string[""] - url for the configuration of the Fitter_CLI
3. --copy - bool[false] - copy information into clipboard
4. --pretty - bool[true] - make readable result(also affect on copy)
5. --verbose - bool[false] - enable logging
6. --omit-error-pretty - bool[false] - Provide pure value if pretty is invalid
7. --plugins - string[""] - path for plugins for Fitter
8. --log-level - enum["info", "error", "debug", "fatal"] - set log level(only if verbose set to true)
9. --input - string[""] - specify input value for formatting. Examples: --input=\""124"\" --input=124 --input='{"test": 5}'

./fitter_cli_${VERSION} --path=./examples/cli/config_cli.json --copy=true

fitter_cli auth — connect an OAuth2 account

One-time interactive login which stores a (refresh) token for the oauth2 connector config:

# device flow (default when the provider supports it): no callback, works headless
./fitter_cli_${VERSION} auth --provider github --client-id <ID> --client-secret <SECRET> --token-file ~/.fitter/tokens/github.json

# custom provider without preset
./fitter_cli_${VERSION} auth --auth-url https://.../authorize --token-url https://.../token --client-id <ID> --token-file ./token.json

Arguments:

1. --provider - preset with known endpoints: github|google|microsoft|gitlab|spotify
2. --client-id / --client-secret - OAuth2 app credentials (some device flows work without secret)
3. --token-file - where to store the received token (0600 permissions); reference the same path in oauth2.token_file
4. --flow - auto (device if available, else browser), device (visit a url + enter a code) or browser (localhost callback with PKCE, default port 8988 — register http://127.0.0.1:8988/callback as the app callback url)
5. --scopes - comma separated scopes
6. --auth-url/--token-url/--device-auth-url/--auth-style - endpoint overrides for providers without preset
7. --port - int[8988] - browser flow callback port (env FITTER_AUTH_PORT); with the default the callback url to register at the provider is http://127.0.0.1:8988/callback
8. --listen - browser flow bind address, default 127.0.0.1; set 0.0.0.0 inside a container so the published port reaches the listener (env FITTER_AUTH_LISTEN)
9. --redirect-url - callback url registered at the provider when it differs from the listen address, e.g. docker port mapping (env FITTER_AUTH_REDIRECT_URL)
10. --no-browser - only print the authorization url

Running inside Docker: see OAuth2 accounts in Docker.

After login the command prints the ready-to-use oauth2 config block. The connector refreshes the access token automatically and writes rotated refresh tokens back to the token file, so the login is needed only once.