Foresea: Autonomous Prediction Market Intelligence & LLM Rationale Benchmark

Foresea (foresea.ink) is an autonomous prediction market intelligence platform and empirical research framework. It combines real-time probability forecasting across Polymarket and Kalshi, statistical edge discovery, multi-model shadow trading tournaments, an autonomous 19-tool ReAct execution agent, and a public Model Context Protocol (MCP) server.

The repository also serves as the artifact for research studying how explicit rationale instructions, evidence injection, and reasoning structures affect LLM forecasting behavior and calibration on Metaculus-style questions.

---

📑 Table of Contents

• Live Platform & Web Views
• Edge Board & Autonomous Shadow Trading Desk
• 5-Minute Crypto Micro-Markets
• Production Deployment & Infrastructure
• Using the API & Endpoints
• Agent: Automated Intelligence Layer & ReAct Tools
• Model Context Protocol (MCP) Integration
• Custom Integrations & Ecosystem Tools
• Trading Execution & KMS Guardrails
• Repository Contents & Prompt Variants
• Foresea Autoresearch
• Reproducing Core Outputs & SQL Analytics
• Quality Checks & Development

---

Live Platform & Web Views

Deployed on Google Cloud Run with high-concurrency scaling, streaming responses, and continuous delivery:

https://foresea.ink

Core Web Applications & Routes

• Market Desk (/): Real-time landing view with market radar, model-vs-market gap highlights, and interactive walkthroughs.
• Forecasting Chat (/chat, /chat/:id): Conversational interface with streaming rationale, news evidence citation, and multi-turn market analysis.
• Edge Board (/edge, /edge/markets): Ranked live Foresea-vs-market pricing discrepancies across Polymarket and Kalshi, backed by calibration and lead-time scores.
• Mark-to-Market View (/edge/mtm): Continuous mark-to-market valuation and PnL tracking for resolved and open market predictions.
• Agentic Trading Desk (/edge/agentic): Multi-model autonomous trading tournament. Independent $10,000 shadow accounts for each model (Gemma 4 26B, Qwen 3.8 27B, GPT-OSS 120B, GLM 5.3, GLM 5.3 Flash, DeepSeek V4 Flash, MiniMax M3, Llama 3.3 70B) executing real-time paper trades, tracking equity curves, and logging hourly cycle health.
• Trading Terminal (/trade): Non-custodial, client-side order preview and execution terminal with Cloud KMS envelope encryption for Polymarket and Kalshi credentials.
• Watchlist (/watchlist): Follow specific markets with automated daily digest emails.
• Public Forecasts (/forecast/:share_id): Shareable forecast permalinks with rationale cards and provenance.

# Health check
curl https://foresea.ink/health

# Single-record prediction
curl -X POST https://foresea.ink/predict \
  -H "Content-Type: application/json" \
  -d '{
    "question": "Will X happen by date Y?",
    "question_type": "binary",
    "description": "Context here.",
    "news_articles": [],
    "attach_evidence": true,
    "evidence_top_k": 5,
    "market_platform": "Polymarket",
    "market_probability": 0.42,
    "variant": "variant0_neutral_baseline"
  }'

When attach_evidence is true and no news_articles are supplied, /predict fetches and ranks current news evidence from GDELT, Google News RSS, and Stooq by default, injects it into the model prompt, and returns the selected evidence_articles with the forecast. Supplying news_articles skips automatic retrieval and uses the caller-provided evidence.

The response includes both the forecast and the evidence used by the model:

{
  "question_type": "binary",
  "predicted_answer": "Yes",
  "confidence": 0.86,
  "options": [],
  "range_forecast": null,
  "rationale": "Model-generated explanation for the forecast.",
  "model_rationale": "Model-generated explanation for the forecast.",
  "variant": "variant0_neutral_baseline",
  "model_key": "gpt-oss-120b",
  "evidence_sources": [
    {
      "source": "Reuters",
      "title": "Article headline",
      "url": "https://example.com/article",
      "publish_date": "2026-05-29T00:00:00Z",
      "relevance_score": 0.82
    }
  ],
  "evidence_articles": [
    {
      "title": "Article headline",
      "summary": "Cleaned article summary.",
      "source": "Reuters",
      "url": "https://example.com/article",
      "publish_date": "2026-05-29T00:00:00Z",
      "relevance_score": 0.82,
      "search_query": "query used for retrieval"
    }
  ],
  "evidence_error": null,
  "market_analysis": {
    "platform": "Polymarket",
    "market_url": "https://example.com/market",
    "outcome": "Yes",
    "market_probability": 0.42,
    "model_probability": 0.86,
    "edge": 0.44,
    "stance": "model_above_market",
    "summary": "Foresea is 44 percentage points above the market on Yes."
  }
}

Use evidence_sources when a client only needs the source list and links. Use evidence_articles when a client needs the article-level details that were attached to the model prompt. rationale and model_rationale are generated by gpt-oss-120b and explain why the model chose its answer and confidence. When market_probability is supplied, market_analysis is computed deterministically from the model probability and the market-implied probability.

Edge Board & Autonomous Shadow Trading Desk

Foresea continuously evaluates state-of-the-art LLMs against real-world prediction markets, tracking statistical edge, calibration accuracy, and shadow portfolio performance across three dedicated views:

1. Live Markets Disagreement (/edge or /edge/markets)

• Real-Time Edge Ranking: Quantifies the statistical gap between model forecasts and live market pricing across Polymarket and Kalshi: $$\text{Edge} = P_{\text{model}} - P_{\text{market}}$$
• Sizing & Recommendation: Computes Fractional Kelly allocations and categorizes actionable trades (buy_yes, buy_no, hold).
• Calibration & Resolution: Backed by hourly snapshots committed to static/track_record_live.json, calculating Brier scores, Expected Calibration Error (ECE), and lead-time skill once markets resolve.

2. Mark-to-Market Evaluation (/edge/mtm)

• Continuous Portfolio Valuation: Evaluates paper positions at live bid/ask quotes rather than relying on eventual settlement alone.
• Unrealized vs. Realized PnL: Provides an institutional-grade view of trade trajectory and volatility exposure across open horizons.

3. Agentic Autonomous Trading Tournament (/edge/agentic)

• Multi-Model Tournament: Evaluates 8 distinct LLMs acting as autonomous portfolio managers:
  • Gemma 4 26B (gemma-4-26b-a4b-it)
  • Qwen 3.8 27B (qwen3-8-27b)
  • GPT-OSS 120B (gpt-oss-120b)
  • GLM 5.3 (glm-5-3) & GLM 5.3 Flash (glm-5-3-flash)
  • DeepSeek V4 Flash (deepseek-v4-flash)
  • MiniMax M3 (minimax-m3)
  • Llama 3.3 70B (llama-3.3-70b-instruct)
  • Baseline: crowd-follow (no-LLM consensus control)
• Independent $10,000 Portfolios: Each model starts with a simulated $10,000 cash balance and autonomy to scan markets, evaluate orderbooks, and execute Immediate-or-Cancel (IOC) paper orders.
• Equity Curves & Cycle Health: Renders cumulative returns ($10,000 reinvested), drawdown metrics, trade logs, and hourly cycle execution health (tracking provider availability, token budgets, and trade triggers).

5-minute crypto markets

The local crypto micro-market model in src/analyzing_llm_rationale/crypto_5m.py is built for 5-minute UP/DOWN markets where the goal is profitable selective trading, not constant action. It combines:

• shrunken-drift lognormal moneyness pricing,
• AR(1) return forecasting with EWMA volatility,
• fixed or adaptive logistic ML features from momentum, reversal, volatility regime, range position, and volume imbalance.

Each forecast returns predicted_outcome, probability_up, component_probabilities, model-vs-market edge, and a fee-aware strategy. The strategy only recommends a trade when net expected value clears fees and the configured no-trade threshold.

.venv/bin/python scripts/crypto_5m_backtest.py \
  --benchmark \
  --symbols BTC,ETH,SOL \
  --days 1 \
  --max-candles 1600 \
  --lookback-minutes 60 \
  --horizon-minutes 5 \
  --market-probability 0.50 \
  --fee-bps 2 \
  --ml-modes fixed,adaptive \
  --edge-thresholds 0,0.01,0.03,0.05,0.08 \
  --selection-fraction 0.6 \
  --folds 4 \
  --training-window 120 \
  --max-rows 80 \
  --benchmark-log data/crypto_5m_benchmark_runs.jsonl

Use fold_aggregate and evidence_quality before risking capital. If selection is unstable or holdout PnL is weak, the correct profitable action is to abstain. --benchmark-log appends a compact JSONL record for tracking whether the selected threshold and model mode keep working across benchmark runs. Resolve completed markets against Binance candles:

.venv/bin/python scripts/crypto_5m_backtest.py \
  --resolve \
  --symbol BTCUSDT \
  --target-price 62400.52 \
  --start-time-ms 1780000000000 \
  --horizon-minutes 5 \
  --predicted-outcome down

The resolver returns pending before expiry and resolved afterward with actual_outcome, resolved_price, and prediction_correct.

Record and resolve paper signals over time:

.venv/bin/python scripts/crypto_5m_backtest.py \
  --paper-signal \
  --symbol BTCUSDT \
  --market-probability 0.50 \
  --fee-bps 2 \
  --signal-log data/crypto_5m_signal_log.jsonl

.venv/bin/python scripts/crypto_5m_backtest.py \
  --resolve-signal-log \
  --signal-log data/crypto_5m_signal_log.jsonl

.venv/bin/python scripts/crypto_5m_backtest.py \
  --signal-summary \
  --signal-log data/crypto_5m_signal_log.jsonl \
  --min-resolved-trades 200 \
  --min-total-pnl 0 \
  --min-hit-rate 0.53

.venv/bin/python scripts/crypto_5m_backtest.py \
  --paper-loop \
  --symbols BTC,ETH,SOL \
  --iterations 12 \
  --sleep-seconds 60 \
  --market-probability 0.50 \
  --fee-bps 2 \
  --signal-log data/crypto_5m_signal_log.jsonl

The signal log is the running dataset for model improvement: each record stores the forecast, recommendation, later actual_outcome, correctness, and pnl_per_contract for actual buy_up/buy_down paper trades. Use --signal-summary to audit whether resolved paper trades are positive after fees; trade_ready stays false until the configured trade count, PnL, and hit rate thresholds are met. Use --dry-run with --paper-loop to preview signals without writing the log.

Production Deployment Notes

Production is served from the custom domain:

https://foresea.ink

The Cloud Run service name, project ID, and region are set at deploy time via gcloud run deploy.

Required runtime environment:

• SCADS_AI_API_KEY: Secret Manager secret used by hosted model calls.
• MODEL_DEVICE=cpu: production Cloud Run runs the CPU image.
• CUSTOM_DOMAIN=foresea.ink: redirects *.run.app requests to the public domain.
• GOOGLE_CLIENT_ID: Google OAuth web client ID used by /auth/config.
• GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET: GitHub OAuth app credentials. The OAuth app's callback URL must be the site origin (e.g. https://foresea.ink/). When unset, the "Continue with GitHub" button is hidden and /auth/github returns 503. Sign-in also works with Google and email/password.
• SESSION_SECRET: long random string used to sign browser session JWTs and derive domain-separated, non-reversible references for authenticated analytics. Rotating it starts a new attribution cohort; it never exposes account emails.

The OAuth client must allow these JavaScript origins:

https://foresea.ink
https://www.foresea.ink
https://<cloud-run-service-url>.run.app

To update non-secret environment variables without replacing the existing SESSION_SECRET, use --update-env-vars:

gcloud run services update <service-name> \
  --region <region> \
  --project <project-id> \
  --update-env-vars MODEL_DEVICE=cpu,CUSTOM_DOMAIN=foresea.ink,GOOGLE_CLIENT_ID='<your-google-client-id>'

Verify the deployed auth config and health endpoint:

curl https://foresea.ink/auth/config
curl https://foresea.ink/health

Scaling and caching

The server is built to scale horizontally on Cloud Run:

• Authentication supports Google One-Tap and email/password (/auth/register, /auth/login). Passwords are stored as salted PBKDF2-HMAC-SHA256 hashes; accounts live in Cloud Datastore.
• Caching and rate limiting use Redis when REDIS_URL is set, so they are shared across instances; otherwise they fall back to per-instance in-memory state and fail open. /predict (non-personalised requests), evidence retrieval, and /extract URL fetches are cached; public GETs send Cache-Control.

Var	Default	Description
REDIS_URL	unset	Memorystore/Redis URL. Shares cache + rate limits across instances.
PREDICT_CACHE_TTL	600	Cache TTL (s) for non-personalised /predict responses. 0 disables.
EVIDENCE_CACHE_TTL	900	Cache TTL (s) for evidence retrieval.
EXTRACT_CACHE_TTL	3600	Cache TTL (s) for /extract URL fetches.
LOCAL_CACHE_MAX	1024	Max entries in the in-memory fallback cache.
SEARXNG_URL / TAVILY_API_KEY / SERPER_API_KEY / BRAVE_API_KEY	unset	Enable web search as an evidence source. A self-hosted SearXNG is preferred when set, then Tavily, Serper, Brave. Tavily/Serper have free no-card tiers. When none is set, evidence comes from GDELT, Google News, and RSS.
NEWSAPI_KEY	unset	Enables NewsAPI as an evidence source.

Live track record

GET /track-record serves the public forecast track record. The heavy tick loop does not run on Cloud Run: .github/workflows/track-record-tick.yml runs hourly on GitHub Actions, updates data/track_record_store.json as the source-of-truth entity store, writes the public aggregate to static/track_record_live.json, and commits both files back to main. At runtime, Cloud Run fetches the committed aggregate from raw GitHub, falling back to the bundled file and then the static backtest in static/track_record.json.

The Action discovers short-to-medium-horizon Polymarket/Kalshi markets in separate close-date bands (2-7, 7-14, 14-30, 30-60 days by default) and calls /predict once per newly snapshotted market/model. If /predict is protected, set the GitHub secret PREDICT_API_KEY; no server-side /track-record/tick endpoint is required. TRACK_RECORD_TOKEN is optional and only enables the agent-enrolled market bridge.

The default scheduled forecast job is deliberately cost-capped: it runs every 6 hours, snapshots at most 2 markets per venue, and forecasts only gpt-oss-120b plus the no-LLM crowd-follow baseline. Use the manual workflow dispatch input reforecast_each_tick=1 for a one-off full refresh instead of forcing every scheduled run to reforecast all open markets.

The homepage market desk uses GET /radar, which is derived from static/track_record_live.json and its edge_board. Radar highlights current model-vs-market gaps and keeps the first screen fast by reusing the committed track-record aggregate instead of scanning venues on every page load.

Raise the Cloud Run throughput ceiling (no idle cost while min-instances=0):

gcloud run services update analyzing-llm-rationale --region us-central1 \
  --max-instances 20 --concurrency 40 --memory 1Gi

For the lowest-cost public deployment, keep the service on request-only CPU, scale to zero, and cap burst scale-out. This is the profile used by the deploy workflow. Startup CPU boost stays enabled because it reduces cold-start latency without keeping an idle instance warm:

gcloud run services update analyzing-llm-rationale \
  --region us-central1 \
  --project brave-drive-471109-d9 \
  --cpu 1 \
  --memory 512Mi \
  --min-instances 0 \
  --max-instances 3 \
  --concurrency 20 \
  --timeout 180 \
  --cpu-throttling \
  --cpu-boost \
  --update-env-vars INTERACTIVE_DEFAULT_MODEL=gemma-4-26b-a4b-it,INTERACTIVE_MAX_TOKENS=384,CHAT_PROVIDER_TIMEOUT_S=15,CHAT_PROVIDER_MAX_RETRIES=0,EVIDENCE_TIMEOUT_S=6,EVIDENCE_MAX_CONCURRENCY=4

Measure deployed forecast latency after each runtime change:

py scripts/measure_forecast_latency.py \
  --url https://foresea.ink \
  --mode stream \
  --models minimax-m3 \
  --runs 3 \
  --no-attach-evidence \
  --max-tokens 384

If cold starts still dominate, raise --min-instances to 1 as an explicit latency/cost tradeoff.

Market search runs in-process in the main API. The optional Go marketd microservice is build/test-only in GitHub Actions and is not deployed to Cloud Run by default.

Artifact Registry retention

CI pushes commit-tagged Docker images to Artifact Registry on every deploy. Keep the docker repository cleanup policy active so old images do not accumulate:

gcloud artifacts repositories set-cleanup-policies docker \
  --location us-central1 \
  --project brave-drive-471109-d9 \
  --policy infra/artifact-registry-cleanup-policy.json \
  --no-dry-run

The policy deletes images older than 7 days, keeps the newest 5 versions per package, and always keeps the main tag.

Docker builds run in GitHub Actions, not Cloud Build; no Cloud Build trigger or staging bucket is required for the normal deploy path.

Once max-instances > 1, provision Memorystore for Redis (billable) and set REDIS_URL so rate limiting and caching stay correct across instances:

gcloud services enable redis.googleapis.com vpcaccess.googleapis.com compute.googleapis.com
gcloud redis instances create foresea-cache --size=1 --region=us-central1 --tier=basic
gcloud compute networks vpc-access connectors create foresea-vpc \
  --region=us-central1 --range=10.8.0.0/28
gcloud run services update analyzing-llm-rationale --region us-central1 \
  --vpc-connector foresea-vpc \
  --update-env-vars REDIS_URL=redis://<instance-host>:6379

Using the API

See additional Kalshi and Polymarket endpoints for historical data, account pagination, order management and native exchange streams.

The public Cloud Run API is the easiest integration target. It accepts forecasting questions and returns a typed forecast, model rationale, and optional evidence articles. It is built for resolvable forecasts, not general Q&A.

Endpoints & Web Routes

Public Web Views

• GET /: landing desk, real-time market radar, and interactive workflow demo.
• GET /chat, GET /chat/{id}: conversational forecast studio with streaming rationales and evidence.
• GET /edge, GET /edge/markets: live market edge board with statistical gap rankings and Kelly sizing.
• GET /edge/mtm: mark-to-market performance of open positions across prediction venues.
• GET /edge/agentic: multi-model autonomous trading tournament, equity curves, cycle health, and paper trade tape.
• GET /trade: non-custodial trading terminal for Polymarket and Kalshi with envelope encryption.
• GET /watchlist: tracked favorite markets with daily digest notifications.
• GET /forecast/{share_id}: public read-only forecast share permalink.
• GET /embed/forecast/{share_id}: lightweight iframe-embeddable forecast widget.
• GET /widget.js: drop-in web component <foresea-card> for publishing live forecasts.

Core Forecasting & Intelligence API

• GET /health: service health check and operational status.
• POST /predict: public probability prediction endpoint with optional evidence retrieval.
• POST /agent/analyze: orchestrated end-to-end analysis of a live market with custom skills and ReAct loops.
• GET /agent/scan: venue scanner identifying mispriced markets ranked by statistical edge.
• GET /radar: homepage market desk payload derived from the live track record.
• GET /track-record: public live track record and historical calibration statistics.
• GET /track-record/digest: shareable markdown summary of the live track record.
• GET /pr-agent: opt-in agent-to-agent outreach packet for Foresea discovery.

Venue & Market Data

• GET /markets/polymarket: fetch live normalized Polymarket quotes, orderbooks, and liquidity data.
• GET /markets/kalshi: fetch live normalized Kalshi quotes, strike ranges, and ticker metadata.

Model Context Protocol (MCP)

• GET /mcp/: public remote Model Context Protocol (Streamable-HTTP) endpoint.
• GET /.well-known/mcp/server.json: public MCP discovery manifest.

Analytics, Sharing & Account Sync

• POST /analytics/visit: privacy-preserving page visit tracking (linked only to non-reversible references).
• POST /analytics/event: funnel event recording (forecast_completed, watchlist_add, share_created, digest_sent).
• GET /analytics/events/summary: 30-day aggregate product analytics summary.
• POST /forecasts/share: generate an explicit public forecast share ID.
• GET|POST /chat/conversations: cloud conversation sync for authenticated users.
• GET|POST|DELETE /favorites: watchlist management and tracking.

Non-Custodial Trading Execution

• GET|PUT|DELETE /trading/connections/{platform}: KMS-encrypted per-user exchange connection credentials.
• POST /trading/preview: dry-run order normalization and limit collar verification.
• POST /trading/orders: live order submission with explicit two-step user confirmation.
• GET /trading/portfolio: authenticated balances, open positions, resting orders, and execution fills.
• POST /trading/orders/{audit_order_id}/reconcile: venue order status and fill reconciliation.
• DELETE /trading/orders/{audit_order_id}: cancel resting orders at venue.

Web app runtime state

Anonymous chats stay in browser localStorage. Signed-in users sync conversations through /chat/conversations, while watchlist tracking uses FavoriteMarket entities exposed through /favorites and /favorites/prices. The favorites digest runs from .github/workflows/favorites-digest.yml via scripts/favorites_digest.py.

Forecast sharing is opt-in: clients call POST /forecasts/share to create a public GET /forecast/{share_id} page. Do not expose full private chat history in shared forecast views.

Agent: automated intelligence layer

POST /agent/analyze runs the whole pipeline autonomously: resolve the market (fetch a live Polymarket/Kalshi price when an identifier is given) → gather evidence + forecast → price the edge → run any custom skills → recommend. It returns one structured report.