Skill: idea-discovery-pipeline

You are the orchestrator for a complete idea discovery workflow for: $ARGUMENTS

---

Overview

This skill chains sub-skills into a single automated pipeline:

/lit-review → /generate-idea → /novelty-check → /idea-review → /experiment-design-pipeline

Each phase builds on the previous one's output. The final deliverables are:

• output/LIT_REVIEW_REPORT.md — literature landscape and ranked gaps
• output/IDEA_REPORT.md — ranked, validated, pilot-tested ideas (the top-level report)
• output/NOVELTY_REPORT.md — deep novelty verdicts for top ideas
• output/IDEA_REVIEW_REPORT.md — external critical review of the top idea
• output/refine-logs/FINAL_PROPOSAL.md + output/refine-logs/REFINE_REPORT.md — refined proposal
• output/EXPERIMENT_PLAN.md + output/EXPERIMENT_TRACKER.md — experiment roadmap
• output/EXP_PIPELINE_SUMMARY.md — pipeline summary from experiment-design-pipeline

Constants

• PILOT_MAX_HOURS = 2 — Skip any pilot experiment estimated to take > 2 hours per GPU. Flag as "needs manual pilot" in the report.
• PILOT_TIMEOUT_HOURS = 3 — Hard timeout: kill any running pilot that exceeds 3 hours. Collect partial results if available.
• MAX_PILOT_IDEAS = 3 — Run pilots for at most 3 top ideas in parallel. Additional ideas are validated on paper only.
• MAX_TOTAL_GPU_HOURS = 8 — Total GPU budget across all pilots. If exceeded, skip remaining pilots and note in report.
• AUTO_PROCEED = true — If user doesn't respond at a checkpoint, automatically proceed with the best option after presenting results. Set to false to always wait for explicit user confirmation.
• REVIEWER_MODEL = gpt-5.4 — Model used via Codex MCP. Must be an OpenAI model (e.g., gpt-5.4, o3, gpt-4o). Passed to sub-skills. If not configured properly, use Claude Code subagent.
• ARXIV_DOWNLOAD = false — When true, /lit-review downloads the top relevant arXiv PDFs during Phase 1. When false (default), only fetches metadata. Passed through to /lit-review.
• COMPACT = false — When true, generate compact summary files for short-context models and session recovery. Writes output/IDEA_CANDIDATES.md (top 3-5 ideas only) at the end of this workflow. Downstream skills read this instead of the full output/IDEA_REPORT.md.
• REF_PAPER = false — Reference paper to base ideas on. Accepts: local PDF path, arXiv URL, or any paper URL. When set, the paper is summarized first (output/REF_PAPER_SUMMARY.md), then idea generation uses it as context. Combine with base repo for "improve this paper with this codebase" workflows.

💡 These are defaults. Override by telling the skill, e.g., /idea-discovery-pipeline "topic" — ref paper: https://arxiv.org/abs/2406.04329 or /idea-discovery-pipeline "topic" — compact: true.

Pipeline

Phase 0: Load Context and Research Plan

At the very start, read the most relevant existing files if they exist (use Read; skip silently if missing). This mirrors the Phase 0 pattern used by each subskill and ensures the pipeline never re-does stable work:

• CLAUDE.md — project dashboard, control flags, canonical output paths
• handoff.json — if present, check pipeline.stage and recovery.resume_skill
• RESEARCH_PLAN.md in the project root (or path passed as $ARGUMENTS) — detailed research brief
• output/LIT_REVIEW_REPORT.md — prior literature review, if any
• output/IDEA_REPORT.md — prior idea candidates, if any
• output/NOVELTY_REPORT.md — prior novelty verdicts, if any
• output/IDEA_REVIEW_REPORT.md — prior external review, if any
• output/REF_PAPER_SUMMARY.md — prior reference paper summary, if any
• output/refine-logs/FINAL_PROPOSAL.md — prior refined proposal, if any

If RESEARCH_PLAN.md is found, extract:

• Problem statement and context
• Constraints (compute, data, timeline, venue)
• What the user already tried / what didn't work
• Domain knowledge and non-goals
• Existing results (if any)

Use this as the primary context for all subsequent phases — it replaces the one-line prompt. If both RESEARCH_PLAN.md and a one-line $ARGUMENTS exist, merge them (the plan takes priority for details; the argument sets the direction). If no plan exists, proceed with $ARGUMENTS as the research direction.

Decide whether to resume from a prior stage or re-run:

• If a prior artifact is stale or materially different from the current request, re-run that phase.
• If it is strong and aligned, reuse it and jump to the next phase.
• When in doubt, prefer re-running over planning on top of the wrong idea.

💡 Create a plan from the template: cp templates/RESEARCH_PLAN_TEMPLATE.md RESEARCH_PLAN.md

Phase 0.5: Reference Paper Summary (when REF_PAPER is set)

Skip entirely if REF_PAPER is false.

Summarize the reference paper before searching the literature:

1. If arXiv URL (e.g., https://arxiv.org/abs/2406.04329):

   • Invoke /arxiv "ARXIV_ID" — download to fetch the PDF
   • Read the first 5 pages (title, abstract, intro, method overview)

2. If local PDF path (e.g., papers/reference.pdf):

   • Read the PDF directly (first 5 pages)

3. If other URL:

   • Fetch and extract content via WebFetch

4. Generate output/REF_PAPER_SUMMARY.md (canonical path — use this everywhere):

# Reference Paper Summary

**Title**: [paper title]
**Authors**: [authors]
**Venue**: [venue, year]

## What They Did
[2-3 sentences: core method and contribution]

## Key Results
[Main quantitative findings]

## Limitations & Open Questions
[What the paper didn't solve, acknowledged weaknesses, future work suggestions]

## Potential Improvement Directions
[Based on the limitations, what could be improved or extended?]

## Codebase
[If `base repo` is also set: link to the repo and note which parts correspond to the paper]

🚦 Checkpoint: Present the summary to the user:

📄 Reference paper summarized:
- Title: [title]
- Key limitation: [main gap]
- Improvement directions: [2-3 bullets]

Proceeding to literature survey with this as context.

Phase 1 and Phase 2 will use output/REF_PAPER_SUMMARY.md as additional context — /lit-review searches for related and competing work, /generate-idea generates ideas that build on or improve the reference paper.

Phase 1: Literature Review

Invoke skill /lit-review to map the research landscape:

/lit-review "$ARGUMENTS"

What this does (per the lit-review subskill):

• Searches local sources first (papers/, Zotero, Obsidian) when configured, then arXiv via tools/arxiv_fetch.py, Semantic Scholar via tools/semantic_scholar_fetch.py, and WebSearch (arXiv, Google Scholar, Semantic Scholar)
• Builds a synthesis matrix and gap analysis
• Writes paper metadata to output/paper-cache/
• Writes the consolidated report to output/LIT_REVIEW_REPORT.md (Findings, Synthesis, Gap Analysis sections)

Pass ARXIV_DOWNLOAD through if the user overrode it.

🚦 Checkpoint: Present the landscape summary from output/LIT_REVIEW_REPORT.md to the user:

📚 Literature survey complete. Here's what I found:
- [key findings, gaps, open problems]

Does this match your understanding? Should I adjust the scope before generating ideas?
(If no response, I'll proceed with the top-ranked direction.)

• User approves (or no response + AUTO_PROCEED=true) → proceed to Phase 2 with best direction.
• User requests changes (e.g., "focus more on X", "ignore Y", "too broad") → refine the search with updated queries, re-run /lit-review with adjusted scope, and present again. Repeat until the user is satisfied.

Phase 2: Idea Generation + Filtering + Pilots

Invoke /generate-idea with the landscape context (and output/REF_PAPER_SUMMARY.md if available):

/generate-idea "$ARGUMENTS"

What this does (per the generate-idea subskill):

• Reads output/LIT_REVIEW_REPORT.md (if present) and skips its own Phase 1 silently
• Reads output/REF_PAPER_SUMMARY.md if present — ideas should build on, improve, or extend the reference paper
• Brainstorms 8-12 concrete ideas
• Filters by feasibility, compute cost, quick novelty search
• Deep-validates top ideas (quick novelty pass + devil's advocate)
• Runs pilot experiments on available GPUs (top MAX_PILOT_IDEAS, respecting MAX_TOTAL_GPU_HOURS and PILOT_TIMEOUT_HOURS)
• Ranks by empirical signal
• Writes output/IDEA_REPORT.md

🚦 Checkpoint: Present the ranked ideas from output/IDEA_REPORT.md:

💡 Generated X ideas, filtered to Y, piloted Z. Top results:

1. [Idea 1] — Pilot: POSITIVE (+X%)
2. [Idea 2] — Pilot: WEAK POSITIVE (+Y%)
3. [Idea 3] — Pilot: NEGATIVE, eliminated

Which ideas should I validate further? Or should I regenerate with different constraints?
(If no response, I'll proceed with the top-ranked ideas.)

• User picks ideas (or no response + AUTO_PROCEED=true) → proceed to Phase 3 with top-ranked ideas.
• User unhappy with all ideas → collect feedback ("what's missing?", "what direction do you prefer?"), update the prompt with user's constraints, re-run Phase 2. Repeat until the user selects at least 1 idea.
• User wants to adjust scope → go back to Phase 1 with refined direction.

Phase 3: Deep Novelty Verification

Invoke /novelty-check to run a thorough novelty check on the top candidates:

/novelty-check "$ARGUMENTS"

What this does (per the novelty-check subskill):

• Reads output/IDEA_REPORT.md (preferred) and runs the remaining phases per idea for the top-ranked candidates; treats $ARGUMENTS as a topic filter
• Reuses abstracts from output/paper-cache/ and papers/ where available
• Runs multi-source literature search (arXiv, Semantic Scholar, WebSearch) and cross-verifies with REVIEWER_MODEL
• Checks for concurrent work in the last 3-6 months and identifies closest existing work plus differentiation points
• Writes output/NOVELTY_REPORT.md
• Updates output/IDEA_REPORT.md in place with each idea's novelty verdict and score

Eliminate any idea that turns out to be already published.

Phase 4: External Critical Review

For the surviving top idea(s), get brutal feedback. The underlying skill is idea-review/

/idea-review "[top idea with hypothesis + pilot results]"

What this does (per the idea-review subskill):

• REVIEWER_MODEL (xhigh reasoning via Codex MCP) acts as a senior reviewer
• Runs a multi-round critical dialogue, scores the idea, identifies weaknesses, suggests minimum viable improvements
• Provides concrete feedback on experimental design
• Writes output/IDEA_REVIEW_REPORT.md

Update output/IDEA_REPORT.md with the reviewer feedback and revised plan.

Phase 4.5: Method Refinement + Experiment Planning

After review, refine the top idea into a concrete proposal and plan experiments by invoking the wrapper pipeline:

/experiment-design-pipeline "[top idea description + pilot results + reviewer feedback]"

What this does (per the experiment-design-pipeline skill, which chains refine-research then experiment-design):

• Freezes a Problem Anchor and preserves it across rounds
• Iteratively refines the method via REVIEWER_MODEL review (up to 5 rounds, target score ≥ 9), respecting the subskill's constants (MAX_PRIMARY_CLAIMS = 2, MAX_NEW_TRAINABLE_COMPONENTS = 2, etc.)
• Generates a claim-driven experiment roadmap with ablations, budgets, and run order
• Outputs:
  • Refinement: output/refine-logs/FINAL_PROPOSAL.md, output/refine-logs/REFINE_REPORT.md, output/refine-logs/SCORE_HISTORY.md
  • Experiments: output/EXPERIMENT_PLAN.md, output/EXPERIMENT_TRACKER.md
  • Pipeline summary: output/EXP_PIPELINE_SUMMARY.md

Do not relocate these artifacts — downstream skills (deploy-experiment, result-to-claim, etc.) expect these canonical paths.

🚦 Checkpoint: Present the refined proposal summary:

🔬 Method refined and experiment plan ready:
- Problem anchor: [anchored problem]
- Method thesis: [one sentence]
- Dominant contribution: [what's new]
- Must-run experiments: [N blocks]
- First 3 runs to launch: [list]

Proceed to implementation? Or adjust the proposal?

• User approves (or AUTO_PROCEED=true) → proceed to Final Report.
• User requests changes → pass feedback to /refine-research for another round, then re-run /experiment-design if needed.
• Lite mode: If reviewer score < 6 or the pilot was weak, run /refine-research only (skip /experiment-design) and note remaining risks in the report.

Phase 5: Final Report

Finalize output/IDEA_REPORT.md with all accumulated information:

# Idea Discovery Report

**Direction**: $ARGUMENTS
**Date**: [today]
**Pipeline**: lit-review → generate-idea → novelty-check → idea-review → experiment-design-pipeline

## Executive Summary
[2-3 sentences: best idea, key evidence, recommended next step]

## Literature Landscape
[from Phase 1 — see output/LIT_REVIEW_REPORT.md]

## Ranked Ideas
[from Phase 2, updated with Phase 3-4 results]

### 🏆 Idea 1: [title] — RECOMMENDED
- Pilot: POSITIVE (+X%)
- Novelty: CONFIRMED (closest: [paper], differentiation: [what's different])
- Reviewer score: X/10
- Next step: implement full experiment → /deploy-experiment

### Idea 2: [title] — BACKUP
...

## Eliminated Ideas
[ideas killed at each phase, with reasons]

## Refined Proposal
- Proposal: `output/refine-logs/FINAL_PROPOSAL.md`
- Refinement report: `output/refine-logs/REFINE_REPORT.md`
- Experiment plan: `output/EXPERIMENT_PLAN.md`
- Experiment tracker: `output/EXPERIMENT_TRACKER.md`
- Pipeline summary: `output/EXP_PIPELINE_SUMMARY.md`

## Supporting Reports
- Literature: `output/LIT_REVIEW_REPORT.md`
- Novelty: `output/NOVELTY_REPORT.md`
- External review: `output/IDEA_REVIEW_REPORT.md`

## Next Steps
- [ ] /deploy-experiment to deploy experiments from the plan
- [ ] /auto-review-loop to iterate until submission-ready
- [ ] Or invoke /full-pipeline for the complete end-to-end flow

Phase 5.5: Write Compact Files (when COMPACT = true)

Skip entirely if COMPACT is false.

Write output/IDEA_CANDIDATES.md — a lean summary of the top 3-5 surviving ideas:

# Idea Candidates

| # | Idea | Pilot Signal | Novelty | Reviewer Score | Status |
|---|------|-------------|---------|---------------|--------|
| 1 | [title] | +X% | Confirmed | X/10 | RECOMMENDED |
| 2 | [title] | +Y% | Confirmed | X/10 | BACKUP |
| 3 | [title] | Negative | — | — | ELIMINATED |

## Active Idea: #1 — [title]
- Hypothesis: [one sentence]
- Key evidence: [pilot result]
- Next step: /experiment-design-pipeline or /deploy-experiment

This file is intentionally small (~30 lines) so downstream skills and session recovery can read it without loading the full output/IDEA_REPORT.md (~200+ lines).

Key Rules

• Large file handling: If the Write tool fails due to file size, immediately retry using Bash (cat << 'EOF' > file) to write in chunks. Do NOT ask the user for permission — just do it silently.

• Don't skip phases. Each phase filters and validates — skipping leads to wasted effort later.

• Checkpoint between phases. Briefly summarize what was found before moving on.

• Mandatory local GPU check before pilots. Phase 2 delegates pilot launch to /generate-idea, which MUST run its Phase 5.0 local GPU presence check (nvidia-smi → CUDA, else MPS, else LOCAL_GPU=none). When a local GPU is detected, every pilot in this pipeline MUST run on it — never fall back to CPU and never silently re-route to remote/Modal while a local GPU is available.

• Kill ideas early. It's better to kill 10 bad ideas in Phase 3 than to implement one and fail.

• Empirical signal > theoretical appeal. An idea with a positive pilot outranks a "sounds great" idea without evidence.

• Document everything. Dead ends are just as valuable as successes for future reference.

• Be honest with the reviewer. Include negative results and failed pilots in the review prompt.

• Do not relocate subskill outputs. Literature, idea, novelty, and review reports live directly under output/. Refinement artifacts live under output/refine-logs/. Experiment artifacts live directly under output/.

• Reuse, don't duplicate. If a subskill already reads a canonical file (e.g., generate-idea reads output/LIT_REVIEW_REPORT.md, novelty-check reads output/IDEA_REPORT.md), do not re-inject that content as a large prompt — let the subskill read it.

Composing with Workflow 2

After this pipeline produces a validated top idea:

/idea-discovery-pipeline "direction"   ← you are here (Workflow 1, includes method refinement + experiment planning)
/deploy-experiment                      ← deploy experiments from the plan
/auto-review-loop "top idea"            ← Workflow 2: iterate until submission-ready

Or use /full-pipeline for the full end-to-end flow.