Corpus Search — Critical Minerals PDF Corpus

Semantic search over a local collection of critical minerals PDFs (USGS, UN Comtrade, World Bank, SEC, WTO, Mindat, MinCan reports). Documents are chunked, embedded with llama-text-embed-v2, and stored in a Pinecone index for fast similarity search.

Usage

Search the corpus:

python3 {baseDir}/scripts/search_corpus.py --query "rare earth separation techniques"

Filter by commodity:

python3 {baseDir}/scripts/search_corpus.py --query "supply chain risks" --commodity lithium

Filter by source organization:

python3 {baseDir}/scripts/search_corpus.py --query "trade flows" --source Comtrade

Reranked results (higher quality):

python3 {baseDir}/scripts/search_corpus.py --query "cobalt extraction" --rerank --top-k 20

JSON output:

python3 {baseDir}/scripts/search_corpus.py --query "graphite processing" --format json

Parameters

Parameter	Description	Default
--query	Semantic search query	Required
--commodity	Filter by commodity keyword (e.g., lithium, cobalt, rare earth)	-
--source	Filter by source organization (e.g., USGS, Comtrade, SEC)	-
--top-k	Number of results to retrieve	10
--rerank	Enable reranking with pinecone-rerank-v0	false
--format	Output format: summary, detailed, json	summary
--index-name	Pinecone index name	scienceclaw-minerals-corpus

Ingestion

Before searching, ingest PDFs into the Pinecone index:

# Dry run — list PDFs that would be ingested:
python3 {baseDir}/scripts/ingest_corpus.py --corpus-dir ~/critical-minerals-data/ --dry-run

# Ingest all PDFs:
python3 {baseDir}/scripts/ingest_corpus.py --corpus-dir ~/critical-minerals-data/

# Force re-ingest (ignore manifest):
python3 {baseDir}/scripts/ingest_corpus.py --corpus-dir ~/critical-minerals-data/ --force-reingest

Ingestion Parameters

Parameter	Description	Default
--corpus-dir	Directory containing PDFs	~/critical-minerals-data/
--index-name	Pinecone index name	scienceclaw-minerals-corpus
--force-reingest	Re-ingest all files, ignoring manifest	false
--dry-run	List files without ingesting	false

Notes

• Requires PINECONE_API_KEY environment variable
• PDFs are chunked at ~600 tokens with 100-token overlap
• Source organization is auto-detected from directory name (e.g., usgs/, sec/)
• Commodity is auto-detected via keyword scanning
• Incremental updates: only new or modified files are re-ingested (SHA-256 manifest)
• Reranking uses pinecone-rerank-v0 for higher quality results at the cost of latency