Process & Upload a Benchmarked Method to TabArena

This skill drives the maintainer workflow that turns a benchmark run's already-present raw results.pkl files into cached, hosted (r2), and registered TabArena artifacts. There is no download and no auto-generation — the raw data is assumed to be on disk already (e.g. you unzipped a submission, or it's a fresh run's output/<run>/data dir).

The authoritative prose for this flow is AGENTS.md → "Processing & uploading method artifacts (maintainers)" and the two scripts' module docstrings (scripts/run_process_method.py, scripts/run_upload_results.py). This skill operationalizes it: Claude executes the flow itself by default, and makes the small deterministic code edits along the way.

What this skill delivers

1. A plan, stated first: which methods, which run dir, which suite, and that the flow ends in a real r2 upload — tell the maintainer before executing (no need to wait for approval unless they asked to review something first).
2. Claude-run execution, in order per method:
   • inspect — read the raw data, print inferred fields + a metadata diff.
   • process — build + cache metadata.yaml + processed/ + results/ locally.
   • upload (dry-run, then --no-dry-run) — push the cached artifacts to r2.
3. Code edits:
   • The model's info.py MethodMetadata — fill in the manual upload fields, fix any raw-data mismatches the inspect diff surfaces.
   • The arena collection registration in methods.py so the method appears in the benchmark.
   • The run's entry in packages/tabflow_slurm/BENCHMARK_LOG.md, if the launch stage never wrote one.
4. A fallback command sheet only for steps the environment can't run.

Execution requirements (check before starting; hand off the affected step if missing):

• a venv with tabarena[benchmark] installed (look under ~/.venvs/tabarena_*; run scripts with its python from the repo root),
• R2 credentials in the environment for the real upload (printenv | grep -o '^R2_[A-Z_]*' should show R2_ACCOUNT_ID / R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY; never pass them as flags).

Processing and uploads are long-running: run them in the background (per-method), watch with a monitor (per-method DONE/FAILED lines + a stall watchdog), and for multi-method batches pipeline the uploads — start each method's dry-run + real upload as soon as its processing finishes.

Step 0: Gather inputs

Parse $ARGUMENTS. Collect (ask only for what's missing or ambiguous):

Input	Example	Notes
run_data_dir	.../output/benchmark_chimeraboost_16062026/data	Dir of raw results.pkl files (searched recursively). Point at the run's data/.
model	chimeraboost	The packages/tabarena/src/tabarena/models/<model>/ folder — or systems/<system>/ when the method is a whole pipeline (AutoML framework, agent, hosted API). Usually inferable from the run/data dir name; confirm.
suite	tabarena-2026-06-30	The dated run/suite id. Must differ from method (see Step 3). Default to tabarena-<run-date>; ask if unclear.
arena	tabarena	Which arena collection to register in (default tabarena; e.g. beyondarena has its own).
verified	False until signed off	Whether the results are verified. Default keep False; flip to True only when the maintainer confirms (Step 3).

Step 1: Locate the method's MethodMetadata

Read packages/tabarena/src/tabarena/models/<model>/info.py (or systems/<system>/info.py) and note the exact variable name of its MethodMetadata (e.g. chimeraboost_method_metadata, tabfm_plus_method_metadata). Both scripts take it as a dotted reference:

tabarena.models.<model>.info:<varname>
tabarena.systems.<system>.info:<varname>

• info.py exists (the normal case — the model was added via the add-model skill): it already carries the raw-inferable fields (ag_key, config_default, can_hpo, is_bag, compute, method_type). validation_protocol is inferable too but was not authored before the run: a new run records it, and --process fails until info.py declares the inferred key. Claude fills it together with the manual upload fields in Step 3.
• No info.py (a raw external submission): the method isn't integrated yet. Run inspect (Step 2) to get the copy-paste MethodMetadata.<type>(...) snippet, then author models/<model>/info.py from it (use the add-model skill if the model also needs a wrapper, or add-system if it is a whole pipeline). Processing requires an explicit committed MethodMetadata — it refuses to guess.
• The method is a system: the inspector cannot tell a system from any other baseline, since the runner records both identically. Use MethodMetadata.system(...) rather than .baseline(...), and set tags — see the add-system skill. Getting this wrong types the method as a model on the leaderboard and puts it in the models-only entrant pool, where it does not belong.

Step 2: Inspect the raw data (Claude runs; confirms inference)

<venv>/bin/python scripts/run_process_method.py <run_data_dir>

This prints the fields inferred from the raw data (method_type, ag_key, compute, config_default, can_hpo, is_bag, validation_protocol with a fold histogram, task/problem-type/metric coverage), a suggested MethodMetadata snippet, and — when an info.py metadata is passed — an inferred-vs-provided diff. Use it to confirm info.py matches the raw data before processing. Two gotchas it catches:

• config_default is compared post-rename: configs are renamed to the method's prefix during processing, so a config_default authored with the raw prefix won't match. The snippet/diff shows the post-rename value — use that. The most common real mismatch is the infix, not the prefix: the TabArena-v0.1 bundle names the first config <Method>_c1_default_BAG_L1 (the _default is the preprocessing pipeline name, appended for HPO and default-only models alike), while info.py files authored before a run, and the BeyondArena bundle, use <Method>_c1_BAG_L1. An info.py written by add-model before the run therefore usually needs this one fix. A single-config method (can_hpo=False) can drop the field instead: an undeclared config_default is not checked (the diff marks it not declared) and --process records the lone config in the cached metadata.yaml.
• Only error rows gate processing: the method row differs whenever the raw ag_name carries the TA- prefix (warn-only) and model_key is never checked (shown for context). A NO on either is expected and needs no edit; a NO on config_default, ag_key, compute, can_hpo, is_bag or method_type does.
• method != suite: process fails if they're equal (suite defaults to method when unset).
• validation_protocol must be declared: a run made after the protocol record existed infers a key (8x1 for TabArena, system for a system, the long BeyondArena key); --process fails while info.py leaves it None or declares another value. Legacy raw artifacts infer None and pass. Add --expect-validation-protocol tabarena (or beyondarena) to also assert that the run used the arena's official protocol; a holdout:/outer/custom key there means the run cannot be uploaded as an official result. KNN's 8x1 requested; 1 child via use_child_oof histogram line is expected.

Multi-method run dirs: if the run's data/ holds several methods' config dirs side by side, run_process_method.py can't split them — write a small gitignored driver in tmp_scripts/ that discovers the results.pkl paths once, splits them by top-level config-dir prefix, and calls _infer_from_raw / verify_method_metadata / process_raw(file_paths=...) per method (see the multi-method-run-upload pattern; a --c1-only pass over just the _c1_ dirs makes config_default inferable even for HPO methods with hundreds of configs).

Step 3: Edit info.py (Claude does now)

Fill in the manual fields the upload requires. These are not inferable from raw data, so the maintainer normally hand-edits them — Claude does it now. Read the file first, then Edit:

Field	Set to	Why
suite	the dated run id, e.g. "tabarena-2026-06-30"	Required, must differ from method. Equal pair fails process's method != suite check (suite defaults to method when unset).
cache_type	"r2"	Required for upload — the upload script is r2-only; "local"/None has no remote store.
cache_kwargs	{"bucket": "tabarena", "prefix": "cache"}	The r2 location (bucket + prefix). Required when cache_type="r2".
date	"YYYY-MM-DD"	The run date. Validated as a real calendar date.
verified	False (until signed off), then True	Manual trust flag. Keep False until the results are verified; flip to True once they are (typically the final step).
method_class / tags	systems only	MethodMetadata.system(...) sets method_class; tags (with-llm, closed-source-api) decide which entrant pools the system competes in. Neither is inferable from raw data.
validation_protocol	the key the inspect step infers, e.g. "8x1"	The inner validation protocol the results record. --process requires it for a new run; run_upload_results.py refuses an upload whose processed artifact records a key info.py does not declare. MethodMetadata.system(...) defaults it to "system".

Leave the other raw-inferable fields (ag_key, config_default, can_hpo, is_bag, compute, method_type) as they are — they came from add-model. Only change one if Step 2's diff shows a genuine mismatch (and then to the inferred value; for a single-config method, deleting a mismatched config_default is equally valid).

Example — making chimeraboost upload-ready (the upload fields added; compare nori's info.py, which is already in this shape):

chimeraboost_method_metadata = MethodMetadata.config(
    method="ChimeraBoost",
    suite="tabarena-2026-06-30",                              # added: distinct dated suite
    ag_key="CHIMERA",
    config_default="ChimeraBoost_c1_default_BAG_L1",             # HPO method: post-rename, `_default` = the run's pipeline
    compute="cpu",
    is_bag=False,
    date="2026-06-15",
    reference_url="https://github.com/bbstats/chimeraboost",
    display_name="ChimeraBoost",
    verified=False,                                           # flip to True once verified
    cache_type="r2",                                          # added
    cache_kwargs={"bucket": "tabarena", "prefix": "cache"},   # added
)

Step 4: Execute process → upload (Claude runs, in the background)

Run these with the resolved dotted reference, per method — background + monitor for the slow ones. The same block doubles as the fallback command sheet if a step must be handed to the maintainer. Using chimeraboost as the example:

# 1. Inspect (confirm inferred fields match info.py) — already done in Step 2

# 2. (info.py edited: suite + cache_type/cache_kwargs + date, plus any inspect-diff fixes)

# 3. Process: build + cache metadata.yaml + processed/ + results/ locally
<venv>/bin/python scripts/run_process_method.py <run_data_dir> \
    --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata --process \
    --expect-validation-protocol tabarena  # the run must record TabArena's official protocol

# 4. Upload dry-run: verifies every part exists locally and prints what/where (no creds needed)
<venv>/bin/python scripts/run_upload_results.py \
    --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata

# 5. Real upload (R2 creds via env, NEVER flags):
<venv>/bin/python scripts/run_upload_results.py \
    --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata --no-dry-run

After the real upload, verify on the bucket (ListObjects via boto3 against the R2 endpoint): each method should have the full object set under cache/artifacts/<suite>/methods/<Method>/ (metadata.yaml, processed.zip, processed/configs_hyperparameters.json, raw.zip, results/*.parquet).

Notes:

• process requires the explicit --method-metadata and verifies it against the raw data first; a real mismatch errors (override with --ignore-metadata-mismatch, a method-name mismatch only warns). Raw + HPO trajectories are cached by default (--no-cache-raw / --no-cache-hpo-trajectories).
• The dry-run is the default for the upload script; it needs no credentials and prints the exact --no-dry-run command. Raw uploads by default (--no-upload-raw to skip).
• For the real upload, set R2_ACCOUNT_ID / R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY in the environment (export them or use a .env) — never as CLI flags (they'd leak into shell history / the process table). The dry-run prints how to obtain them if unset (MethodMetadata.r2_credentials_help()).

Step 5: Register in the arena collection (Claude does now)

Add the method to the collection so it appears in the benchmark. For the default tabarena arena, edit packages/tabarena/src/tabarena/contexts/tabarena/methods.py (read it first):

1. Import the model's info.py metadata in the alphabetical import block, matching the file's existing style (most recent additions use the plain name, e.g. from tabarena.models.nori.info import nori_method_metadata; alias only on a name collision).
2. Add the entry to tabarena_method_metadata_collection.method_metadata_lst, under the matching group comment (# Default tabular models (CPU) vs # Neural / GPU / foundation models), placed by compute/type.

It flows into tabarena_method_metadata_complete_collection automatically (no separate edit). Other arenas (e.g. beyondarena) register in their own collection's methods.py.

When the upload is a rerun of an already-registered method (a new library version, a remeasured run), the entry is a swap, not an addition: point the collection at the new metadata and append the one it replaced to methods_superseded, so the predecessor's hosted artifacts stay reachable through the complete collection. Say in the report that the default leaderboard now carries only the new run.

Caveat to surface in the report: the collection entry only resolves to downloadable artifacts once the real upload (Step 4.5) has actually run. The code edit is safe to land now, but the method won't load for others until uploaded.

Step 6: Record the run in the benchmark log (Claude does now)

packages/tabflow_slurm/BENCHMARK_LOG.md is the committed record of every cluster run (the tmp_scripts/run_<model>.py that launched it is gitignored, so this log is the only surviving copy of the setup). The benchmark-model skill merely offers to write the entry at launch time, before any results exist, so in practice runs reach upload unlogged. Check here, where the run is finished:

1. grep -n "^## " packages/tabflow_slurm/BENCHMARK_LOG.md | head — if the run's benchmark_name is already there, nothing to do.
2. Otherwise add an entry at the top of the log (append-only, newest-first), following the template in the file's "Conventions" section: ## YYYY-MM-DD — <benchmark_name>, then Model(s) / Git SHA / Purpose / Notes, then the verbatim plan.

Two fields need care:

• Git SHA is HEAD at setup time, not HEAD now. git reflog --date=iso shows when HEAD sat on which commit; bracket the launch with the run's own timestamps (the earliest results.pkl mtime under the run's data/, or the earliest file in <workspace>/slurm_out/<benchmark_name>/) and pick the commit that was HEAD then.
• The python block is the launch script's setup() body copied verbatim, with its module-level constants (BENCHMARK_NAME, WORKSPACE, PYTHON_PATH, MODEL, NUM_CONFIGS) inlined as literals and _path_setup() expanded, so the snippet stands alone against its SHA. Never refactor a neighbouring entry to match the current API.

The launch script's module docstring usually holds the why (issue link, what changed versus the previous run, partition choice, model-specific caveats) — that is the Purpose/Notes material. Add the processed suite id and the metadata variable name too, so the log ties the run to its artifacts.

Step 7: Lint touched files

ruff check <touched-files>
ruff format --check <touched-files>

Touched files are the model's info.py and contexts/<arena>/methods.py. Fix anything reported (the from __future__ import annotations import is already present in both — don't drop it). BENCHMARK_LOG.md is markdown, so ruff does not apply to it.

Step 8: Report

Tell the maintainer:

• What was executed and verified: inspect diff results, processing outcome per method, and the r2 destinations confirmed after the real upload — plus any data-quality warnings from processing (e.g. Not close TEST prediction-fidelity lines, with affected datasets and severity).
• Edits Claude made: the info.py upload fields (suite / cache_type / cache_kwargs / date / verified, plus any inspect-diff fixes), the methods.py import + collection entry (and the methods_superseded append, for a rerun), and the BENCHMARK_LOG.md entry (with the SHA it recorded and how it was determined, since that one is inferred).
• Steps handed off (only if the env couldn't run one): the exact command(s) from Step 4.
• Open decisions / TODOs: whether to flip verified to True (only after sign-off), committing the working-tree edits, and — if the method should appear on the website — that update-leaderboard is the next lifecycle step.

When asked to open the PR, use .github/pull_request_template.md and delete its "Model or system submission" section (that one is for external submissions). The summary names the method and what was uploaded; the collapsed Details block carries the method key, suite and date, the r2 destinations, the verified state, and whether update-leaderboard is the next step. Do not paste this Report into the PR body.