Add a System to TabArena

Model or system?

Ask this first, because it decides everything else.

• Model — one method that TabArena tunes, using the shared search-space protocol and compute constraints. It plugs into AutoGluon, has an ag_key, and gets default / tuned / tuned+ensembled variants. Use the add-model skill.
• System — a pipeline that manages its own budget, model selection, tuning and ensembling. TabArena hands it the data and the constraints and records what comes back. AutoML frameworks, agents, hosted APIs, and models run through a self-managing interface all land here.

A useful test: if you would have to invent a search space for it, it is a model. If inventing one makes no sense because the thing does its own searching, it is a system.

Layout

Every system lives in one folder at packages/tabarena/src/tabarena/systems/<system_key>/, mirroring models/:

systems/<system_key>/
  __init__.py   re-exports the three below
  system.py     the ExternalSystemModel subclass
  hpo.py        the SystemConfigGenerator (which configurations to benchmark)
  info.py       the SystemInfo + its MethodMetadata

systems/_registry.py::discover_systems() walks these info modules into SYSTEM_REGISTRY, keyed by method_metadata.method. Read systems/autogluon/ (a framework) and systems/tabfm_plus/ (a model through a heavier interface) before writing a new one.

Systems stay out of the AutoGluon model registry on purpose: no ag_key, no search space, and they run through the experiment bundle's system_experiments=True mode.

Step 1: system.py

Subclass ExternalSystemModel (tabarena/benchmark/exec_models/external.py) and implement _fit_system, _predict and _predict_proba. Read that class's docstring for the full argument contract; the parts that matter most:

• Everything the fit needs is passed in, never read off self: the raw frames, target_name, problem_type, eval_metric, validation_metadata, the compute budget (num_cpus / num_gpus / memory_limit / time_limit) and the per-split random_state.
• X is yours to edit in place. There is no validation split; carve your own from X/y if the system wants one.
• Add __init__ arguments for the system's settings and forward **kwargs to super().__init__. Those arguments are what the config generator varies.
• The compute and time budgets are not init knobs. They come per split from the runner, so every system is held to the same constraints.
• Add cleanup to free files and memory. Only delete a directory you created; an explicitly passed path belongs to the caller.

Keep the library import inside _fit_system (or the method that needs it), never at module top level, so an install without the extra still imports.

A system's environment work is its own. TabArena adds no system-specific warm-up, persist or checkpoint prefetch code: a system that imports its stack, downloads checkpoints or loads from disk inside its fit or predict is measured the way it ships (the AutoGluonSystemModel docstring spells this out for AutoGluon). Two generic hooks exist, documented in the ExternalSystemModel docstring:

• Warm-up: a system may declare warmup_modules = ("yourlib",) and/or warmup_torch_device = True; that is import and CUDA-context work only, and the synthetic dummy fit is off for systems. Do not write per-system warm-up logic.
• uses_ray: keep the base default (True) unless no code path of the system can start Ray; the SLURM worker skips the Ray runtime for fits that answer False.

SystemInfo.prefetch_weights is a hook for the system's own tooling; the benchmark setup does not call it, and offline_weights="auto" stays off while a system is selected because its checkpoints are not prefetched.

Step 2: hpo.py

gen_<system_key> = SystemConfigGenerator(
    model_cls=<SystemName>SystemModel,
    name="<SystemName>",          # a system has no ag_name/ag_key, so this is required
    manual_configs=[{}, {"preset": "high_quality"}],
)

Each config becomes one benchmarked variant. Prefer a small set of meaningful presets over a search space: the point of a system is that it searches for you.

Step 3: info.py

<system_key>_method_metadata = MethodMetadata.system(
    method="<SystemName>",
    name="<SystemName>",
    suite="tabarena-<YYYY-MM-DD>",   # required, must differ from `method`
    compute="cpu" | "gpu",
    date="<YYYY-MM-DD>",
    date_introduced="<YYYY-MM>",
    reference_url="...",
    tags=(),                         # see below
    verified=False,                  # until signed off
)

<system_key>_info = SystemInfo(
    system_cls=<SystemName>SystemModel,
    config_generator=gen_<system_key>,
    method_metadata=<system_key>_method_metadata,
    pip_extra=("<package>==<version>",),
    prefetch_weights=None,           # optional hook for the system's own tooling; the benchmark setup does not call it
)

MethodMetadata.system(...) fixes method_type="baseline" (a system's raw results are recorded that way by the runner) and sets method_class="system". SystemInfo asserts the latter, so a misdeclared system fails at import instead of misclassifying on the leaderboard.

Choosing tags

tags is what lets a reader rule a system out, and it decides which entrant pools it competes in (evaluation/entrants.py). Only two values exist; ask the user when either is unclear rather than guessing.

tag	when	effect
with-llm	an LLM is involved anywhere, agents included	competes only where the llm category is selected
closed-source-api	runs behind a remote API we cannot inspect	competes only where the api category is selected

A system carrying both tags needs both categories selected, so it never appears on the strength of a property the reader excluded.

No tags means open-source, local and LLM-free, which is the common case (AutoGluon, LightAutoML, FLAML, TabFM+) and puts the system in the open category. The two tags are independent: an open-source agent is ("with-llm",), and a hosted non-LLM predictor is ("closed-source-api",). Each combination of categories is published as its own pool, so a new tag doubles the artifact count.

If the system needs a property neither tag covers, do not invent one inline. Add it to MethodTag in models/_method_metadata.py, give it presentation in website_format.TAG_SPECS, and decide which pools admit it in evaluation/entrants.py. All three or none, otherwise it will not render.

Step 4: the pip extra

Add the system's dependency to the [project.optional-dependencies] block in packages/tabarena/pyproject.toml, matching SystemInfo.pip_extra.

Step 5: register and verify

• Add the metadata to tabarena_method_metadata_collection in contexts/tabarena/methods.py once results exist (see the upload-method skill for processing and hosting them).
• pytest tests/tabarena/systems/ -q — the registry test checks the new system is discovered and declares method_class="system".
• ruff check and ruff format --check on every touched file.

There is no per-system fit test. Verify the wrapper with the quickstart in examples/benchmarking/run_quickstart_tabarena_system.py, which runs two configs of the demo system on the small datasets' first split (examples/beyondarena/run_quickstart_beyondarena_system.py is the BeyondArena counterpart and runs the shipped AutoGluon wrapper).

Step 6: Report and open the PR

Summarize the new files, the edited files, the chosen tags and why, and the open TODOs (results, verified, the methods.py registration once artifacts exist).

When asked to open the PR, use .github/pull_request_template.md: a two-to-four sentence summary, everything longer inside the collapsed <details><summary>Details</summary> block, the commands run under Tests. Fill in the "Model or system submission" section for a system (delete the model lines) and keep the closing contribution line. Do not paste the Report into the PR body; the Report is for the chat, the PR body is for the reviewers. State the TabArena-Lite (or BeyondArena core) results with the hardware and the entry-point script if they exist; if they do not, say so, since a maintainer will ask (TabArena verifies submitted results by re-running them, it does not benchmark on request). Questions go through the issue forms in .github/ISSUE_TEMPLATE/.

What happens on the leaderboard

The system appears under the 📊 System family, typed from method_class rather than from its name, with a chip per tag. It shows up in whichever entrant pools admit it, and its presence changes every other entrant's Elo and Improvability in those pools, since both are measured against the field.