Adapt TabArena for a New Domain Benchmark

This skill is for a different repository building its own benchmark on top of TabArena, not for contributing to this repo. If the user wants to add a model, system, or feature to TabArena itself, use add-model / add-system instead.

The concrete reference implementation this skill distills is RamanBench (github.com/ml-lab-htw/RamanBench), which migrated its model/metrics/splitting layer onto tabarena/bencheval for its v1 release, replacing hand-rolled patterns that were "inspired by" TabArena with the real thing. Read that repo's src/raman_bench/ for a worked example alongside this skill.

Every module path below was checked against this checkout. Both packages are pre-1.0 and the API still moves (several paths cited by earlier drafts of this skill no longer exist), so grep a path before writing it into the downstream repo, and read the docstring of anything you depend on.

What you reuse and what you write

Layer	Reuse from tabarena / bencheval	You write
Datasets and outer splits	UserTask, TaskMetadataCollection, TaskMetadataSource, SubsetPredicate	dataset loaders, the split construction, a committed metadata CSV
Models and search spaces	the model registry, ConfigGenerator, every model's hpo.py	domain models registered via register_model_info
Fitting protocol	the experiment bundles, AGModelBagExperiment, the AutoGluon wrappers, the ValidationProtocol the context asserts	a bundle subclass carrying your defaults, a context declaring your protocol
Preprocessing	TabArenaModelAgnosticPreprocessing, build_feature_generator, model-specific hyperparameter injection	a domain feature generator or a model mixin
Running	AbstractArenaContext.build_and_run_jobs, ExperimentBatchRunner, JobBatch, tabflow_slurm	a cluster profile
Leaderboard math	bencheval.evaluator.BenchmarkEvaluator, reached through compare	nothing, or one LeaderboardMetric
Publishing	MethodMetadata artifact tiers, format_leaderboard, the interactive HTML explorers	a methods.py roster and a place to host results

The canonical downstream loop, from examples/benchmarking/run_quickstart_tabarena_custom_datasets.py:

from tabarena.benchmark.experiment import TabArenaExperimentBundle
from tabarena.benchmark.task.metadata import TaskMetadataCollection
from tabarena.contexts import AbstractArenaContext

tasks = [make_user_task(ds) for ds in my_datasets]            # Step 2
collection = TaskMetadataCollection.from_user_tasks(tasks)     # your suite
experiments = TabArenaExperimentBundle(                        # Step 5
    models=[("LightGBM", 0), ("RandomForest", 0), (my_gen, 10)],
    n_random_configs=50, preprocessing_pipelines=["default"],
).build_experiments(time_limit=3600)
context = AbstractArenaContext(task_metadata=collection, methods=[])   # no TabArena baselines
context.build_and_run_jobs(experiments, expname=results_dir, user_tasks=tasks, debug_mode=True)
leaderboard = context.compare(output_dir=eval_dir)             # Step 9

Everything below explains the decisions hidden in those ten lines.

Step 0: Scope the domain benchmark

Ask (or infer from context) what the downstream benchmark actually needs:

Question	Why it matters
Full model zoo, or only leaderboard math over results you already have?	tabarena (models, runner, splitting) vs. bencheval alone (Elo, win-rates, ranks, improvability from a results DataFrame; no tabarena dependency).
Do datasets have replicate or group structure (several rows per specimen, patient, sample)?	Both the outer splits (Step 3) and the inner bagging folds (Step 4) must be group-aware, and the inner ones need the data-foundry extra.
Do datasets have temporal structure?	Outer splits must be forward-in-time; TabArena's inner protocol handles time_on but not time_on together with group_on.
Do features arrive as thousands of numeric columns (spectra, expression arrays, embeddings)?	Foundation models carry feature and row caps; ModelConstraints (Step 5) skips incompatible pairs instead of failing them, and dimensionality reduction becomes a preprocessing decision (Step 7).
Do the domain's metrics differ from roc_auc / log_loss / rmse?	Any metric must be registered with AutoGluon by name before a task can reference it (Step 2).
Are there domain-specific models or whole pipelines?	Decides between a registered model (Step 6) and an ExternalSystemModel.
Will you host results so others can compare against them without rerunning?	Decides whether you need a methods.py roster, MethodMetadata storage config, and your own arena context (Step 10).
Cluster? PyPI release?	Steps 8 and 11.

Step 1: Depend on the right package(s)

bencheval is standalone and light (numpy, pandas, scipy, scikit-learn). It computes leaderboards from a results DataFrame you already have. Pull it in alone if the domain benchmark keeps its own models and splitting and only wants TabArena-grade leaderboard math. Its __init__ is empty, so import from submodules: from bencheval.evaluator import BenchmarkEvaluator.

tabarena adds the model registry, config generators and search spaces, the task and experiment runner, the contexts, plotting, and the artifact tiers. It depends on bencheval and on an AutoGluon pre-release. Extras that matter downstream: [benchmark] for model fitting (the core model set plus plot, text, preprocessing, data-foundry), [data-foundry] on its own if you only need grouped inner splits (see Step 4), [plot] for the figures compare renders, and one extra per extended model ([tabm], [realmlp], [tabpfn], ...). The registry's pip_extra on each ModelInfo names the extra a model needs.

Both packages publish PyPI pre-releases at one shared version, and every tabarena release pins bencheval==<same version>. Install with pip install --pre tabarena (or uv pip install --prerelease=allow), and pin the pair together in the downstream pyproject.toml. See Step 11 before choosing between the PyPI release and a git URL.

Step 2: Turn each dataset into a UserTask

tabarena.benchmark.task.UserTask is the local, OpenML-free task type. Its create_task computes the task's TabArenaTaskMetadata (problem type, sizes, dtype flags, per-split statistics) and save_task pickles dataset, splits and metadata to one file. Reference implementations: examples/benchmarking/run_quickstart_tabarena_custom_datasets.py (plain DataFrames), tabarena/benchmark/task/data_foundry/adapter.py::convert_curated_container_to_user_task (the converter BeyondArena uses, a complete "third-party dataset object to UserTask" example), and tests/tabarena/benchmark/task/test_user_task.py.

from sklearn.model_selection import RepeatedStratifiedKFold
from tabarena.benchmark.task import UserTask
from tabarena.benchmark.task.metadata import GroupLabelTypes
from tabarena.benchmark.task.user_task import from_sklearn_splits_to_user_task_splits

n_folds, n_repeats = 3, 10
cv = RepeatedStratifiedKFold(n_splits=n_folds, n_repeats=n_repeats, random_state=0)
splits = from_sklearn_splits_to_user_task_splits(cv.split(X, y), n_splits=n_folds)

task = UserTask(task_name="ramanbench/bacteria-id", task_cache_path=task_cache_dir)
wrapper = task.create_task(
    dataset=df, target_feature="species", problem_type="classification", splits=splits,
    eval_metric="log_loss", group_on="specimen_id", group_labels=GroupLabelTypes.PER_GROUP,
)
task.save_task(wrapper)
task.load().validate_metadata()   # recomputes from disk and raises on any diverging field

The data contract create_task enforces:

• dataset is one DataFrame that includes the target column, has a default RangeIndex, and has resolved dtypes. Numeric, category, string and datetime columns are supported; object columns raise. Categorical columns must already be category, not strings.
• splits is {repeat: {fold: (train_indices, test_indices)}} with positional Python int lists. Numpy integers fail validation, so call .tolist() (the sklearn helper does). Train and test must not overlap, test indices must not overlap across folds of one repeat, and every repeat must have the same number of folds. A single holdout is {0: {0: (train, test)}}.
• task_name must be unique on the machine: the cache file, the numeric task id and the results dataset key (a slug plus a hash of the name) all derive from it.
• eval_metric must be an AutoGluon metric name. Defaults come from tabarena.benchmark.task.metrics.DEFAULT_EVAL_METRIC_BY_PROBLEM_TYPE (roc_auc, log_loss, rmse). To use a domain metric, register it once with autogluon.core.metrics.make_scorer and insert it into autogluon.core.metrics.METRICS[<problem_type>], exactly as tabarena/metrics/custom_metrics.py does for the AMEX metric. Put the registration in a module every worker imports (your package __init__ or the module that defines your tasks), otherwise a Ray worker resolving the metric by name will not find it.
• group_on, group_labels, time_on, group_time_on, stratify_on, split_time_horizon and its unit are recorded on the task. They do not build the outer splits (you supply those); they drive the inner validation protocol (Step 4) and the preprocessing that wants group or time columns (Step 7). GroupLabelTypes.PER_GROUP means every row of a group shares the label (one specimen, many spectra); PER_SAMPLE means the label varies within a group (one patient, many visits). time_on together with group_on raises NotImplementedError downstream, and split_time_horizon is carried and logged but no split logic reads it yet.

Where tasks live: without task_cache_path a UserTask is saved under the OpenML cache root in tabarena_tasks/, which CacheConfig(openml=...) relocates (Step 8). A task saved there has a path-free task_id_str, so TaskMetadataCollection.from_user_tasks(tasks) is runnable as-is. A task saved to a custom task_cache_path should be handed to the runner explicitly via build_and_run_jobs(..., user_tasks=tasks), as the example does. UserTask.from_task_id_str reconstructs a task from the id string, which is what lets a compute node load a task it never created.

Step 3: Outer splits: repeated k-fold, adaptive repeats, groups, time

TabArena does not ship its own splitter. You build the outer splits with scikit-learn (or Data Foundry) and hand them to create_task. What to reproduce is the protocol:

• Repeated k-fold, not a single holdout. TabArena v0.1 uses 3 outer folds and a repeat count that depends on the training-set size: 10 repeats below 2,500 training rows, 1 repeat above 250,000, otherwise 3. The policy is _get_n_repeats in tabarena/benchmark/task/metadata/fetch_metadata.py (a private function; port the three lines rather than importing it) and the constants are restated in examples/advanced/run_get_tabarena_datasets_from_openml.py. Verify a ported policy against load_curated_task_metadata() from the same module, not just against the docstring.
• Grouped data has no repeated splitter in scikit-learn. Loop StratifiedGroupKFold / GroupKFold with shuffle=True and a different random_state per repeat, then feed the flat iterator to from_sklearn_splits_to_user_task_splits. Data Foundry's get_recommended_grouped_splits (in data_foundry.curation_recommendations) is the splitter TabArena itself uses for grouped inner folds, so using it for the outer splits keeps the two layers consistent.
• Temporal data needs forward-in-time outer splits (train strictly before test). TabArena's inner helper split_time_index_into_intervals in tabarena/benchmark/exec_models/autogluon_utils.py bins a time column into row-balanced contiguous intervals; reuse it to define the outer test window, then hold out the latest interval.
• Audit every dataset for group structure before the first real run. A dataset wrongly treated as row-independent leaks train information into test and inflates every model's score. RamanBench caught this late, on datasets that had already shipped without it.

Decide the repeat count from data instead of guessing. compare(output_dir=..., compute_fold_similarity=True, fold_similarity_kwargs={"target_reliability": 0.8}) writes fold_similarity.csv with a folds_needed_for_stability@0.8 column per dataset (a Spearman-Brown extrapolation over the per-split rankings, from BenchmarkEvaluator.rank_datasets_by_fold_similarity). Take min(folds_needed, num_folds) per dataset, commit the resulting (dataset, split) table, and expose it as a "core" subset predicate with tasks_in_frame (Step 10). That is exactly how BeyondArena's core subset was derived; the recipe is examples/!experimental/run_generate_beyondarena_core_subset.py. The same mixin offers dataset_representativeness and jitter_all_datasets for pruning redundant or noisy datasets while curating the suite.

Step 4: Inner validation and bagging parity

The inner validation protocol is one object, ValidationProtocol (tabarena.benchmark.validation_protocol), owned by the arena context. TabArenaContext declares TABARENA_V0PT1_VALIDATION_PROTOCOL (8 bagging folds x 1 set, plain stratified splits) and BeyondArenaContext declares BEYONDARENA_VALIDATION_PROTOCOL (8x1, 5 folds x 5 sets at or below 500 training group instances, task-specific inner splits, class-adaptive folds). build_jobs stamps the protocol onto every bagged experiment and refuses one that carries another protocol unless the context was built with official_validation_protocol=False. Declare your arena's protocol the same way, on your context subclass (Step 10):

from tabarena.benchmark.validation_protocol import ValidationProtocol

class MyArenaContext(AbstractArenaContext):
    benchmark_name = "MyArena"
    OFFICIAL_VALIDATION_PROTOCOL = ValidationProtocol(
        tiny_num_bag_folds=5, tiny_num_bag_sets=5, tiny_max_group_instances=500,
        task_specific_validation=True, adapt_num_folds_to_n_classes=True, name="MyArena",
    )
    OFFICIAL_BUNDLE_HINT = "MyArenaExperimentBundle"

Without a subclass, a bare AbstractArenaContext(validation_protocol=...) enforces exactly the protocol it is given. Pick a protocol and keep it: an implicit, size-dependent bagging behavior is not a reproducible protocol and makes historical and new numbers incomparable in ways that are hard to detect later. Every result records the protocol it ran under (results["validation_protocol"]), the processed MethodMetadata.validation_protocol carries its key, and context.validation_protocol_status tells official from custom results, so a change of protocol is visible rather than silent. If an earlier version of the domain benchmark ran without explicit bagging control, say so in its changelog rather than presenting old and new numbers as comparable.

The task's group_on / time_on reach the inner folds only under a protocol with task_specific_validation=True (BeyondArena's; TabArena-v0.1's has it off). A grouped domain benchmark run under the TabArena protocol therefore leaks groups across bagging folds while its outer splits look correct. Declare a task-specific protocol for grouped or temporal data.

With task-specific validation, resolve_validation_splits in tabarena/benchmark/exec_models/autogluon_utils.py builds group-disjoint or forward-in-time inner folds and passes them to AutoGluon as custom splits. The grouped branch imports Data Foundry, so install tabarena[data-foundry] (part of [benchmark]). Fold counts adapt at run time and the adaptation is recorded in the result (clamps): fewer groups than folds clamps the fold count and sets one repeat, a minority class smaller than the fold count does the same, and a time_on task always uses one repeat.

Two further execution modes exist for models that carry their own validation: holdout_experiments=True keeps a single task-aware holdout split without bagging, and outer_experiments=True fits once on all training data with no validation split (the mode most foundation models are benchmarked in; see examples/beyondarena/advanced/run_quickstart_beyondarena_without_bagging.py). Neither is an official flavour: they run outside the protocol by construction, need no flag, and their results are recorded as such (holdout:<key> / outer). A method run this way still gets the shared outer protocol and scoring, but no HPO simulation.

Step 5: Pick the models and define your bundle

Enumerate the zoo from the registry instead of hardcoding names:

from tabarena.models import get_model_registry
for key, info in get_model_registry().items():
    print(key, info.method_metadata.display_name, info.method_metadata.compute, info.pip_extra)

tabarena.models.utils.get_configs_generator_from_name("TabM") returns a model's generator, whose model_cls and manual_configs[0] are what examples/advanced/run_tabarena_model_on_your_data.py uses to fit a TabArena model outside the benchmark. tabarena.models.prefetch.prefetch_weights([...]) downloads foundation-model checkpoints before a cluster run.

Three bundles ship; pick by protocol, not by name:

Bundle	Random configs	Preprocessing	Task-aware validation	Time limit
TabArenaV0pt1ExperimentBundle	200	AutoGluon "default"	off	1 h
TabArenaExperimentBundle	required field	preprocessing_pipelines required	on	1 h
BeyondArenaExperimentBundle	25	"tabarena_default"	on	4 h

A domain benchmark should subclass TabArenaExperimentBundle and pin its own defaults, so every run script shares one protocol:

from dataclasses import dataclass, field
from typing import ClassVar
from tabarena.benchmark.experiment import ModelConstraints, TabArenaExperimentBundle

@dataclass(kw_only=True)
class RamanBenchExperimentBundle(TabArenaExperimentBundle):
    n_random_configs: int = 50
    preprocessing_pipelines: list[str] = field(default_factory=lambda: ["default"])
    DEFAULT_TIME_LIMIT: ClassVar[int] = 3600
    custom_model_constraints: dict[str, ModelConstraints] = field(
        default_factory=lambda: {"TA-TABICL": ModelConstraints(max_n_features=2000)},
    )

ModelConstraints (fields max_n_features, max_n_samples_train_per_fold, min_n_samples_train_per_fold, max_n_classes, regression_support) drops (model, dataset) jobs that violate a model's limits instead of letting them fail at fit time. The bundle already carries TabPFNv2, TabICL and Mitra caps under their AutoGluon keys; high-dimensional domains (spectra with thousands of wavenumbers) will hit these and should add their own.

models entries are (name_or_generator, n_configs) with n_configs as 0 (default only), an int, or "all"; a third tuple element pins hyperparameters into every config of that model, e.g. ("XGBoost", 0, {"n_estimators": 100}). build_experiments(time_limit=..., num_cpus=..., num_gpus=..., memory_limit=...) bakes the compute budget into every experiment. For GPU models pass the card's VRAM in GB as memory_limit: AutoGluon budgets parallel bagging folds against the reported memory, and node RAM lets co-scheduled folds OOM the GPU (this is what tabflow_slurm's fake_memory_for_estimates does).

Step 6: Model registry: reuse vs. layer your own

Do not fork TabArena's model wrappers. Two registries coexist:

TabArena's registry supplies the general-purpose zoo. A domain registry, layered in the downstream package, holds only the models that do not belong upstream. tabarena.models.register_model_info is the documented hook for this: it exists so an extension package can add ModelInfo entries that discover_models()'s walk over tabarena.models cannot see. Call it from your package's __init__, and the model becomes addressable by name in a bundle exactly like a built-in one.

from tabarena.models import ModelInfo, register_model_info
from tabarena.models._method_metadata import ModelDescriptor
from tabarena.utils.config_utils import ConfigGenerator

spectral_cnn = ModelDescriptor(display_name="SpectralCNN", compute="gpu", is_bag=True,
                               reference_url="https://...", date_introduced="2026-03")
gen_spectral_cnn = ConfigGenerator(model_cls=SpectralCNNModel, manual_configs=[{}],
                                   search_space={...})
register_model_info(ModelInfo(
    model_cls=SpectralCNNModel,
    search_space=gen_spectral_cnn,
    method_metadata=spectral_cnn.method_metadata(
        method="SpectralCNN", ag_key="SPECCNN", config_default="SpectralCNN_c1_default_BAG_L1",
        suite="ramanbench-2026-09",
    ),
    pip_extra=("ramanbench[cnn]",),
))

Build the model class the way TabArena builds its own: an AutoGluon AbstractModel (or AbstractTorchModel for torch) subclass with ag_key and ag_name (the generator asserts both), _supported_problem_types, _fit, _preprocess, and optionally a warmup classmethod for untimed one-off costs such as imports, JIT and CUDA context (the fairness contract in tabarena/models/warmup.py says a warm-up may never touch task data). The add-model skill in this repo describes the wrapper anatomy and points at reference models per family; it applies unchanged to a model living in another package, minus the pyproject.toml edit. Config names are {method}_c{i}<suffix>_BAG_L1 for curated configs and _r{i}<suffix> for random ones, where the bundle appends the preprocessing pipeline name as the suffix (_default for the AutoGluon "default" pipeline used above, none for "tabarena_default", see _build_experiments_for_pipeline in benchmark/experiment/bundle.py); a declared config_default must match the post-rename name, so read it off scripts/run_process_method.py <run>/data once results exist instead of authoring it by hand, and a single-config model (can_hpo=False) can leave it unset (processing records the lone config). Mirror tests/tabarena/models/test_all_models.py for a registry-driven smoke test of your own models.

Two things bite here. A model class must live in an importable module, never in __main__: Ray workers cannot unpickle it otherwise (debug_mode=True runs in-process and hides this). And a model that does its own validation, tuning or ensembling (a chemometrics pipeline, an AutoML tool, an LLM agent) is a system, not a model: subclass tabarena.benchmark.exec_models.ExternalSystemModel, implement _fit_system / _predict / _predict_proba, pair it with SystemConfigGenerator, and run the bundle with system_experiments=True (examples/benchmarking/run_quickstart_tabarena_system.py). Its results are recorded as baselines with no HPO simulation, which is the honest representation of a self-tuning pipeline.

If a domain-specific model turns out to be broadly useful beyond the domain, that is a signal it belongs upstream. Point the user at add-model and at Step 13.

Step 7: Preprocessing: layer, do not fork

TabArena splits preprocessing into a model-agnostic AutoGluon feature generator (applied to the whole frame) and a model-specific step injected into each model's hyperparameters. tabarena/benchmark/preprocessing/pipeline.py::resolve_preprocessing_pipeline resolves a named pipeline to that pair, and build_feature_generator forwards the task's group_cols, group_labels and group_time_on to any generator class whose __init__ accepts them. examples/!experimental/run_tabarena_preprocessing.py demonstrates both stages on a frame with numeric, categorical, text, datetime and grouped columns.