K-Fold Model Averaging

Overview

Train K separate models on K-fold splits, save each checkpoint, then average their predictions at inference. Simpler than stacking — no meta-learner needed — yet typically captures 80% of the ensemble benefit. Works with any model type (CNN, transformer, tree-based).

Quick Start

import numpy as np
import tensorflow as tf

# Training: save best model per fold
from sklearn.model_selection import GroupKFold

gkf = GroupKFold(n_splits=5)
for fold, (train_idx, val_idx) in enumerate(gkf.split(df, groups=df.patient_id)):
    model = build_model()
    checkpoint = tf.keras.callbacks.ModelCheckpoint(
        f'model_fold{fold}.h5', save_best_only=True,
        monitor='val_loss', mode='min'
    )
    model.fit(train_data, epochs=20, callbacks=[checkpoint],
              validation_data=val_data)

# Inference: load all folds, average predictions
models = [tf.keras.models.load_model(f'model_fold{i}.h5') for i in range(5)]
predictions = sum(m.predict(test_data) for m in models) / len(models)

Key Decisions

• Simple average: equal weights work well when folds are balanced; use weighted average if fold quality varies
• Best checkpoint per fold: save_best_only prevents averaging poorly-converged models
• GroupKFold: prevent data leakage when samples share a group (e.g., same patient)
• Memory tradeoff: K models in memory simultaneously — use sequential prediction if GPU-constrained

References

• Source: siim-cov19-efnb7-yolov5-infer
• Competition: SIIM-FISABIO-RSNA COVID-19 Detection