Greedy Mask Overlap Resolution

Overview

Instance segmentation models often produce overlapping masks, but many evaluation metrics (and biological reality) require non-overlapping instances. This technique processes masks in descending confidence order and subtracts already-claimed pixels from each new mask. Simple, fast, and effective — commonly used in cell segmentation where dense packing makes overlap inevitable.

Quick Start

import numpy as np

def resolve_overlaps(masks, scores, min_pixels=75):
    order = np.argsort(-scores)
    used = np.zeros(masks[0].shape, dtype=np.uint8)
    result = []
    for idx in order:
        mask = (masks[idx] > 0).astype(np.uint8)
        mask = mask * (1 - used)  # remove already-claimed pixels
        if mask.sum() >= min_pixels:
            used = np.clip(used + mask, 0, 1)
            result.append(mask)
    return result

# Usage with Mask R-CNN output
masks = output['masks'].cpu().numpy()  # (N, H, W)
scores = output['scores'].cpu().numpy()
clean_masks = resolve_overlaps(masks, scores, min_pixels=75)

Workflow

1. Sort all predicted masks by confidence score descending
2. Initialize an empty occupancy map (zeros, same H×W as image)
3. For each mask: subtract occupied pixels, check remaining area ≥ min_pixels
4. If large enough, add to result and update occupancy map
5. Encode surviving masks as RLE for submission

Key Decisions

• Sort by confidence: highest-confidence masks get priority for contested pixels
• min_pixels threshold: discard masks that become too small after overlap removal; tune per dataset (75-150 for cells)
• Per-class min_pixels: use different thresholds per class if instance sizes vary significantly
• Binary occupancy: simple and fast; for soft overlap, use IoU-based merging instead

References

• Sartorius - Starter Torch Mask R-CNN
• Positive score with Detectron 3/3 - Inference