Divide and Conquer Self-Supervised Learning for High-Content Imaging

Lucas Farndale, Paul Henderson, Edward W Roberts, Ke YuanSubmission to ICLR 2026 · Withdrawn

TL;DR

Self-supervised learning (SSL) models tend to suffer from "simplicity bias," often ignoring subtle, complex patterns in favor of dominant, easily learned features. The authors introduce SpliCER (Split Component Embedding Registration), a training framework that deconstructs an image into meaningful components (e.g., individual image channels or segmentation masks) and registers each to a dedicated chunk of the primary embedding space. This divide-and-conquer strategy forces the network to learn subtle, high-value features alongside simple ones without requiring image deconstruction at inference time.

What Problem Does This Paper Address?

In scientific and engineering applications—such as multiplex spatial proteomics, hyperspectral satellite imaging, and digital pathology—images pack immense information across multiple channels and localized regions. However, common discriminative SSL methods (like SimCLR or VICReg) are prone to shortcut learning: when simple, high-variance features and subtle, low-variance features coexist, the joint-embedding objective readily minimizes loss using only the simplest features, treating subtle patterns as superfluous noise.

Prior multi-modal attempts to align separate views into a single shared latent space often backfire: forcing full representations to align penalizes unshared information. If a critical feature exists only in one channel or image sub-region, the network actively penalizes gradients representing that feature. Consequently, existing models frequently fail to extract subtle biomarkers, cellular subtypes, or fine-grained spectral signatures.

Methods

SpliCER retains a standard SSL backbone while restructuring how the embedding is trained:

  1. Image Deconstruction: An input image $\bar{x}$ is decomposed into $n$ distinct components $x_1^*, \dots, x_n^*$. Deconstruction can be natural (e.g., grouping spectral or protein channels in multiplex imagery) or derived via off-the-shelf automated segmentation tools (e.g., HoVer-Net for cell nuclei vs. connective tissue, or HybridGNet for lung masks).
  2. Chunked Embedding Registration: The composite image is passed through a primary encoder and projector to yield embedding $\bar{e}$. Each deconstructed component is passed through a paired encoder/projector to yield component embeddings $e_i^*$.
  3. Decoupled Objective: Rather than aligning the entire primary embedding to each component, $\bar{e}$ is partitioned into $n$ contiguous chunks: $\bar{e} = [\bar{e}_1 \parallel \bar{e}_2 \parallel \dots \parallel \bar{e}_n]$. Each component embedding $e_i^*$ is paired strictly with its dedicated slice $\bar{e}_i$ under any standard SSL loss (e.g., VICReg, SimCLR).
  4. Inference Simplicity: Because the primary encoder receives the full, non-deconstructed image, the deconstruction pipeline and paired branches are discarded after pretraining. Standard downstream evaluation runs on the primary encoder alone.

Mathematically, this chunking decouples the gradient flow. Features from an easy component cannot monopolize the optimization pathways of chunks assigned to more subtle components.

Composite Image ──> Primary Encoder ──> Primary Projector ──> [ Chunk 1 | Chunk 2 | ... | Chunk n ]
                                                                   │         │               │
Component 1     ──> Paired Encoder  ──> Projector 1       ──> [ Embed 1 ]   │               │
                                                                   ▲         │               │
                                                              SSL Loss 1     ▼               │
Component 2     ──> Paired Encoder  ──> Projector 2       ─────────────> [ Embed 2 ]         │
                                                                             ▲               ▼
                                                                        SSL Loss 2      SSL Loss n

Datasets

The authors evaluated SpliCER across a synthetic benchmark and diverse real-world domains:

  • MNIST-CIFAR (Synthetic Benchmark): Concatenates MNIST digits with CIFAR-10 objects of identical class labels to isolate simplicity bias under input randomization ($r\text{MNIST}$, $r\text{CIFAR}$, and $r\text{NONE}$).
  • Orion-CRC (Spatial Proteomics): Multiplex immunofluorescence data. Evaluated on a broader, simple Cell Types classification task and a fine-grained, complex T Cells subtyping task driven by low-variance markers (CD4, CD8, FOXP3).
  • EuroSAT & MMEarth (Hyperspectral Earth Observation): 13-band Sentinel-2 satellite imagery used to assess land-use classification and cross-dataset distribution shift.
  • NCT-CRC-100K & Camelyon17 (Histopathology): Colorectal and breast cancer histology slides deconstructed into nuclei vs. connective tissue via HoVer-Net to test out-of-distribution metastasis detection.
  • ChestMNIST & PneumoniaMNIST (Radiology): Chest X-rays deconstructed into lung fields vs. background using HybridGNet to evaluate subtle pneumonia detection.

Key Insights and Ideas

  • Information Bottlenecks Filter Out Subtle Information: In standard SSL, mutual information maximization naturally prioritizes high-variance features shared across views. Complex or fine-grained signals are compressed away because they do not noticeably decrease the overall objective.
  • Shared Latent Spaces Overconstrain Models: Mapping multiple specialized inputs into one identical embedding vector forces non-overlapping features to align with noise or constant representations, creating gradient conflicts that wash out subtle signals.
  • Gradient Decoupling Prevents Feature Erasure: By assigning component-specific objectives to separate embedding chunks, each sub-pathway maintains supervisory pressure for its respective features, compelling the shared encoder backbone to retain both simple and complex patterns.

Key Results

SpliCER consistently outperformed standard SSL baselines and shared-embedding methods ($\Sigma\text{-JE}$, TriDeNT) across multiple domains:

  • Overcoming Simplicity Bias (MNIST-CIFAR): On randomized MNIST evaluations (where models must rely on CIFAR features), SpliCER prevented shortcut collapse, raising accuracy up to 68.4% (approaching the 72.6% ceiling achieved by training exclusively on CIFAR without MNIST shortcuts).
  • Spatial Proteomics (Orion-CRC): On the challenging T-cell subtyping task, SpliCER dramatically outperformed baseline VICReg (81.36% vs. 56.65%) and SimCLR (84.69% vs. 55.39%) while maintaining ~89–90% accuracy on standard cell types. It also yielded significantly higher correlations with individual marker intensities.
  • Hyperspectral Land Use: On 13-band EuroSAT classification, SpliCER achieved 89.21% (VICReg) and 90.44% (SimCLR) versus baselines around ~84.8%. Pretraining on MMEarth and transferring to EuroSAT yielded a massive gain under distribution shift (70.15% vs. 49.76% for VICReg; 70.61% vs. 59.86% for SimCLR).
  • Segmentation-Aided Medical Imaging:
  • On Camelyon17 metastasis detection, SpliCER achieved 82.30% (VICReg) and 86.74% (SimCLR) compared to baselines of 68.22% and 83.46%.
  • On PneumoniaMNIST, accuracy jumped to 87.18% (VICReg) and 91.19% (SimCLR) compared to ~84% baselines.

Limitations and Future Work

  • Downstream Simplicity Bias Persists: While SpliCER forces the encoder to preserve complex features, downstream linear probes or classifier heads can still latch onto simple shortcut features during supervised fine-tuning.
  • Dependency on Deconstruction Quality: The method requires an informed strategy to segment or split images. Poorly segmented masks or arbitrary chunking could diminish performance.
  • Cross-Component Features: Features that inherently span across multiple segmented regions or channels might suffer from reduced supervisory signal if separated rigidly.

Summary updated

Abstract

Self-supervised representation learning methods often fail to learn subtle or complex features, which can be dominated by simpler patterns which are much easier to learn. This limitation is particularly problematic in applications to science and engineering, as complex features can be critical for discovery and analysis. To address this, we introduce Split Component Embedding Registration (SpliCER), a novel architecture which splits the image into sections and distils information from each section to guide the model to learn more subtle and complex features without compromising on simpler features. SpliCER is compatible with any self-supervised loss function and can be integrated into existing methods without modification. The primary contributions of this work are as follows: i) we demonstrate that existing self-supervised methods can learn shortcut solutions when simple and complex features are both present; ii) we introduce a novel self-supervised training method, SpliCER, to overcome the limitations of existing methods, and achieve significant downstream performance improvements; iii) we demonstrate the effectiveness of SpliCER in cutting-edge medical and geospatial imaging settings. SpliCER offers a powerful new tool for representation learning, enabling models to uncover complex features which could be overlooked by other methods.
BibTeX
@misc{farndale2025divide,
  author = {Lucas Farndale and Paul Henderson and Edward W Roberts and Ke Yuan},
  title = {{Divide and Conquer Self-Supervised Learning for High-Content Imaging}},
  year = {2025},
  eprint = {2503.07444},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.07444},
}

Loading related content...

Figure