ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images

Nicolas Bourriez, Ihab Bendidi, Ethan Cohen, Gabriel Watkinson, Maxime Sanchez, Guillaume Bollot, Auguste Genovesio

TL;DR

This paper introduces ChAda-ViT, a novel Vision Transformer architecture that can effectively learn from microscopy images with varying numbers and types of channels. By incorporating "Inter-Channel Attention," ChAda-ViT can capture relationships between different channels, leading to superior performance on biological tasks and enabling the creation of a unified representation space for diverse bioimaging datasets.

What Problem Does This Paper Address?

Biological images, unlike standard RGB photos, are highly heterogeneous. They come from various microscopy techniques, each producing images with different numbers and types of channels, where each channel holds unique biological information. Existing deep learning models, often designed for RGB images, struggle with this variability. They typically focus on spatial attention within individual channels, neglecting the crucial inter-channel relationships. This limitation prevents the creation of a unified representation that can bridge different experiments and microscopy types, hindering large-scale pre-training and interdisciplinary studies.

Methods

The core innovation is ChAda-ViT, a Channel Adaptive Vision Transformer. It tackles the channel heterogeneity problem through several key mechanisms:

  • Token Padding and Masking: To handle variable channel counts, ChAda-ViT uses a padding strategy to create uniform-length sequences of image patches. A masking mechanism then ensures that these padding tokens are ignored during attention computations.
  • Patch-wise Channel Processing: Instead of processing the entire image at once, ChAda-ViT processes each channel independently, splitting it into patches.
  • Channel and Positional Embeddings: Each patch is enriched with both positional embeddings (to retain spatial context) and channel embeddings (to indicate the channel's origin). This dual embedding helps the model distinguish between patches from different channels at the same spatial location.
  • Inter-Channel Attention: This is the key novelty. ChAda-ViT incorporates an attention mechanism that explicitly models relationships between different channels, complementing the standard intra-channel (spatial) attention. This allows the model to leverage information across channels for a more comprehensive understanding.

Datasets

  • IDRCell100k: A newly introduced dataset curated from the Image Data Resource. It comprises 79 experiments across 7 microscopy modalities, with a multitude of channel types and counts (1-10 channels per image), totaling 104,093 multiplexed images. This dataset is designed to be a benchmark for evaluating channel-adaptive architectures.
  • Downstream Datasets: The models were evaluated on several existing bioimaging datasets for various tasks:
  • BloodMNIST: For cell type classification.
  • CYCLOPS: For protein localization.
  • BBBC048: For cell cycle stage classification.
  • NF-kB Nuclear Translocation Assay: For regression tasks.
  • BBBC021: For channel reconstruction.
  • Bray et al.: For MoA prediction.

Key Insights and Ideas

  • Inter-channel relationships are critical: The paper strongly suggests that ignoring how different channels interact is a significant missed opportunity in bioimage analysis, and ChAda-ViT's success hinges on capturing these relationships.
  • Unified representation is achievable: ChAda-ViT demonstrates that it's possible to create a single, consistent embedding space for images from vastly different microscopy setups and with varying channel configurations.
  • Robustness in low-data regimes: ChAda-ViT maintains its performance advantage even when trained with very limited data, making it practical for real-world biological research scenarios where data can be scarce.
  • Attention maps reveal focus: Visualizations of ChAda-ViT's attention maps show how it effectively identifies and correlates features across different channels, highlighting biologically relevant areas that might be missed by single-channel approaches.

Key Results

  • ChAda-ViT outperforms existing methods (including a standard one-channel ViT) in 6 out of 8 downstream tasks evaluated.
  • The model shows strong performance in linear probing across various datasets, even in low-data regimes (100%, 10%, and 1% of training data).
  • In channel prediction tasks (e.g., BBBC021), ChAda-ViT demonstrates enhanced spatial distribution accuracy in reconstructing images compared to the one-channel approach.
  • The paper shows that using "All Tokens" for evaluation (rather than just the CLS token) leads to better performance, and the model exhibits good scalability with increased parameters and embedding sizes.

Limitations and Future Work

  • Intra-channel reliance: The authors acknowledge that ChAda-ViT might fall short in tasks heavily reliant only on intra-channel information. Future work could aim to better integrate both inter- and intra-channel dynamics.
  • Foundation Model Potential: They suggest exploring ChAda-ViT as a basis for a foundation model, leveraging massive datasets to unlock new capabilities and applications.
  • Dataset Augmentation: The unified embedding space could be used to augment existing datasets with additional channels based on learned representations, reducing experimental costs.
  • Bridging Datasets: The ability to create a unified embedding space is highlighted as a key enabler for cross-experimental studies and for bridging numerous, disconnected biological datasets.

Summary updated

Abstract

Unlike color photography images, which are consistently encoded into RGB channels, biological images encompass various modalities, where the type of microscopy and the meaning of each channel varies with each experiment. Importantly, the number of channels can range from one to a dozen and their correlation is often comparatively much lower than RGB, as each of them brings specific information content. This aspect is largely overlooked by methods designed out of the bioimage field, and current solutions mostly focus on intra-channel spatial attention, often ignoring the relationship between channels, yet crucial in most biological applications. Importantly, the variable channel type and count prevent the projection of several experiments to a unified representation for large scale pre-training. In this study, we propose ChAda-ViT, a novel Channel Adaptive Vision Transformer architecture employing an Inter-Channel Attention mechanism on images with an arbitrary number, order and type of channels. We also introduce IDRCell100k, a bioimage dataset with a rich set of 79 experiments covering 7 microscope modalities, with a multitude of channel types, and counts varying from 1 to 10 per experiment. Our architecture, trained in a self-supervised manner, outperforms existing approaches in several biologically relevant downstream tasks. Additionally, it can be used to bridge the gap for the first time between assays with different microscopes, channel numbers or types by embedding various image and experimental modalities into a unified biological image representation. The latter should facilitate interdisciplinary studies and pave the way for better adoption of deep learning in biological image-based analyses. Code and Data available at https://github.com/nicoboou/chadavit.
BibTeX
@inproceedings{bourriez2024chada,
  author = {Nicolas Bourriez and Ihab Bendidi and Ethan Cohen and Gabriel Watkinson and Maxime Sanchez and Guillaume Bollot and Auguste Genovesio},
  title = {{ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01098},
}
Figure