Pitfalls of Conditional Batch Normalization for Contextual Multi-Modal Learning

Ivaxi Sheth, Aamer Abdul Rahman, Mohammad Havaei, Samira Ebrahimi KahouarXiv 2022 · DOI · arXiv

Abstract

Conditional Batch Normalization (CBN) is a popular technique for fusing multi-modal data, such as images and auxiliary text attributes. However, this paper reveals a significant "shortcut learning" risk: models often ignore primary visual features entirely in favor of auxiliary metadata. This leads to impressive training performance that collapses instantly if metadata is missing or noisy at test time. Through "network surgery" and systematic masking, the authors demonstrate how CBN can inadvertently turn computer vision tasks into simple metadata lookups.

Introduction

In deep learning, combining different data modalities—such as an image of a bird with a textual description of its attributes—is a common strategy to improve classification performance. Conditional Batch Normalization (CBN) has emerged as a standard tool for this fusion. It works by using auxiliary "context" data to modulate the internal activations of a Convolutional Neural Network (CNN) via affine transformations (shifting and scaling).

The fundamental challenge is ensuring the model learns a true fusion of modalities rather than relying on one while ignoring the other. If a model achieves near-perfect accuracy by ignoring the image and only processing metadata, it has failed to learn robust visual features. This paper investigates whether CBN encourages "shortcut learning," where the network finds the path of least resistance to high accuracy rather than learning generalizable representations.

Methodology

The authors employ several diagnostic techniques to assess the reliance of CBN-equipped models on visual versus auxiliary data:

  1. CBN Baseline: A standard ResNet-18 architecture where Batch Normalization layers are replaced with CBN layers modulated by a multi-layer perceptron (MLP) processing auxiliary attributes.
  2. Test-time Masking: Systematically removing fractions of auxiliary data during inference to measure accuracy degradation.
  3. Train-time Masking: Randomly hiding attributes during training to force the network to utilize the visual stream.
  4. Supplementary Network: A dual-network approach where a standard vision-only BN network guides the CBN network via Kullback–Leibler (KL) divergence loss, attempting to "ground" the CBN model's visual representations.
  5. Ablation (No Visual Data): A sanity check where image data is removed entirely to determine if the network can predict labels using only auxiliary metadata.

Datasets

The study evaluates two distinct datasets to determine if shortcut behavior is universal:

  • CUB-200-211: A dataset of natural bird images featuring 312 binary and continuous attributes (e.g., "has a red beak").
  • TIL (Tumor-Infiltrating Lymphocytes): A medical histology dataset for cancer type classification, with metadata including clinical information like patient age, gender, and tumor size.

Results and Discussion

The Shortcut Trap

On the CUB dataset, the CBN model achieved a near-perfect 99.2% accuracy, significantly outperforming the vision-only BN model. However, as shown in Figure 1, this performance is highly suspicious due to the speed of convergence.

Accuracy vs epoch for CUB and TIL datasets comparing BN and CBN models.

The diagnostic "network surgery" revealed the extent of the shortcut. When attributes were removed at test time for the CUB model, accuracy plummeted from 99.2% to a mere 0.5%—worse than random guessing. This indicates the model learned virtually zero visual features, treating the attributes as a "cheat sheet."

Test-Time Sensitivity

The impact of missing metadata is further explored through test-time masking. Figure 2 illustrates how accuracy scales with the fraction of available auxiliary data.

Accuracy vs fraction of test-time auxiliary data on CUB and TIL datasets.

For CUB, the model requires very little metadata to achieve high accuracy, but its performance is non-existent without it. In contrast, the TIL dataset shows a different trend: while the model still relies on metadata, it retains some visual capability (dropping from 62.9% to 19.3% when metadata is removed). This suggests that if metadata is less predictive of the label than the image, the shortcut is less dominant.

Failure of Grounding and Data Types

Surprisingly, the "supplementary network" approach failed to prevent shortcut learning; the pull of the metadata was too strong for the KL-divergence loss to overcome. Furthermore, using continuous attribute values (0.0 to 1.0) instead of binary values exacerbated the issue, with models converging to 100% accuracy in just two epochs, a clear sign of memorization.

Training Model Testing Model CUB Accuracy TIL Accuracy
Standard BN (Vision Only) BN 79.9% 51.1%
CBN Baseline CBN Baseline 99.2% 62.9%
CBN Baseline CBN (No attributes) 0.5% 19.3%
CBN (No visual data) CBN (No visual data) 99.1% 20.8%

The "No visual data" ablation for CUB is particularly telling: the model performs nearly identically (99.1%) without looking at the image at all.

Conclusion

The effectiveness of CBN is brittle and highly dependent on the quality of contextual information. While it can boost performance, it often does so by enabling "shortcuts" that bypass primary visual learning. This is a critical concern in high-stakes fields like medicine, where metadata may be inconsistent or missing. Future research must focus on fusion mechanisms that guarantee the utilization of primary modalities rather than allowing them to serve as mere backdrops for metadata.

Summary updated

Abstract

Humans have perfected the art of learning from multiple modalities through sensory organs. Despite their impressive predictive performance on a single modality, neural networks cannot reach human level accuracy with respect to multiple modalities. This is a particularly challenging task due to variations in the structure of respective modalities. Conditional Batch Normalization (CBN) is a popular method that was proposed to learn contextual features to aid deep learning tasks. This technique uses auxiliary data to improve representational power by learning affine transformations for convolutional neural networks. Despite the boost in performance observed by using CBN layers, our work reveals that the visual features learned by introducing auxiliary data via CBN deteriorates. We perform comprehensive experiments to evaluate the brittleness of CBN networks to various datasets, suggesting that learning from visual features alone could often be superior for generalization. We evaluate CBN models on natural images for bird classification and histology images for cancer type classification. We observe that the CBN network learns close to no visual features on the bird classification dataset and partial visual features on the histology dataset. Our extensive experiments reveal that CBN may encourage shortcut learning between the auxiliary data and labels.
BibTeX
@misc{sheth2022pitfalls,
  author = {Ivaxi Sheth and Aamer Abdul Rahman and Mohammad Havaei and Samira Ebrahimi Kahou},
  title = {{Pitfalls of Conditional Batch Normalization for Contextual Multi-Modal Learning}},
  year = {2022},
  eprint = {2211.15071},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.15071},
}
Figure