Multi-layer Representation Learning for Robust OOD Image Classification

Aristotelis Ballas, Christos Diou

Abstract

Standard Convolutional Neural Networks (CNNs) often struggle with Out-of-Distribution (OOD) generalization, frequently relying on spurious correlations between objects and their backgrounds. This paper introduces a robust classification framework that adapts the Hypercolumns technique to a ResNet-18 architecture. By integrating low-level structural features from early layers with high-level semantic features from deeper layers, the model learns invariant representations that focus on the object of interest rather than its context. Evaluated on the NICO dataset using a leave-one-domain-out protocol, the proposed method demonstrates significant accuracy improvements and superior object localization in saliency maps compared to baseline models.

Introduction

Deep learning models typically achieve high accuracy when test data is Independent and Identically Distributed (IID) relative to training data. However, they often fail in Domain Generalization (DG) scenarios. A primary cause is the learning of "spurious correlations"—for instance, a model might associate a blue background with an airplane rather than learning the airplane's physical structure. This research aims to force models to learn invariant features—physical characteristics that remain constant across different environments, such as the four legs of a horse regardless of whether it is in snow or on a farm.

Methodology: Hypercolumn Adaptation

The authors leverage the concept of Hypercolumns, originally used for image segmentation, to enhance classification robustness. Unlike standard CNNs where classification relies solely on the final layer, this architecture preserves "local" information from earlier layers to prevent the loss of structural detail.

Proposed Hypercolumn-based ResNet-18 architecture showing multi-layer feature extraction and fusion.

The proposed pipeline involves several key stages:

  1. Feature Extraction: Extracting feature maps from multiple intermediate convolutional layers and residual blocks of a ResNet-18 backbone.
  2. Processing: Each map undergoes a $1 \times 1$ convolution followed by bilinear interpolation for upsampling.
  3. Fusion: To capture class-level essence without pixel-level noise, maps are passed through a $56 \times 56$ MaxPool2D layer.
  4. Classification: These features are concatenated into a single "hyper vector" and processed by a final fully connected layer with a softmax activation.

By simultaneously processing low-level geometric features (edges, shapes) and high-level semantic concepts, the model effectively disentangles objects from their backgrounds.

Experimental Setup

The method was evaluated using the NICO (Non-I.I.D. Image Classification with Objects) dataset, which is specifically designed for OOD tasks. NICO contains 19 classes across various contexts (e.g., "dog on grass" vs. "dog in water"). The authors employed a "leave-one-domain-out" protocol, training on a subset of contexts and testing on entirely unseen ones to measure true generalization performance.

Results and Discussion

The Hypercolumn-adapted model consistently outperformed the baseline ResNet-18 across varying levels of difficulty. In tests where 7 contexts were left out, the proposed model achieved 83.0% accuracy, a 4.1% improvement over the baseline.

Comparison of saliency maps between baseline ResNet-18 and the proposed method across various NICO classes.

Qualitative analysis using saliency maps confirms that the model focuses more tightly on the actual object (e.g., the cat or car) while the baseline is often distracted by background elements like snow, water ripples, or grass. This suggests that the inclusion of early-layer features provides a "geometric anchor" that prevents the model from over-relying on contextual cues.

Conclusion

This work demonstrates that multi-layer representation learning via Hypercolumns significantly improves OOD robustness without requiring complex data augmentation or adversarial training. Future research may explore scaling this approach to larger architectures like Vision Transformers or incorporating attention mechanisms to dynamically weight the importance of different intermediate layers.

Summary updated

Abstract

Convolutional Neural Networks have become the norm in image classification. Nevertheless, their difficulty to maintain high accuracy across datasets has become apparent in the past few years. In order to utilize such models in real-world scenarios and applications, they must be able to provide trustworthy predictions on unseen data. In this paper, we argue that extracting features from a CNN's intermediate layers can assist in the model's final prediction. Specifically, we adapt the Hypercolumns method to a ResNet-18 and find a significant increase in the model's accuracy, when evaluating on the NICO dataset.
BibTeX
@inproceedings{ballas2022multi,
  author = {Aristotelis Ballas and Christos Diou},
  title = {{Multi-layer Representation Learning for Robust OOD Image Classification}},
  booktitle = {Setn},
  year = {2022},
  doi = {10.1145/3549737.3549780},
}
Figure