Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains

Hafiz Tiomoko Ali, Umberto Michieli, Ji Joong Moon, Daehyun Kim, Mete Ozay

Abstract

Standard deep learning paradigms involve the simultaneous optimization of a feature-extracting backbone and a linear classification head. While effective for in-domain tasks, this co-evolution often leads to the capture of domain-specific noise, hindering the model's ability to generalize to novel domains. This paper explores the impact of fixing the final classification layer as a static, random Equiangular Tight Frame (ETF). By forcing the backbone to map features into a rigid, pre-defined geometric structure, the model achieves a state of "Neural Collapse" more efficiently. Experimental results demonstrate that models trained with a fixed ETF classifier learn more robust, separable features that significantly outperform standard trainable models in out-of-domain transfer learning tasks.

Introduction

In the pursuit of universal feature extractors, researchers have increasingly focused on the phenomenon of Neural Collapse (NC). Neural Collapse describes a state at the end of training where within-class feature variability vanishes, and class means align with the vertices of an Equiangular Tight Frame (ETF)—a geometric structure where all vectors are spread as far apart as possible.

Typically, NC emerges naturally during the terminal phase of training. However, the authors of this study argue that allowing the classifier to evolve alongside the backbone creates a "moving target" problem. This flexibility allows the network to exploit domain-specific correlations that assist in classifying the source data but fail to translate to new contexts. By fixing the classifier from the outset, the authors propose a method to enforce feature purity and improve cross-domain transferability.

Methodology

The Fixed ETF Classifier

The core proposal is to replace the standard trainable linear head with a fixed, random matrix that satisfies the properties of an ETF. In this setup, the "slots" for each class are determined at initialization and remain unchanged throughout the training process.

This approach introduces a form of Implicit Regularization. Because the classifier cannot adapt to the backbone, the backbone is forced to do the heavy lifting. It must learn to eliminate within-class covariance to fit features into the narrow, pre-defined ETF slots. This pressure forces the backbone to discard non-essential features that do not contribute to the rigid geometric separation required by the fixed head.

Theoretical Foundation

Drawing on Random Matrix Theory (RMT), the authors demonstrate that a linear random projector is the optimal configuration for achieving feature collapse with minimal class covariance. They compare this "Fixed Model" against two primary baselines:

  1. Trainable Model: The standard approach where both the backbone and the head are optimized.
  2. Switchable Whitening (SW): A sophisticated method designed to explicitly normalize feature covariance during the training process.

Experimental Setup

Models (specifically ResNet-50 and ResNet-101) were pre-trained on ImageNet to serve as the source domain. To evaluate the robustness of the learned features, the models were then tested on a variety of target datasets categorized by their similarity to the source:

  • In-Domain: CIFAR-10, CIFAR-100, and STL-10.
  • Out-of-Domain: SVHN (Street View House Numbers), Oxford Pets, Oxford Flowers, and the Describable Textures Dataset (DTD).

Key Insights

Overcoming the "Moving Target"

In standard training, the classifier and backbone can "collude" to minimize loss using shortcuts—features that are predictive in the training set but irrelevant in the real world. A fixed classifier acts as an uncompromising anchor, preventing the backbone from relying on these shifting correlations.

Feature Purity and Covariance

Analysis of feature covariance heatmaps reveals a stark difference between the models. The Fixed Model exhibits a feature covariance of approximately 0.01, whereas the standard Trainable Model maintains a much higher covariance of 0.16. This lower covariance indicates that the Fixed Model produces features that are much more tightly clustered around their class means, leading to "cleaner" and more discriminative representations.

Results

While the Fixed Model performed comparably to standard models on in-domain tasks, it showed a "superpower" for difficult, cross-domain transfers. The performance gains in out-of-domain scenarios were substantial:

  • Oxford Flowers: The Fixed Model achieved 60.30% accuracy, compared to only 38.05% for the Trainable Model.
  • DTD (Textures): Accuracy rose from 29.09% to 40.90%.
  • Oxford Pets: Accuracy improved from 40.50% to 48.32%.

These results suggest that the features learned under the constraint of a fixed ETF classifier are fundamentally more robust to domain shift.

Conclusion and Future Work

Fixing the final classifier as an ETF is a deceptively simple yet powerful modification to the standard training pipeline. By enforcing Neural Collapse from the start of training, the model is forced to learn high-quality, transferable features that excel in out-of-domain tasks.

Future research may explore how this strategy scales to larger architectures, such as Vision Transformers (ViTs), and how to handle scenarios where the target domain has a significantly different number of classes than the source. Ultimately, this method provides a practical path toward building more robust foundation models that require less fine-tuning for niche downstream applications.

Summary updated

Abstract

The recently discovered Neural collapse (NC) phenomenon states that the last-layer weights of Deep Neural Networks (DNN), converge to the so-called Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training. This ETF geometry is equivalent to vanishing within-class variability of the last layer activations. Inspired by NC properties, we explore in this paper the transferability of DNN models trained with their last layer weight fixed according to ETF. This enforces class separation by eliminating class covariance information, effectively providing implicit regularization. We show that DNN models trained with such a fixed classifier significantly improve transfer performance, particularly on out-of-domain datasets. On a broad range of fine-grained image classification datasets, our approach outperforms i) baseline methods that do not perform any covariance regularization (up to 22%), as well as ii) methods that explicitly whiten covariance of activations throughout training (up to 19%). Our findings suggest that DNNs trained with fixed ETF classifiers offer a powerful mechanism for improving transfer learning across domains.
BibTeX
@inproceedings{ali2024deep,
  author = {Hafiz Tiomoko Ali and Umberto Michieli and Ji Joong Moon and Daehyun Kim and Mete Ozay},
  title = {{Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447986},
}
Figure