Abstract
In this paper, based on an asymptotic analysis of the Softmax layer, we show that when training neural networks for classification tasks, the weight vectors corresponding to each class of the Softmax layer tend to converge to the class-wise means computed at the representation layer (for specific choices of the representation activation). We further show some consequences of our findings to the context of transfer learning, essentially by proposing a simple yet effective initialization procedure that significantly accelerates the learning of the Softmax layer weights as the target domain gets closer to the source one. Experiments are notably performed on the datasets: MNIST, Fashion MNIST, Cifar10, and Cifar100 and using a standard CNN architecture.
Introduction
In the contemporary landscape of AutoML and Transfer Learning (TL), the standard practice involves repurposing pre-trained models for novel tasks. However, selecting the optimal source model for a specific target domain typically necessitates a computationally intensive process: training the final classification layer for every candidate model to evaluate performance. This paper addresses this bottleneck by investigating the theoretical relationship between learned data representations and the final Softmax classification layer. We demonstrate that Softmax weights tend to converge to the class-wise means of the representations, enabling "training-free" model evaluation and accelerated fine-tuning.
Methodology
The core of our analysis relies on an asymptotic study of Softmax classifier weights under specific statistical assumptions. We model data representations at the penultimate layer as a Gaussian Mixture Model (GMM), a choice supported by empirical evidence that deep features in high dimensions tend toward Gaussian distributions.
A critical concept in our work is the near-optimal representation. We define this as a state where the means of different classes are nearly orthogonal (well-separated) and the within-class noise is isotropic. Under these conditions, we derive a mathematical expression showing that the Softmax weight vector for a given class is asymptotically proportional to the centered mean of that class's representations.
The Mean-Alignment Phenomenon
Our most significant finding is that a Softmax layer essentially "learns" to function as a Nearest Class Mean (NCM) classifier. The weights align themselves with the average representation of each class. This alignment is particularly pronounced when using symmetric activation functions like tanh at the representation layer. In such cases, almost all class-separability information is encoded within the means, making the weights highly interpretable.
In contrast, for ReLU networks, the information is split between means and covariances, which slightly degrades the "cleanliness" of the weight-mean alignment, though the principle still holds.
Practical Applications in Transfer Learning
Leveraging these theoretical insights, we propose two strategies:
- Weights Initialized with Centred Means (WICM): Instead of random initialization, the Softmax layer is initialized using the class-wise means of the target data. This provides a "head start" that significantly accelerates convergence.
- Centred Means (CM): A training-free method for model selection where Softmax weights are replaced by class means to estimate potential accuracy without gradient updates.
Empirical Results
Experiments on MNIST, Fashion MNIST, Cifar10, and Cifar100 validate our findings. On MNIST, the "CM" method achieved a test accuracy of 99.6%, matching the fully trained model. Furthermore, cosine similarity between learned weights and class-wise means approached 1.0 during training with tanh activations. WICM initialization proved particularly effective as the target domain became more similar to the source, drastically reducing training time.
Conclusion and Limitations
While our results provide a rigorous theoretical backbone for NCM-like behavior in deep networks, they rely on the GMM assumption for features. While empirically sound for deep architectures, this may not hold for shallow networks. Additionally, the strongest alignment requires symmetric activations. Future work will explore the scalability of these findings to massive datasets like ImageNet-21k and non-visual domains.
Summary updated