Beyond Categorical Label Representations for Image Classification

Boyuan Chen, Yu Li, Sunand Raghupathi, Hod Lipson

Abstract

Standard image classification relies on "one-hot" categorical labels, which often leads to data inefficiency and vulnerability to adversarial attacks. This research explores replacing these simple labels with high-dimensional, high-entropy representations, such as audio spectrograms of class names. By training models to regress these complex matrices rather than simple probabilities, the authors demonstrate significant improvements in model robustness and data efficiency without compromising accuracy.

Introduction

The conventional approach to training image classifiers involves one-hot encoding, where a class is represented as a sparse vector. While effective for standard benchmarks, this method is fundamentally data-hungry and produces models that are easily fooled by adversarial perturbations—tiny, imperceptible changes to pixels that lead to confident misclassifications.

Researchers from Columbia University suggest that this categorical approach lacks the rich, multi-modal signals present in human learning. This study investigates whether providing neural networks with high-dimensional, high-entropy label representations can provide a stronger error signal, leading to more discriminative features and improved generalization.

Methods

The researchers replaced the standard classification head and cross-entropy loss with a generative "label decoder" ($I_{sd}$) and Smooth L1 (Huber) loss. Instead of predicting a single category index, the model is tasked with regressing a high-dimensional matrix.

The proposed training pipeline maps images through an encoder-decoder architecture to predict high-dimensional label matrices, such as speech spectrograms or shuffled noise.

The primary novel representation used was a Log Mel Spectrogram of a human voice saying the class name. To isolate the factors contributing to performance, the team compared speech spectrograms against several other label types, including shuffled spectrograms (high entropy, no structure), Gaussian compositions (random blobs), and constant matrices (low entropy). At inference time, the model classifies an image by identifying which ground-truth label matrix is closest to its prediction via a nearest-neighbor search.

Key Insights: The Entropy Hypothesis

A central finding of the study is the Entropy Hypothesis: high-dimensional labels only improve performance if they also possess high entropy. A $64 \times 64$ matrix of constant values performed worse than standard one-hot labels. High-entropy labels, such as speech spectrograms or even random noise, provide a "denser" error signal during backpropagation. This essentially provides the model with more information on how to correct its weights during every training step.

Furthermore, visualizations using Grad-CAM revealed that models trained with speech labels focused more precisely on the central objects within images, whereas categorical models often exhibited more scattered attention. t-SNE analysis also showed that these models formed distinct class clusters much earlier in the training process.

Results and Robustness

The proposed method was evaluated on CIFAR-10 and CIFAR-100 using VGG and ResNet architectures. The results highlighted three major advantages:

  1. Data Efficiency: In low-data regimes (using only 1% or 2% of the training set), speech-label models outperformed standard categorical models by a wide margin, demonstrating superior feature generalization.
  2. Adversarial Defense: Models trained on high-entropy labels were significantly more resistant to adversarial attacks like FGSM. Under strong attacks ($\epsilon = 0.3$), standard models often dropped to near-zero accuracy, while speech-label models maintained 20-40% higher accuracy without any specific adversarial training.
  3. Comparable Accuracy: Despite the increased complexity of the task, these models achieved top-1 accuracy comparable to standard one-hot baselines on full datasets.

Conclusion

By moving beyond one-hot labels to high-dimensional, high-entropy representations, researchers can develop image classifiers that are both more robust and more data-efficient. While predicting complex matrices introduces a slight computational overhead, the "free" robustness and faster convergence suggest that the simplicity of current categorical labels may be a bottleneck in neural network training. Future work will explore scaling these findings to larger datasets like ImageNet and optimizing synthetic label structures.

Summary updated

Abstract

We find that the way we choose to represent data labels can have a profound effect on the quality of trained models. For example, training an image classifier to regress audio labels rather than traditional categorical probabilities produces a more reliable classification. This result is surprising, considering that audio labels are more complex than simpler numerical probabilities or text. We hypothesize that high dimensional, high entropy label representations are generally more useful because they provide a stronger error signal. We support this hypothesis with evidence from various label representations including constant matrices, spectrograms, shuffled spectrograms, Gaussian mixtures, and uniform random matrices of various dimensionalities. Our experiments reveal that high dimensional, high entropy labels achieve comparable accuracy to text (categorical) labels on the standard image classification task, but features learned through our label representations exhibit more robustness under various adversarial attacks and better effectiveness with a limited amount of training data. These results suggest that label representation may play a more important role than previously thought. The project website is at \url{https://www.creativemachineslab.com/label-representation.html}.
BibTeX
@inproceedings{chen2021beyond,
  author = {Boyuan Chen and Yu Li and Sunand Raghupathi and Hod Lipson},
  title = {{Beyond Categorical Label Representations for Image Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=inRji9D3A1i},
}
Figure