Unsupervised Representation Learning via Neural Activation Coding

Yookoon Park, Sangho Lee, Gunhee Kim, David M. Blei

Introduction

In unsupervised representation learning, the goal is to train a deep encoder on unlabeled data so that its output features are useful for downstream tasks, such as training a simple linear classifier. Recently, contrastive learning methods like SimCLR and MoCo have dominated this space. However, these methods rely heavily on specific architectural tricks—most notably $L_2$ normalization—and are generally driven by the Information Maximization (InfoMax) principle, which does not fully explain their empirical success.

This paper introduces Neural Activation Coding (NAC), a novel unsupervised representation learning method that explicitly maximizes a deep encoder's nonlinear expressivity. For a network using piecewise linear activations like ReLU, expressivity is defined by the number of distinct linear regions it divides the input space into. The authors argue that the more regions a network creates, the better a downstream linear model can untangle complex data. By training the network to transmit its ReLU activation patterns through a noisy communication channel, NAC forces the encoder to learn maximally separable representations. This approach yields highly competitive performance on linear classification, state-of-the-art results in deep hashing, and significantly improved training dynamics for Variational Autoencoders (VAEs).

Methods

NAC treats the activation pattern of a network's hidden units as a binary code (e.g., +1 if a ReLU neuron is active, -1 if it is not). To maximize the number of distinct linear regions, NAC formulates a communication problem:

  1. The Sender (Encoder): Maps an augmented input image to a binary activation codeword.
  2. The Noisy Channel: The codeword is passed through a simulated symmetric noise channel where each bit has a probability $p$ of flipping.
  3. The Receiver (Inference Network): Attempts to reconstruct the original message from the noisy codeword.
NAC Architecture Diagram

To successfully communicate through this noise, the encoder is forced to learn codewords that are maximally distant from one another (i.e., having a high Hamming distance). This naturally maximizes the number of distinct linear regions the network uses. To make this process differentiable, the authors use a continuous relaxation ($\tanh$ instead of a hard sign function) during training, alongside variational inference and subsampling techniques similar to those used in contrastive learning.

Notably, unlike SimCLR, NAC does not use $L_2$ normalization. Instead, it allows the norm of the feature vector to dynamically control the sharpness of the distribution, effectively acting as a measure of the model's confidence.

Key Insights

  • Expressivity via Noise: Forcing a neural network to communicate its internal state through a noisy channel is an elegant way to force feature separation. If the codes are too similar, the noise will make them indistinguishable.
  • Feature Norm as Uncertainty: By dropping the standard practice of $L_2$ normalizing features onto a unit hypersphere, NAC allows the network to express uncertainty. When the encoder is confident, it outputs a vector with a large norm (creating a sharp distribution); when uncertain, it outputs a small norm (smoothing the distribution).
  • Dual-Purpose Representations: NAC is uniquely versatile. Because it explicitly models binary activation patterns, it simultaneously learns high-quality continuous representations (useful for linear classification) and discrete representations (useful for fast, memory-efficient hash-based image retrieval).
  • Solving the VAE "Cold Start": Variational Autoencoders often suffer from suboptimal encoders during early training, leading to ignored latent variables. Pretraining the encoder with NAC maximizes its expressivity from step one, drastically improving VAE training dynamics.

Key Results

  • Linear Classification: On CIFAR-10, NAC achieves 93.9% top-1 accuracy, outperforming strong contrastive baselines like SimCLR (92.8%) and MoCo-v2 (91.6%).
CIFAR-10 Linear Classification Results

On ImageNet-1K, it reaches a respectable 65.0% (slightly behind SimCLR's 66.6%, though the authors note a lack of compute for extensive hyperparameter tuning).

  • Deep Hashing (Retrieval): NAC significantly outperforms existing deep hashing baselines. On CIFAR-10, it achieves 40.5% mAP, compared to DistillHash (29.0%) and discretized SimCLR (34.2%). On FLICKR-25K, it achieves 70.8% mAP.
  • Generative Pretraining: When used to pretrain a VAE on CIFAR-10, NAC achieved a log-likelihood of -2865 and a KL divergence of 71.8. This vastly outperforms a randomly initialized VAE (LL: -3202, KL: 33.0) and even a SimCLR-pretrained VAE (LL: -3174, KL: 38.9), proving it utilizes the latent space much more effectively.

Limitations and Future Work

While highly effective, NAC's performance on ImageNet-1K slightly lags behind the highly optimized state-of-the-art contrastive methods. The authors attribute this to computational constraints that prevented large-scale hyperparameter tuning. Additionally, the theoretical motivation relies heavily on the piecewise linear nature of ReLU networks, though the authors suggest the principles could apply to smooth approximations like GELU or Swish. Future work will explore the application of Neural Activation Coding to other data domains outside of computer vision, such as natural language processing.

Summary updated

Abstract

We present neural activation coding (NAC) as a novel approach for learning deep representations from unlabeled data for downstream applications. We argue that the deep encoder should maximize its nonlinear expressivity on the data for downstream predictors to take full advantage of its representation power. To this end, NAC maximizes the mutual information between activation patterns of the encoder and the data over a noisy communication channel. We show that learning for a noise-robust activation code increases the number of distinct linear regions of ReLU encoders, hence the maximum nonlinear expressivity. More interestingly, NAC learns both continuous and discrete representations of data, which we respectively evaluate on two downstream tasks: (i) linear classification on CIFAR-10 and ImageNet-1K and (ii) nearest neighbor retrieval on CIFAR-10 and FLICKR-25K. Empirical results show that NAC attains better or comparable performance on both tasks over recent baselines including SimCLR and DistillHash. In addition, NAC pretraining provides significant benefits to the training of deep generative models. Our code is available at https://github.com/yookoon/nac.
BibTeX
@inproceedings{park2021unsupervised,
  author = {Yookoon Park and Sangho Lee and Gunhee Kim and David M. Blei},
  title = {{Unsupervised Representation Learning via Neural Activation Coding}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure