Learning Contrastive Embedding in Low-Dimensional Space

Shuo Chen, Chen Gong, Jun Li, Jian Yang, Gang Niu, Masashi SugiyamaNeurIPS 2022 · OpenReview

Contrastive Learning (CL) has become the gold standard for unsupervised representation learning. By teaching models to "push apart" different images and "pull together" augmented versions of the same image, researchers have achieved results that rival fully supervised training. However, there is a hidden assumption in most CL recipes: bigger is better. Standard practice involves projecting these features into massive 2048- or 4096-dimensional spaces to maximize "instance discrimination."

A paper from NeurIPS 2022, "Learning Contrastive Embedding in Low-Dimensional Space," argues that this high-dimensional habit might actually be hindering model performance.

TL;DR

Standard contrastive learning often fails to capture intrinsic similarities because high-dimensional spaces become "too sparse," making every data point look equally far from every other point. To solve this, the authors propose CLLR (Contrastive Learning with Low-dimensional Reconstruction), a framework that uses sparse and low-rank regularizers to force models into meaningful low-dimensional embeddings. This approach significantly improves performance across vision, language, and robotics tasks.

What Problem Does This Paper Address?

The core goal of CL is instance discrimination: making every data point unique in the feature space. To do this effectively, researchers typically use very high-dimensional embedding spaces.

However, the authors point out a geometric trap known as the curse of dimensionality. In a high-dimensional hypercube, almost all the volume is concentrated in the "corners." As the number of dimensions increases, data points scattered by CL tend to migrate to these corners. Mathematically, the ratio between the minimum and maximum distance between any two points approaches 1.

Distance distributions across different dimensionalities

The result? Everything becomes far away from everything else. The model loses the ability to capture "intrinsic similarity"—the subtle relationships that tell us a golden retriever is more like a beagle than a toaster. If you simply try to fix this by shrinking the output layer, you hit "dimensional collapse," where the model stops being able to discriminate between instances at all.

Methods

To combat this, the authors introduce CLLR, which adds a clever "reconstruction" requirement to the standard contrastive objective. Instead of just maximizing discrimination in a high-dimensional space, they introduce a projection matrix that transforms the high-dimensional embedding into a low-dimensional latent space.

The CLLR Framework

The model is trained with a dual objective:

  1. Standard Contrastive Loss (NCE): Keep instances discriminable.
  2. Reconstruction Loss: Ensure the original high-dimensional feature can be reconstructed from the low-dimensional projection.

To make this low-dimensional space truly effective, they apply two types of regularization to the projection matrix:

  • Sparsity ($\ell_{2,1}$-norm): This forces the model to select only the most important "neurons" or features, effectively performing feature selection.
  • Low-rank (Nuclear-norm): This encourages linear dependence between columns, ensuring the underlying information lives on a low-dimensional manifold.

By combining these, the model is forced to pack all the discriminative information into a compact, low-dimensional structure without losing the ability to tell instances apart.

Key Insights and Results

The authors validated CLLR across three distinct domains: Computer Vision (STL-10, CIFAR-10, ImageNet-100), Natural Language Processing (BookCorpus), and Reinforcement Learning (DeepMind Control Suite).

Major Takeaways: * The "Corner" Problem is Real: The paper provides a rigorous geometric proof showing that in high dimensions, random data points become nearly equidistant. This explains why standard CL models often struggle with fine-grained similarity. * Reconstruction as a Regularizer: Unlike a standard Autoencoder which reconstructs raw input, CLLR reconstructs the embedding. This acts as a bottleneck that filters out high-dimensional noise while preserving semantic structure. * Plug-and-Play Genericity: CLLR isn't a standalone model; it’s a framework. The authors showed it could be added to SimCLR, DCL, HCL, CMC, and CURL, improving all of them. * Performance Gains: On ImageNet-100, adding CLLR to the CMC baseline improved Top-1 accuracy from 73.58% to 76.91%. In language tasks, it consistently outperformed baselines by 1–2% across text classification benchmarks. In RL, CLLR-enhanced models achieved higher scores with lower variance.

Limitations and Future Work

While CLLR is highly effective, the authors note a few areas for future exploration:

  • Negative-Free CL: The current study focuses on mainstream CL models that use both positive and negative pairs (like SimCLR). It remains to be seen how this reconstruction approach fares with "negative-free" methods like BYOL or SimSiam.
  • Computational Overhead: While the additional projection layer is small, the nuclear-norm regularization involves singular value decomposition (SVD) or similar operations, which can be computationally expensive for extremely large feature sizes.
  • Optimal Dimension: The paper demonstrates that low-D is better, but finding the exact optimal number of dimensions for a specific dataset still requires empirical tuning.

Summary updated

Abstract

Contrastive learning (CL) pretrains feature embeddings to scatter instances in the feature space so that the training data can be well discriminated. Most existing CL techniques usually encourage learning such feature embeddings in the highdimensional space to maximize the instance discrimination. However, this practice may lead to undesired results where the scattering instances are sparsely distributed in the high-dimensional feature space, making it difficult to capture the underlying similarity between pairwise instances. To this end, we propose a novel framework called contrastive learning with low-dimensional reconstruction (CLLR), which adopts a regularized projection layer to reduce the dimensionality of the feature embedding. In CLLR, we build the sparse / low-rank regularizer to adaptively reconstruct a low-dimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning contrastive embeddings that alleviate the above issue. Theoretically, we prove a tighter error bound for CLLR; empirically, the superiority of CLLR is demonstrated across multiple domains. Both theoretical and experimental results emphasize the significance of learning low-dimensional contrastive embeddings.
BibTeX
@inproceedings{chen2022learning,
  author = {Shuo Chen and Chen Gong and Jun Li and Jian Yang and Gang Niu and Masashi Sugiyama},
  title = {{Learning Contrastive Embedding in Low-Dimensional Space}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=Ce7MGxQJnm},
}
Figure