Abstract
The maximum entropy encoding framework provides a unified perspective for many non-contrastive learning methods like SimSiam, Barlow Twins, and MEC. Inspired by this framework, we introduce Matrix-SSL, a novel approach that leverages matrix information theory to interpret the maximum entropy encoding loss as matrix uniformity loss. Furthermore, Matrix-SSL enhances the maximum entropy encoding method by seamlessly incorporating matrix alignment loss, directly aligning covariance matrices in different branches. Experimental results reveal that Matrix-SSL outperforms state-of-the-art methods on the ImageNet dataset under linear evaluation settings and on MS-COCO for transfer learning tasks. Specifically, when performing transfer learning tasks on MS-COCO, our method outperforms previous SOTA methods such as MoCo v2 and BYOL up to 3.3% with only 400 epochs compared to 800 epochs pre-training. We also try to introduce representation learning into the language modeling regime by fine-tuning a 7B model using matrix cross-entropy loss, with a margin of 3.1% on the GSM8K dataset over the standard cross-entropy loss.
Introduction
Self-supervised learning (SSL) has traditionally been split into two camps: contrastive methods, which use negative samples to ensure "uniformity" in the feature space, and non-contrastive methods, which avoid negatives but often struggle with theoretical unification. While recent theories like Maximum Entropy Coding (MEC) attempted to provide a unified view, they often failed to explicitly differentiate between feature matrices from different augmentation branches. This makes it difficult to integrate "alignment" (ensuring different views of the same image look similar) with "uniformity" (ensuring features don't collapse into a single dimension). Matrix-SSL seeks a single, encompassing framework that harmonizes these two virtues.
Methods
The core of Matrix-SSL is the transition from scalar-based Shannon information theory to Matrix Information Theory. The authors propose a dual-component loss function:
- Matrix-Uniformity Loss: Uses Matrix Cross-Entropy (MCE) to align the cross-covariance of feature branches with an identity matrix, preventing dimensional collapse.
- Matrix-Alignment Loss: Directly aligns the auto-covariance matrices of different branches. This provides a richer signal than standard vector-to-vector alignment because it considers the structural relationship of the entire feature batch.
The authors prove that the existing MEC loss is mathematically equivalent to their matrix uniformity loss. Furthermore, they establish a closed-form relationship between "effective rank" and Matrix KL divergence, providing a rigorous way to monitor and prevent "dimensional collapse."
Key Results
The authors evaluated Matrix-SSL across several domains, including computer vision (ImageNet), transfer learning (MS-COCO), and mathematical reasoning (GSM8K).
- Efficiency: On ImageNet, Matrix-SSL with only 100 epochs of pre-training outperformed SimCLR by 4.6%.
- Transfer Superiority: On MS-COCO transfer tasks, Matrix-SSL outperformed previous SOTA methods like MoCo v2 and BYOL by up to 3.3%, despite being trained for only half the time (400 vs 800 epochs).
- Reasoning Boost: Fine-tuning a 7B parameter language model with Matrix-LLM loss yielded a 3.1% margin on the GSM8K benchmark over standard cross-entropy. The model learns that predicting a synonym is "less wrong" than predicting an unrelated word, leading to better reasoning performance.
Conclusion
Matrix-SSL provides a unified framework for SSL grounded in matrix information theory. By extending classical concepts like entropy and KL divergence to feature covariance matrices, the authors bridge the gap between contrastive and non-contrastive learning. The method consistently prevents dimensional collapse and achieves state-of-the-art results in both computer vision and large language model reasoning tasks. Future work includes exploring "auto-covariance uniformity loss" to further refine the framework.
Summary updated