SMEC:Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression

Biao Zhang, Lixin Chen, Tong Liu, Bo Zheng

Abstract

Large language models (LLMs) generate high-dimensional embeddings that capture rich semantic and syntactic information. However, high-dimensional embeddings exacerbate computational complexity and storage requirements, thereby hindering practical deployment. To address these challenges, we propose a novel training framework named Sequential Matryoshka Embedding Compression (SMEC). This framework introduces the Sequential Matryoshka Representation Learning (SMRL) method to mitigate gradient variance during training, the Adaptive Dimension Selection (ADS) module to reduce information degradation during dimension pruning, and the Selectable Cross-batch Memory (S-XBM) module to enhance unsupervised learning between high- and low-dimensional embeddings. Experiments on image, text, and multimodal datasets demonstrate that SMEC achieves significant dimensionality reduction while maintaining performance. For instance, on the BEIR dataset, our approach improves the performance of compressed LLM2Vec embeddings (256 dimensions) by 1.1 points and 2.7 points compared to the Matryoshka-Adaptor and Search-Adaptor models, respectively.

Introduction

Modern Large Language Models (LLMs) produce incredibly rich embeddings, but they come with a significant "dimensionality tax." Vectors with thousands of dimensions—such as the 3,584 dimensions in LLM2Vec-7B—are expensive to store and slow to search in production Information Retrieval (IR) systems. While Matryoshka Representation Learning (MRL) was proposed to allow a single high-dimensional vector to contain smaller, functional sub-vectors, it suffers from several critical flaws.

The authors identify three major limitations in standard MRL:

  1. Gradient Fluctuation: Training all dimensions simultaneously creates a "tug-of-war" where gradients from lower dimensions overwhelm the rest, leading to unstable convergence.
  2. Information Degradation: Standard MRL relies on "static truncation" (taking the first $k$ dimensions), which often discards critical information stored in the "tail" of the vector.
  3. Sample Limitations: Unsupervised learning between high- and low-dimensional versions of the same vector is often constrained by small batch sizes, limiting the model's ability to learn robust representations.

The SMEC Framework

To overcome these hurdles, the SMEC framework introduces a multi-pronged approach to embedding compression.

Sequential Matryoshka Representation Learning (SMRL)

Unlike traditional MRL, which trains all dimensions in parallel, SMRL adopts a sequential training strategy. It trains dimensions in stages (e.g., $D \to D/2 \to D/4$), which stabilizes gradients and prevents the noise from lower dimensions from hindering the convergence of larger dimensions. This approach also enables "continued training," allowing researchers to compress existing models further without starting from scratch.

Comparison of adaptation strategies: (a) Search-Adaptor, (b) Matryoshka-Adaptor, and (c) the proposed SMEC framework.

Adaptive Dimension Selection (ADS)

One of the key insights of this work is that the most important features are not always located at the beginning of a vector. The ADS module uses a learnable component powered by Gumbel-Softmax to identify which specific dimensions are most critical for maintaining performance. This "smart pruning" ensures that SMEC retains over 80% of critical information even at high compression ratios, whereas standard static truncation often drops below 20%.

Selectable Cross-batch Memory (S-XBM)

To sharpen the model's discriminative power in low-dimensional spaces, S-XBM maintains a queue of historical embeddings across batches. It specifically mines "hard samples"—items that are highly similar but distinct—to improve the model's ability to distinguish between complex inputs even when the embedding space is significantly reduced.

Experimental Results

The authors evaluated SMEC across text (BEIR), image (Products-10K), and multimodal (Fashion-200K) domains to prove its versatility.

Massive Lossless Compression

On the BEIR Quora dataset, SMEC achieved a remarkable 14x lossless compression for LLM2Vec-7B and 12x for LLM2Vec-1B. As shown in the performance plots, the SMEC-enhanced models at lower dimensions frequently match or exceed the performance of the full-dimensional base models.

Performance comparison of SMEC against baselines on the BEIR dataset, highlighting significant lossless dimension compression.

Superior Accuracy and Robustness

On the overall BEIR benchmark, SMEC improved the performance of 256-dimension embeddings by 1.1 points over Matryoshka-Adaptor and 2.7 points over Search-Adaptor. Furthermore, SMEC consistently outperformed PCA and other adaptive methods in image-to-text and text-to-image retrieval tasks, demonstrating its robustness across different modalities.

Conclusion and Future Work

SMEC demonstrates that it is possible to achieve significant efficiency gains in embedding storage and retrieval without sacrificing accuracy. By rethinking the training dynamics of Matryoshka learning and introducing adaptive selection, the framework provides a practical path for deploying massive LLM embeddings in resource-constrained environments.

Future work may address current limitations, such as the reliance on domain-specific labeled data for peak performance. The authors also suggest exploring "full-parameter" training to integrate SMEC principles directly into the base training of future LLMs, potentially eliminating the need for post-hoc adapters.

Summary updated

Abstract

Large language models (LLMs) generate highdimensional embeddings that capture rich semantic and syntactic information.However, high-dimensional embeddings exacerbate computational complexity and storage requirements, thereby hindering practical deployment.To address these challenges, we propose a novel training framework named Sequential Matryoshka Embedding Compression (SMEC).This framework introduces the Sequential Matryoshka Representation Learning(SMRL) method to mitigate gradient variance during training, the Adaptive Dimension Selection (ADS) module to reduce information degradation during dimension pruning, and the Selectable Cross-batch Memory (S-XBM) module to enhance unsupervised learning between high-and low-dimensional embeddings.Experiments on image, text, and multimodal datasets demonstrate that SMEC achieves significant dimensionality reduction while maintaining performance.For instance, on the BEIR dataset, our approach improves the performance of compressed LLM2Vec embeddings (256 dimensions) by 1.1 points and 2.7 points compared to the Matryoshka-Adaptor and Search-Adaptor models, respectively.
BibTeX
@misc{zhang2025smec,
  author = {Biao Zhang and Lixin Chen and Tong Liu and Bo Zheng},
  title = {{SMEC:Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression}},
  year = {2025},
  eprint = {2510.12474},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12474},
  howpublished = {arXiv},
}
Figure