Balancing hierarchical representation and quantization: A performance trade-off in text embeddings

Kai Ponel, Marco Polignano, Erasmo Purificato, Aylin Julia Temizkan, Marco Levantesi, Giovanni Semeraro, Ernesto William De LucaarXiv 2026 · DOI · arXiv

TL;DR

This paper presents a comprehensive empirical benchmark on the interaction between hierarchical (Matryoshka) embedding truncation and quantization. The authors find that combining 8-bit quantization with an 8x reduction in dimensionality can achieve a 32x reduction in storage while retaining 94.4% of baseline effectiveness, though they identify retrieval tasks as being uniquely sensitive to these aggressive optimizations.

What Problem Does This Paper Address?

Text embeddings are the backbone of modern AI applications like Retrieval-Augmented Generation (RAG), but they come with significant costs. A million-chunk index using standard 1024-dimensional float32 vectors requires about 4 GB of memory. As indices scale to billions of vectors, storage and search latency become major bottlenecks.

Two popular solutions exist: Matryoshka Representation Learning (MRL), which allows embeddings to be truncated to smaller sizes (like nesting dolls), and Quantization, which reduces the bit-precision of each number (e.g., from 32-bit floats to 8-bit integers). While both techniques are well-established, practitioners often use them together based on the untested assumption that they don't interfere with each other. This paper provides the first large-scale "factorial" study to see how these two orthogonal compression methods actually interact across different types of NLP tasks.

Methods

The researchers conducted a massive evaluation involving 168 different configurations per task. Their approach included:

  • Factorial Design: They tested every combination of hierarchical size ($m$), quantization technique ($q$), and—for 2D models—the inference layer ($l$).
  • Quantization Levels: They compared standard float32 against int8 (scalar quantization) and binary (1-bit) quantization.
  • Hierarchical Truncation: They tested successive halvings of the embedding dimension (e.g., from 1024 down to 8).
  • 2D Matryoshka: For specific models, they tested "layer reduction," where embeddings are extracted from earlier transformer layers (e.g., layer 12 instead of 24) to save on computation time.
  • Statistical Rigor: Instead of just looking at averages, they used Friedman and Nemenyi tests to determine if performance drops were statistically significant or just noise.
Table 1: A worked example showing how truncation and quantization affect storage and similarity scores for a RAG pipeline.

Datasets

The study utilized the Massive Text Embedding Benchmark (MTEB), a standard in the field. They evaluated 38 tasks across seven categories:

  1. Classification
  2. Clustering
  3. Pair Classification
  4. Re-ranking
  5. Retrieval
  6. Semantic Textual Similarity (STS)
  7. Summarization

To keep the compute manageable (each task was repeated 168 times), they excluded the largest retrieval benchmarks, noting that their results for retrieval should be considered a "lower bound" on degradation.

Key Insights and Ideas

  • The "One-Halving" Rule: Statistically, halving the embedding dimension once (e.g., 1024 to 512) is often "free"—the performance loss is not statistically detectable. However, two successive halvings (e.g., 1024 to 256) almost always result in a significant drop.
  • Retrieval is the Weak Link: Retrieval tasks are far more fragile than classification or clustering. The authors explain that ranking requires fine-grained similarity nuances that live in the "trailing dimensions" of an embedding. Truncating these or binarizing them destroys the subtle margins needed to distinguish between two similar documents.
  • Storage vs. Speed: The study clarifies a common misconception: while quantization and truncation drastically reduce storage, they don't necessarily speed up evaluation time in standard software setups. The only reliable "lever" for speed was 2D Matryoshka (using fewer layers), which reduces the actual encoder computation.
  • Compositionality: Truncation and int8 quantization compose remarkably well. They act multiplicatively on storage savings without creating a performance "cliff" beyond what each does individually.
Table 2: A comparison of various embedding compression techniques, highlighting the pros and cons of the 1D/2D MR + int8 approach studied here.

Key Results

  • The Sweet Spot: Using int8 quantization at $m=128$ (for a 1024d model) yields a 32x storage reduction while maintaining 94.4% effectiveness.
  • Default Recommendation: For general-purpose RAG where the workload is unknown, the authors recommend int8 at $m=256$, which keeps 96% effectiveness at a 16x storage reduction.
  • Binary Quantization Warning: Binary (1-bit) quantization is significantly worse across every task category, costing roughly 28 percentage points in performance. It is only recommended for the first stage of a two-stage pipeline where a high-precision model reranks the results.
  • 2D Speedup: Sampling from layer 12 (instead of 24) combined with $m=128$ and int8 roughly halved the end-to-end evaluation time, but at a steeper cost of 18.1% in effectiveness.

Limitations and Future Work

  • Scale: Because the largest MTEB retrieval tasks were excluded, the catastrophic failure of retrieval at small dimensions might be even more pronounced in real-world, million-scale indices.
  • Hardware Dependence: The timing results were collected on NVIDIA T4 GPUs using specific libraries (Sentence-Transformers). Results might vary on different hardware or with highly optimized custom kernels (like those using AVX-512 for int8).
  • Comparison Baselines: The study did not directly compare Matryoshka against other compression methods like Product Quantization (PQ) or PCA under the same controlled protocol. The authors suggest this as the most valuable direction for future research.

Summary updated

Abstract

Hierarchical text embedding models offer variable-length representations, while quantization reduces numerical precision; both aim to improve computational efficiency. This work is an empirical benchmark study of how these two orthogonal, already-established techniques interact. We evaluate 38 tasks from the Massive Text Embedding Benchmark (MTEB) using four transformer-based embedding models, covering 1D and 2D hierarchical representations combined with 32-bit, 8-bit, and binary quantization, for 168 configurations per task. We report each efficiency dimension separately: theoretical per-vector storage and end-to-end evaluation time. Under a Friedman test with Nemenyi post-hoc analysis, int8 quantization is statistically indistinguishable from float32 for pair classification and re-ranking and costs at most 1.5 percentage points overall, whereas binary quantization is significantly worse across every task category. Halving the embedding dimension once is not statistically detectable above $m=16$; two successive halvings always are. Combining an 8x dimensionality reduction with int8 yields a 32x reduction in per-vector storage at 94.4% of baseline effectiveness. Because evaluation times were measured once, we treat them as indicative: we find no evidence that quantization speeds up evaluation, whereas sampling from intermediate layers shortens it substantially. Retrieval degrades far faster than every other category, so we report findings per task category.
BibTeX
@misc{ponel2026balancing,
  author = {Kai Ponel and Marco Polignano and Erasmo Purificato and Aylin Julia Temizkan and Marco Levantesi and Giovanni Semeraro and Ernesto William De Luca},
  title = {{Balancing hierarchical representation and quantization: A performance trade-off in text embeddings}},
  year = {2026},
  eprint = {2611.06859},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06859},
}
Figure