TL;DR
This paper investigates the relationship between the "squared norm" of text embeddings and their "information content" (IC). The authors propose theoretical conditions for this correspondence to hold and empirically show that certain composition functions and models preserve it, particularly when aiming for linear growth of IC with text length, aligning with the Constant Entropy Rate hypothesis.
What Problem Does This Paper Address?
Modern Natural Language Processing (NLP) heavily relies on text embeddings, which represent words and sentences as numerical vectors. The utility of these embeddings depends not only on their quality but also on how they are combined (composition) and how their inherent information content (IC) can be measured. Previous work has suggested a link between the squared norm of an embedding and its IC, particularly for word-level embeddings in specific models. However, this relationship at the sequence (text) level, and how it's affected by different composition mechanisms and modern contextual embeddings, remains less understood. This paper aims to bridge this gap by formally analyzing and empirically validating the correspondence between squared norm and IC for text sequences, focusing on mechanisms that preserve the expected linear growth of IC with text length (Constant Entropy Rate hypothesis).
Methods
The authors propose two theoretical conditions for the correspondence between IC and squared norm to hold:
- Information Monotonicity: The IC of a represented text must grow monotonically with its embedding's norm.
- Information Additivity: Each dimension of the embedding must contribute independently to the estimation of IC.
They then analyze various composition functions for combining token embeddings into text embeddings. This includes:
- Summation and Averaging: Standard methods that are shown to not preserve the linear growth of IC.
- Scaled Embedding Average: A novel proposal that normalizes the sum of embeddings by the square root of the number of tokens, which does satisfy the linear growth requirement.
- Information Theory-based Functions (Find, FInf, FJoint): Functions designed to capture specific semantic relationships, with Find and FInf showing promise for preserving CER.
- Token-based Representations (CLS, EOS): Using a single token (like BERT's CLS or LLaMA's EOS) to represent the entire sequence.
The empirical evaluation involves correlating word-level IC with squared norms across various static and contextual embedding models, and then analyzing how the squared norm of text embeddings evolves with increasing sequence length.
Datasets
The paper uses the Brown corpus for word-level analysis of information content and squared norms. For sequence-level experiments, they utilize text sequences from the C4 dataset (specifically, the realnewslike subset), sampled to represent varying lengths.
Key Insights and Ideas
- Theoretical Foundation: The paper provides formal conditions (monotonicity and additivity) that theoretically link an embedding's squared norm to its information content, offering a principled way to analyze this relationship.
- Composition Matters: Simple summation and averaging of word embeddings do not preserve the expected linear growth of information content with text length (CER hypothesis). This highlights the importance of choosing appropriate composition functions.
- Scaled Average as a Solution: The proposed "scaled embedding average" offers a simple yet effective way to maintain the linear growth of IC, aligning with CER.
- Token-based Limitations: Representations relying on a single token (like BERT's CLS or LLaMA's EOS) tend to plateau in information content growth for longer sequences, suggesting they are not ideal for capturing IC in extended texts.
- ModernBERT's Promise: The CLS token in ModernBERT shows more promising behavior regarding CER than in standard BERT, hinting at architectural improvements.
- Information Theory Functions: Certain information-theoretic composition functions (Find, FInf) demonstrate an ability to preserve the linear growth of IC, offering more sophisticated alternatives to simple averaging.
Key Results
- Word Level: While initial correlations between word IC and squared norm were low, after applying smoothing and aggregation, models showed strong correlations (above 0.95 for LLaMA, and around 0.985 for ModernBERT) when using appropriate composition functions. Static models showed lower average absolute dimension correlation, indicating less independence between dimensions.
- Sequence Level:
- Scaled Average: Consistently showed linear growth of squared norm with sequence length across models.
- Summation: Exhibited accelerated growth.
- Averaging: Showed decreasing behavior.
- Token-based (CLS/EOS): Showed decelerated growth, plateauing after a certain length (e.g., ~400 tokens for BERT CLS, ~100 for LLaMA EOS).
- Information Theory Functions: Find and FInf generally showed linear growth, while FJoint exhibited decreasing behavior in some cases.
- ModernBERT CLS: Displayed a more CER-consistent behavior than standard BERT's CLS token.
Limitations and Future Work
- Non-Compositional Approaches: The study excluded some non-compositional approaches that are trained specifically for sentence representation, leaving open the possibility of interesting behavior at longer text lengths.
- Model-Specific Functions: Some models might exhibit consistent behavior if specific estimation functions for information content are defined for them.
- Dataset Limitations: The word-level correlation experiments used limited datasets, and more sophisticated methods could be applied to analyze dimension independence.
- Sequence Probabilities: The paper did not extend to analyzing empirical probabilities of word sequences due to the difficulty of estimation.
- Downstream Tasks: The impact of the correspondence between IC and embedding norm on downstream NLP tasks was not investigated.
- Future Work: The authors suggest exploring the parametrization of composition functions by empirically analyzing entropy growth rates and potentially adjusting embedding processes. They also propose further investigation into word-level aspects and the impact on downstream tasks.
Summary updated