Nugget: Neural Agglomerative Embeddings of Text

Guanghui Qin, Benjamin Van Durme

Abstract

Embedding text sequences is a widespread requirement in modern language understanding. Existing approaches focus largely on constant-size representations. This is problematic, as the amount of information in a text naturally scales with its length. NUGGET introduces a novel text encoding strategy that represents language using a dynamic, fractional number of embeddings instead of a single fixed-size vector. By learning to select the most informative tokens as "nuggets," the model achieves high compression while maintaining enough semantic detail to outperform standard baselines in document retrieval and long-context language modeling.

The "Goldilocks" Problem of Text Embeddings

In Natural Language Processing (NLP), researchers frequently encounter a "Goldilocks" problem when designing text embeddings.

On one end of the spectrum, passage-level models (such as BERT's [CLS] token) attempt to compress the meaning of an entire document into a single vector. As researcher Ray Mooney famously noted, "You can't cram the meaning of a whole %&!$# sentence into a single $&!#* vector." This limitation becomes especially apparent with long, complex texts.

On the other end, token-level models (like ColBERT) store a vector for every single word. While highly expressive, this approach is incredibly memory-intensive and computationally expensive to search.

NUGGET addresses this gap by proposing a flexible middle ground: a representation that scales dynamically with the length of the input but remains significantly more compact than a full token-level index.

Methodology: The Nugget Generator

The core of the proposed approach is the Nugget Generator, which utilizes a modified Transformer encoder-decoder architecture to dynamically compress text.

  1. Selection via Hard Attention: A feedforward network scores every token in the input sequence. The model then selects the top $k$ tokens as "nuggets," where $k$ is determined by a predefined compression ratio ($r$) of the total input length.
  2. Ensuring Differentiability: Selecting the "top $k$" is inherently non-differentiable, meaning standard backpropagation gradients cannot flow through the operation. To resolve this, the authors introduce a residual connection that appends the selection scores directly to the decoder's cross-attention mechanism. This effectively "tricks" the model into learning which tokens are most useful for reconstructing the text.
  3. Informed Encoding: Rather than selecting nuggets only at the final layer, the model "prepones" the selection to an earlier layer. It then injects "type embeddings" to explicitly mark tokens as either a nugget or a non-nugget. This allows the encoder to focus its attention behavior on the most critical parts of the text during subsequent processing.
  4. Training Objectives: The architecture is trained using either an Autoencoding (AE) objective (reconstructing the original input from the selected nuggets) or a Machine Translation (MT) objective (generating a translation based solely on the nuggets).

Experimental Setup and Datasets

The researchers evaluated NUGGET across several distinct tasks and datasets:

  • WMT19 (English-Chinese): Utilized for the primary training of the Autoencoding and Machine Translation models.
  • PARABANK: The authors constructed a new document-level paraphrase identification benchmark by concatenating sentences and randomly deleting segments to increase task difficulty.
  • WikiText-103: Used for a novel passage re-ranking task and for evaluating long-range language modeling (predicting the next word based on a historical context of nuggets).

Key Insights

  • Punctuation as a Semantic Summary: Surprisingly, without being programmed with any explicit linguistic rules, the model naturally learns to select punctuation (such as commas and periods) and conjunctions (like "and" or "but") as its nuggets. The authors hypothesize that these tokens act as natural "clausal delimiters" that effectively aggregate the semantics of the preceding text segment.
  • Divide-and-Conquer Representation: Probing experiments revealed that each individual nugget primarily encodes the segment of text immediately preceding it. This suggests that NUGGET organically segments language into meaningful, bite-sized semantic units.
  • Near-Lossless Compression: At a compression ratio of just 10% ($r=0.1$), the model can reconstruct the original text almost perfectly (achieving a BLEU score > 0.99). This proves that the vast majority of tokens in a standard sentence are redundant when it comes to capturing core meaning.

Performance and Results

  • Document Similarity: When trained with the Machine Translation objective, NUGGET significantly outperformed the single-vector baseline (TSDAE). Most impressively, at a highly restrictive 5% compression ratio, NUGGET matched the performance of the token-heavy ColBART model while utilizing 20x fewer vectors.
  • Language Modeling: When deployed as a "long-term memory" module for a language model, NUGGET reduced perplexity (a metric indicating how well a model predicts a sample) more effectively than Compressive Transformers. This demonstrates that NUGGET provides a higher-quality, more actionable summary of past context.
  • Configurable Efficiency: The model offers a highly configurable trade-off. Practitioners can select a lower compression ratio ($r$) for massive memory savings or a higher $r$ for maximum performance, easily adapting to specific hardware constraints.

Limitations and Future Work

While NUGGET presents a compelling step forward for dynamic text embeddings, the authors note a few areas for future exploration:

  • Contrastive Learning: The current study focused heavily on reconstruction tasks (AE/MT). Incorporating contrastive learning—training the model to explicitly distinguish between similar and dissimilar passages—could further enhance its performance in dense retrieval tasks.
  • Scaling to LLMs: The experiments were conducted on BART-sized models (approximately 600M parameters). Investigating how NUGGET behaves within massive Large Language Models (LLMs) to efficiently extend their context windows is a highly promising next step.
  • Computational Overhead: Although NUGGET drastically saves memory during storage and indexing, the Top-K selection process and the gradient feedback mechanism do introduce some additional computational complexity during the initial encoding phase.

Summary updated

Abstract

Embedding text sequences is a widespread requirement in modern language understanding. Existing approaches focus largely on constant-size representations. This is problematic, as the amount of info...
BibTeX
@inproceedings{qin2023nugget,
  author = {Guanghui Qin and Benjamin Van Durme},
  title = {{Nugget: Neural Agglomerative Embeddings of Text}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=LhFE049fh5},
}
Figure