Nugget: Neural Agglomerative Embeddings of Text

Guanghui Qin, Benjamin Van Durme

Abstract

Representing text for downstream tasks typically involves a trade-off between the extreme compression of single-vector embeddings and the high computational cost of token-level representations. This paper introduces NUGGET, a novel text encoding strategy that dynamically selects a subset of "nugget" embeddings to represent language. By utilizing a differentiable hard-attention mechanism and a configurable compression ratio, NUGGET achieves near-lossless text reconstruction while significantly reducing memory overhead. Our results demonstrate that NUGGET outperforms single-vector baselines in document similarity and enhances long-range language modeling, providing a scalable middle ground for efficient natural language processing.

1. Introduction

In Natural Language Processing (NLP), the representation of text often falls into two extremes. Models like BERT typically condense the semantic meaning of an entire passage into a single, fixed-size vector (the [CLS] token). However, as noted by researchers like Ray Mooney, squeezing a complex sentence into a single mathematical point inevitably loses vital nuance. Conversely, multi-vector models like ColBERT store an embedding for every token in a document. While this preserves detail, the memory and computational requirements scale linearly with document length, making it prohibitively expensive for massive datasets or long-form content.

NUGGET (Neural Agglomerative Embeddings) proposes a "fractional" middle ground. It produces a representation that grows with the length of the input but remains significantly more compact than token-level approaches, allowing for a tunable balance between fidelity and efficiency.

2. Methodology

2.1 The Nugget Generator

The core component of the architecture is the Nugget Generator. Unlike standard encoders that produce a fixed number of outputs, the Nugget Generator uses a hard-attention mechanism to select the most informative tokens from the input to serve as "nuggets."

2.2 Technical Innovations

To enable effective learning and selection, the authors introduced several key innovations:

  • Dynamic Selection: The number of nuggets ($k$) is defined as a fraction of the input length ($n$), controlled by a compression ratio ($r$). This ensures the representation scales logically with the amount of information provided.
  • Differentiable Top-K Selection: Selecting discrete tokens is a non-differentiable operation. To allow for end-to-end training via backpropagation, the authors implemented a residual connection that appends selection scores directly to the decoder's cross-attention logits. This allows the gradient to flow back to the selection mechanism.
  • Nugget Feedback: To optimize the encoding process, the model identifies nuggets at an intermediate layer and injects "type embeddings" into the hidden states. This informs the upper layers of the encoder which tokens have been selected, allowing the model to focus its processing power on refining those specific representations.

2.3 Training Objectives

The model is trained using two primary objectives: an Autoencoding (AE) objective, where the model must reconstruct the original input from the nuggets, and a Machine Translation (MT) objective, where nuggets from a source language are used to generate text in a target language.

3. Empirical Evaluation

3.1 Datasets and Tasks

The authors evaluated NUGGET across diverse tasks:

  • WMT19 (English-Chinese): Primary training for MT and AE models.
  • ParaBank: A custom document-level paraphrase identification dataset designed to simulate real-world noise.
  • WikiText-103: Used for passage re-ranking and evaluating long-range language modeling capabilities.
  • Cross-lingual Analysis: Testing token selection patterns across 10 languages to ensure the method is not language-specific.

3.2 Key Results

  • Document Similarity: NUGGET (trained with MT) significantly outperformed single-vector baselines like TSDAE. In passage re-ranking tasks, it achieved performance parity with ColBERT while utilizing 6.7x to 20x fewer vectors.
  • Language Modeling: When integrated as an auxiliary memory for a Language Model (LM), NUGGET allowed the model to leverage context from up to 1,152 tokens away, consistently reducing perplexity compared to standard Transformers.
  • Lossless Compression: At a compression ratio of $r=0.1$ (10% of original tokens), the model achieved a BLEU score exceeding 0.99 in reconstruction tasks, indicating near-lossless information preservation.

4. Discussion and Linguistic Analysis

Analysis of the model's behavior revealed that NUGGET naturally learns to identify linguistic structures without explicit supervision. The model frequently selects "clausal delimiters"—such as punctuation (commas, periods) and conjunctions—as nuggets. Probability gain analysis suggests that each nugget primarily encodes the segment of text immediately preceding it, effectively "saving" the semantic progress of the sentence at natural breakpoints.

5. Limitations and Future Work

Despite its success, NUGGET faces challenges regarding training stability; the nugget feedback mechanism currently requires freezing the bottom layers of the encoder to prevent divergence. Future research may explore incorporating contrastive learning to further enhance similarity task performance and scaling the architecture to Large Language Models (LLMs) to manage extremely long-context histories.

6. Conclusion

NUGGET provides a flexible and efficient framework for text representation. By moving away from the "one-size-fits-all" approach of single-vector embeddings and the redundancy of token-level storage, it offers a scalable solution for high-fidelity document processing and long-range language modeling.

Summary updated

BibTeX
@inproceedings{qin2023nugget,
  author = {Guanghui Qin and Benjamin Van Durme},
  title = {{Nugget: Neural Agglomerative Embeddings of Text}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=LhFE049fh5},
}
Figure