Towards Improved Sentence Representations using Token Graphs

Krishna Sri Ipsit Mantri, Carola-Bibiane Schönlieb, Zorah Lähner, Moshe Eliasof

TL;DR

GLOT (Graph-based Learning Over Tokens) is a lightweight pooling module that reframes sentence embedding as relational learning over token graphs. By sitting on top of frozen Large Language Models (LLMs), it achieves state-of-the-art performance with 20x fewer parameters and 100x faster training than standard fine-tuning methods like LoRA.

The Problem with Standard Pooling

To use an LLM for sentence-level tasks—such as sentiment analysis or document retrieval—we must "collapse" a sequence of token-level hidden states into a single vector. Standard practices rely on simple heuristics: averaging all tokens (Mean pooling), taking the maximum value (Max pooling), or using a special [CLS] or [EOS] token.

The authors argue these methods are fundamentally flawed because they treat tokens as an independent set, discarding the rich relational structure the LLM worked so hard to learn via self-attention. This "compressive" approach is particularly problematic for decoder-only models (like Llama or Mistral), which are optimized for next-token prediction rather than holistic sentence representation. Furthermore, these simple methods are highly susceptible to "signal dilution"—where a few important words are drowned out by many irrelevant "distractor" tokens.

Introducing GLOT: Graph-based Learning Over Tokens

The authors introduce GLOT, a structure-aware pooling module that operates in three stages:

  1. Token Graph Construction: It builds a latent graph where each token is a node. Edges are created between tokens if their cosine similarity exceeds a certain threshold ($\tau$). This creates a sparse, semantic map of the sentence.
  2. Relational Learning (TOKEN-GNN): A lightweight Graph Neural Network refines the token representations. Through message passing, tokens "talk" to their neighbors, allowing the model to capture complex dependencies (like a "not" negating a "good") before any compression happens.
  3. Readout Layer: A learnable attention mechanism aggregates these refined representations into a final sentence vector $\mathbf{z}$.

Crucially, the underlying LLM remains entirely frozen. Only the tiny GLOT head is trained, making it extremely resource-efficient.

Overview of the GLOT architecture showing graph construction, GNN refinement, and readout.

Evaluation and Datasets

The researchers evaluated GLOT across a wide array of benchmarks using six different LLM backbones (BERT, RoBERTa, SmolLM2, TinyLlama, LLaMA-3B, and Mistral-7B):

  • GLUE Benchmark: Eight diverse tasks for general language understanding.
  • MTEB (Massive Text Embedding Benchmark): Seven tasks including classification, retrieval, and summarization.
  • IMDB: Long-text sentiment classification.
  • Diagnostic Stress Test: A newly introduced synthetic benchmark designed to test robustness. It injects a "signal phrase" (e.g., "The file has [X] but not [Y]") into a "haystack" of random distractor tokens, with noise levels ranging from 20% to 90%.

Key Insights and Results

  • Efficiency without Sacrifice: GLOT uses 20x fewer trainable parameters than LoRA and speeds up training time by over 100x per batch. It requires only ~0.42 GB of GPU memory for training on a 7B model, compared to >32 GB for full fine-tuning.
  • Extreme Robustness: In the 90% distractor stress test, GLOT maintained over 97% accuracy, while baseline methods like Mean pooling or AdaPool saw their performance plummet toward near-chance levels. GLOT's graph-based approach allows it to ignore irrelevant nodes and focus on the semantic "anchors" of a sentence.
  • Sparsity is a Feature: The similarity threshold ($\tau$) is critical. By pruning weak edges, the model focuses on the most important semantic connections, which actually improves performance compared to a fully connected graph.
  • Performance: GLOT consistently outperformed all pooling baselines across all tested LLMs on the GLUE benchmark. On tasks like CoLA (grammaticality), it improved BERT's performance by a relative 62%.
Performance of GLOT vs baselines on CoLA, RTE, and STS-B benchmarks across different parameter scales.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Complexity: Constructing the similarity graph has an $O(L^2)$ complexity relative to sequence length. While they showed this is manageable for up to 32k tokens (~1.3% overhead), it could become a bottleneck for even longer contexts.
  • Graph Construction: Currently, the graph is built using simple cosine similarity. Future work could explore more sophisticated, learnable ways to define edges.
  • Other Modalities: The "relational learning before compression" paradigm could potentially be applied to other fields, such as pooling patch embeddings in Vision Transformers.

Summary updated

Abstract

Obtaining a single-vector representation from a Large Language Model's (LLM) token-level outputs is a critical step for nearly all sentence-level tasks. However, standard pooling methods like mean or max aggregation treat tokens as an independent set, discarding the rich relational structure captured by the model's self-attention layers and making them susceptible to signal dilution. To address this, we introduce GLOT, a lightweight, structure-aware pooling module that reframes pooling as relational learning followed by aggregation. Operating on the outputs of a frozen LLM, GLOT first constructs a latent token-similarity graph, then refines token representations with a graph neural network, and finally aggregates them using a readout layer. Experimentally, our approach is remarkably robust and efficient: on a diagnostic stress test where 90% of tokens are random distractors, GLOT maintains over 97% accuracy while baseline methods collapse. Furthermore, it is competitive with state-of-the-art techniques on benchmarks like GLUE and MTEB with 20x fewer trainable parameters and speeds up the training time by over 100x compared with parameter-efficient fine-tuning methods. Supported by a theoretical analysis of its expressive power, our work shows that learning over token graphs is a powerful paradigm for the efficient adaptation of frozen LLMs. Our code is published at https://github.com/ipsitmantri/GLOT.
BibTeX
@inproceedings{mantri2026towards,
  author = {Krishna Sri Ipsit Mantri and Carola-Bibiane Schönlieb and Zorah Lähner and Moshe Eliasof},
  title = {{Towards Improved Sentence Representations using Token Graphs}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=stMX9KBhUI},
}
Figure