Towards Improved Sentence Representations using Token Graphs

Krishna Sri Ipsit Mantri, Carola-Bibiane Schönlieb, Zorah Lähner, Moshe Eliasof

Abstract

Obtaining a single-vector representation from a Large Language Model's (LLM) token-level outputs is a critical step for nearly all sentence-level tasks. However, standard pooling methods like mean or max aggregation treat tokens as an independent set, discarding the rich relational structure captured by the model's self-attention layers and making them susceptible to signal dilution. To address this, we introduce GLOT, a lightweight, structure-aware pooling module that reframes pooling as relational learning followed by aggregation. Operating on the outputs of a frozen LLM, GLOT first constructs a latent token-similarity graph, then refines token representations with a graph neural network, and finally aggregates them using a readout layer. Experimentally, our approach is remarkably robust and efficient: on a diagnostic stress test where 90% of tokens are random distractors, GLOT maintains over 97% accuracy while baseline methods collapse. Furthermore, it is competitive with state-of-the-art techniques on benchmarks like GLUE and MTEB with 20x fewer trainable parameters and speeds up the training time by over 100x compared with parameter-efficient fine-tuning methods. Supported by a theoretical analysis of its expressive power, our work shows that learning over token graphs is a powerful paradigm for the efficient adaptation of frozen LLMs.

Introduction

Large Language Models (LLMs) produce a sequence of vectors—one for every token in a sentence. However, most downstream tasks, such as sentiment analysis or document retrieval, require a single, fixed-length "sentence vector." The current industry standard relies on simple heuristics like mean pooling (averaging all token vectors) or max pooling.

These approaches suffer from two fundamental flaws. First, the Independence Assumption treats tokens as an independent set, ignoring the rich relational structure (e.g., how a negation word like "not" modifies the entire phrase). Second, Signal Dilution occurs in long or noisy texts, where a few critical tokens are drowned out by irrelevant "distractor" tokens. While full fine-tuning can mitigate these issues, it is computationally prohibitive for billion-parameter models. GLOT (Graph-based Learning Over Tokens) addresses this by providing a smart, structure-aware pooling layer that works with a frozen backbone.

The GLOT Architecture

GLOT is a lightweight, three-stage pipeline designed to operate on the hidden states of a frozen LLM. By reframing pooling as a learning process rather than a simple mathematical reduction, GLOT preserves the logical dependencies within a sentence.

The GLOT architecture: A lightweight graph-based pooling module for frozen LLMs.
  1. Token Graph Construction: GLOT treats the sentence as a graph rather than a sequence. It calculates cosine similarity between all pairs of token vectors and creates edges only where similarity exceeds a threshold ($\tau$). This threshold acts as a semantic filter, pruning weak connections and focusing the model on meaningful relationships.
  2. TOKEN-GNN: The graph is processed through a lightweight Graph Neural Network (GNN). This allows tokens to "communicate" with their neighbors, refining their representations based on local context. This step captures critical dependencies like subject-verb relationships.
  3. Readout Layer: Finally, a learnable attention mechanism assigns importance scores to the refined tokens, collapsing them into a single final vector ($z$).

Crucially, the LLM remains frozen; only the GNN and readout layer (approximately 9 million parameters) are trained.

Experimental Evaluation

The researchers evaluated GLOT across a diverse range of models (from BERT to Mistral-7B) and benchmarks:

  • GLUE Benchmark: Nine classic natural language understanding tasks (e.g., CoLA, MNLI, SST-2).
  • MTEB: Seven diverse tasks including retrieval and summarization.
  • Diagnostic Stress Test: A synthetic benchmark where a "signal phrase" is buried within a "haystack" of random distractor words at ratios up to 90%.

Key Results and Insights

GLOT consistently outperforms standard pooling methods and recent learnable baselines like AdaPool across nearly every benchmark. As shown in the performance-to-parameter comparison, GLOT achieves state-of-the-art results while remaining significantly more efficient than alternatives like LoRA.

Performance vs. Trainable Parameters across GLUE tasks. GLOT (blue) achieves higher performance with significantly fewer parameters.
  • Efficiency and Speed: Compared to LoRA, GLOT uses 20x fewer trainable parameters and is 100x faster per training batch. On a Mistral-7B backbone, GLOT required only 0.42 GB of VRAM for training, whereas full fine-tuning would require over 32 GB.
  • Extreme Robustness: In the 90% distractor stress test, GLOT maintained over 97% accuracy, while standard pooling methods collapsed to near-random chance (50-60%). This demonstrates GLOT's unique ability to isolate the "needle in the haystack" by focusing on relational patterns.
  • Adapting Causal Models: Causal models like Llama and Mistral are optimized for next-token prediction, not holistic representation. GLOT effectively "re-targets" these models for sentence-level understanding without modifying their internal weights.

Conclusion and Future Work

GLOT demonstrates that learning over token graphs is a powerful paradigm for the efficient adaptation of frozen LLMs. While the current implementation has a quadratic complexity $O(L^2)$ relative to sequence length, it currently accounts for only 1.3% of total inference time. Future work may explore dynamic similarity thresholds or the application of graph-based pooling to cross-modal architectures, such as Vision Transformers.

Summary updated

Abstract

Obtaining a single-vector representation from a Large Language Model's (LLM) token-level outputs is a critical step for nearly all sentence-level tasks. However, standard pooling methods like mean or max aggregation treat tokens as an independent set, discarding the rich relational structure captured by the model's self-attention layers and making them susceptible to signal dilution. To address this, we introduce GLOT, a lightweight, structure-aware pooling module that reframes pooling as relational learning followed by aggregation. Operating on the outputs of a frozen LLM, GLOT first constructs a latent token-similarity graph, then refines token representations with a graph neural network, and finally aggregates them using a readout layer. Experimentally, our approach is remarkably robust and efficient: on a diagnostic stress test where 90% of tokens are random distractors, GLOT maintains over 97% accuracy while baseline methods collapse. Furthermore, it is competitive with state-of-the-art techniques on benchmarks like GLUE and MTEB with 20x fewer trainable parameters and speeds up the training time by over 100x compared with parameter-efficient fine-tuning methods. Supported by a theoretical analysis of its expressive power, our work shows that learning over token graphs is a powerful paradigm for the efficient adaptation of frozen LLMs. Our code is published at https://github.com/ipsitmantri/GLOT.
BibTeX
@inproceedings{mantri2026towards,
  author = {Krishna Sri Ipsit Mantri and Carola-Bibiane Schönlieb and Zorah Lähner and Moshe Eliasof},
  title = {{Towards Improved Sentence Representations using Token Graphs}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=stMX9KBhUI},
}
Figure