TL;DR
GLOT (Graph-based Learning Over Tokens) is a lightweight pooling module that reframes sentence embedding as relational learning over token graphs. By sitting on top of frozen Large Language Models (LLMs), it achieves state-of-the-art performance with 20x fewer parameters and 100x faster training than standard fine-tuning methods like LoRA.
The Problem with Standard Pooling
To use an LLM for sentence-level tasks—such as sentiment analysis or document retrieval—we must "collapse" a sequence of token-level hidden states into a single vector. Standard practices rely on simple heuristics: averaging all tokens (Mean pooling), taking the maximum value (Max pooling), or using a special [CLS] or [EOS] token.
The authors argue these methods are fundamentally flawed because they treat tokens as an independent set, discarding the rich relational structure the LLM worked so hard to learn via self-attention. This "compressive" approach is particularly problematic for decoder-only models (like Llama or Mistral), which are optimized for next-token prediction rather than holistic sentence representation. Furthermore, these simple methods are highly susceptible to "signal dilution"—where a few important words are drowned out by many irrelevant "distractor" tokens.
Introducing GLOT: Graph-based Learning Over Tokens
The authors introduce GLOT, a structure-aware pooling module that operates in three stages:
- Token Graph Construction: It builds a latent graph where each token is a node. Edges are created between tokens if their cosine similarity exceeds a certain threshold ($\tau$). This creates a sparse, semantic map of the sentence.
- Relational Learning (TOKEN-GNN): A lightweight Graph Neural Network refines the token representations. Through message passing, tokens "talk" to their neighbors, allowing the model to capture complex dependencies (like a "not" negating a "good") before any compression happens.
- Readout Layer: A learnable attention mechanism aggregates these refined representations into a final sentence vector $\mathbf{z}$.
Crucially, the underlying LLM remains entirely frozen. Only the tiny GLOT head is trained, making it extremely resource-efficient.
Evaluation and Datasets
The researchers evaluated GLOT across a wide array of benchmarks using six different LLM backbones (BERT, RoBERTa, SmolLM2, TinyLlama, LLaMA-3B, and Mistral-7B):
- GLUE Benchmark: Eight diverse tasks for general language understanding.
- MTEB (Massive Text Embedding Benchmark): Seven tasks including classification, retrieval, and summarization.
- IMDB: Long-text sentiment classification.
- Diagnostic Stress Test: A newly introduced synthetic benchmark designed to test robustness. It injects a "signal phrase" (e.g., "The file has [X] but not [Y]") into a "haystack" of random distractor tokens, with noise levels ranging from 20% to 90%.
Key Insights and Results
- Efficiency without Sacrifice: GLOT uses 20x fewer trainable parameters than LoRA and speeds up training time by over 100x per batch. It requires only ~0.42 GB of GPU memory for training on a 7B model, compared to >32 GB for full fine-tuning.
- Extreme Robustness: In the 90% distractor stress test, GLOT maintained over 97% accuracy, while baseline methods like Mean pooling or AdaPool saw their performance plummet toward near-chance levels. GLOT's graph-based approach allows it to ignore irrelevant nodes and focus on the semantic "anchors" of a sentence.
- Sparsity is a Feature: The similarity threshold ($\tau$) is critical. By pruning weak edges, the model focuses on the most important semantic connections, which actually improves performance compared to a fully connected graph.
- Performance: GLOT consistently outperformed all pooling baselines across all tested LLMs on the GLUE benchmark. On tasks like CoLA (grammaticality), it improved BERT's performance by a relative 62%.
Limitations and Future Work
The authors acknowledge a few areas for further exploration:
- Complexity: Constructing the similarity graph has an $O(L^2)$ complexity relative to sequence length. While they showed this is manageable for up to 32k tokens (~1.3% overhead), it could become a bottleneck for even longer contexts.
- Graph Construction: Currently, the graph is built using simple cosine similarity. Future work could explore more sophisticated, learnable ways to define edges.
- Other Modalities: The "relational learning before compression" paradigm could potentially be applied to other fields, such as pooling patch embeddings in Vision Transformers.
Summary updated