TRIAL: Token Relations and Importance Aware Late-interaction for Accurate Text Retrieval

Hyukkyu Kang, Injung Kim, Wook-Shin HanEMNLP 2025

TL;DR

This paper introduces TRIAL, a novel late-interaction retrieval model that significantly improves text retrieval accuracy by explicitly modeling token relationships and their importance. By going beyond simple token similarity, TRIAL achieves state-of-the-art results on several benchmarks, offering a more nuanced understanding of query-document relevance.

What Problem Does This Paper Address?

Accurate text retrieval is crucial for many applications, from search engines to question-answering systems. While modern retrieval models have made great strides, existing "late-interaction" methods, which compare token-level representations, often suffer from two main issues:

  1. Phrase Fragmentation: When words or phrases are broken down into multiple tokens (e.g., "Scott Derrickson" becoming "scott", "derrick", "##son"), simple token-wise matching can lead to inaccurate relevance scores, especially if some tokens appear in unrelated contexts.
  2. Ignoring Term Importance: Traditional methods like TF-IDF recognize that not all words are equally important. Current late-interaction models often treat all tokens equally, failing to prioritize critical terms in a query.

These limitations hinder the ability of retrieval systems to precisely capture semantic similarity and can lead to suboptimal search results.

Methods

TRIAL addresses these challenges by introducing two key innovations to the late-interaction framework:

  • Token Relations: TRIAL explicitly models the semantic relationships between tokens. It computes a "token relation score" by comparing how tokens relate to each other within the query and the document. This helps in understanding multi-token entities as cohesive units, mitigating the noise introduced by phrase fragmentation. The relation between two tokens is encoded using a two-layer MLP and then compared via a dot product.
  • Token Importance Weighting: TRIAL incorporates a "token importance weight" for each query token. This weight is learned by a gating mechanism, inspired by techniques used to identify contextual stopwords. This allows the model to prioritize semantically important query terms, similar to how TF-IDF emphasizes important words, leading to more accurate relevance scoring.

The overall scoring function in TRIAL combines the standard token similarity with these new relation and importance scores.

Datasets

The authors evaluated TRIAL on three benchmark datasets:

  • MSMARCO: An in-domain dataset for web search queries and passages.
  • BEIR: A collection of 13 diverse datasets for out-of-domain evaluation, covering various retrieval tasks like fact-checking and question answering.
  • LoTTE: A benchmark focusing on long-tail topics and natural search queries, derived from StackExchange, with both search and forum datasets.

Key Insights and Ideas

  • Beyond Simple Similarity: The core insight is that simply matching individual tokens, even with contextual embeddings, is insufficient. Understanding how tokens relate to each other and which tokens are most important is crucial for accurate relevance.
  • Mitigating Fragmentation Noise: The token relation mechanism effectively treats multi-token phrases as single semantic units, preventing irrelevant matches from fragmented tokens from dominating the score.
  • Reintroducing Term Importance: By learning query token importance, TRIAL brings back a crucial aspect of traditional retrieval that was largely missing in many modern neural models, allowing it to focus on the most critical parts of a query.
  • Practicality of Late-Interaction: TRIAL builds upon the efficiency of late-interaction models, striking a balance between accuracy and computational cost, making it suitable for large-scale retrieval.

Key Results

  • State-of-the-Art Accuracy: TRIAL achieved state-of-the-art nDCG@10 scores on MSMARCO (46.3) and competitive average nDCG@10 scores on BEIR (51.09) and LoTTE Search (72.15).
  • Ablation Studies Confirm Contributions: Removing either the token relation or token importance components led to a noticeable drop in performance, demonstrating the effectiveness of both proposed mechanisms.
  • Competitive Retrieval Speed: Despite the added complexity of relation modeling, TRIAL maintains competitive retrieval speeds comparable to existing late-interaction methods like ColBERTv2. The paper shows that even with quadratic complexity w.r.t. document tokens, GPU parallelization makes it practical.
  • Qualitative Improvements: Examples from BEIR datasets show TRIAL successfully ranking gold documents higher when they contained fragmented phrases that were previously missed or misinterpreted by simpler models. Analysis of token importance weights also shows TRIAL prioritizing semantically rich tokens (nouns, adjectives) over functional ones.

Limitations and Future Work

  • Computational Overhead: While practical, TRIAL introduces additional computational overhead due to relation modeling. The quadratic complexity with respect to document tokens might limit scalability for extremely long queries.
  • Domain Shift Robustness: TRIAL's performance showed a slight decrease on the LoTTE-Forum dataset compared to LoTTE-Search, suggesting that its current modeling might be less effective under significant domain shifts or for informal, user-generated content.
  • Language Specificity: The current evaluation is limited to English-language datasets. The effectiveness of TRIAL on morphologically rich or non-English languages remains unexplored.

Future work could focus on improving robustness to diverse linguistic styles and informal discourse, exploring its applicability to other languages, and potentially developing more efficient ways to handle very long documents.

Summary updated

BibTeX
@inproceedings{kang2025trial,
  author = {Hyukkyu Kang and Injung Kim and Wook-Shin Han},
  title = {{TRIAL: Token Relations and Importance Aware Late-interaction for Accurate Text Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure