Do Retrieval Ranking Models Capture Event Order? A Study of Timeline-Constrained Retrieval

Alon Eirew, Lidan Avisar, Kfir Bar, Ido DaganarXiv 2026 · arXiv

Modern search engines are incredibly good at finding documents that match the topic of your query. If you search for "earthquake relief," you’ll get plenty of relevant reports. But what if the sequence of events matters? If you need reports where a volcanic eruption was followed by lava flows, then infrastructure damage, and finally aid delivery, standard models might fail you. They often treat documents like a "bag of events," ignoring the chronological thread that connects them.

A new paper from researchers at Bar-Ilan University and Reichman University, titled "Do Retrieval Ranking Models Capture Event Order?", investigates this gap. They find that even state-of-the-art semantic models struggle with temporal constraints and propose a "timeline-aware" approach that significantly boosts performance.

TL;DR

Current semantic retrieval models (like E5 or BGE) excel at topical matching but largely ignore the temporal order of events within documents. By introducing a novel "Order-Preserving Alignment" method that explicitly enforces chronological consistency, the researchers achieved massive improvements in retrieval accuracy for timeline-constrained queries.

What Problem Does This Paper Address?

In domains like healthcare (patient trajectories), law enforcement (modus operandi), and disaster management, information needs are often defined by a progression of events.

The authors identify two core requirements for "timeline-constrained" relevance:

  1. Semantic Support: Does the document contain evidence for every event in the query?
  2. Temporal Consistency: Do these events happen in the document in the same chronological order as specified in the query?

As shown in Figure 1, a document might discuss all the right events but in the wrong order, making it irrelevant to a specific timeline-based information need.

Figure 1: Given a timeline query and a chronologically ordered timeline of events extracted from a candidate disaster report, relevance requires both semantic sup- port for all query statements and consistency with their temporal order. The top candidate is relevant because its events progression aligns with all query statements in an order-preserving manner, while unrelated sentences are skipped. The bottom candidate, although supports the same events, is inconsistent with the progression specified by the query timeline.

Methods

The researchers developed a framework to test and improve how models handle event order. Their approach involves three main steps:

  1. Chronological Document Representation: Since news and reports often describe events non-chronologically (e.g., "The investigation found that the fire started..."), they use an LLM to re-order document sentences into their true real-world chronological sequence.
  2. Statement-Sentence Compatibility: They use a cross-encoder (a high-accuracy neural reranker) to score how well each query event matches each document sentence.
  3. Order-Preserving (OP) Alignment: This is the "secret sauce." Instead of just picking the best matches, they use a dynamic programming algorithm (similar to those used in DNA sequence alignment) to find the highest-scoring match that preserves the order of the query events.

They compared this to an Order-Agnostic (OA) baseline, which uses the same similarity scores but allows events to match in any order.

Figure 3: The order-preserving and order-agnostic statement–sentence alignment framework for timeline- aware reranking (§4.3).

Datasets

The authors introduced a new focused evaluation collection:

  • Source: 6,145 disaster reports from ReliefWeb.
  • Queries: 66 timeline queries (e.g., "Insect outbreak began -> Food insecurity increased -> Response plan developed").
  • Labels: They used a rigorous "decomposed" protocol where an LLM (GPT-5.2) assessed each query statement for entailment and then verified the global temporal order. This was validated by human annotators with high agreement (Cohen’s $\kappa = 0.83$).

Key Insights and Ideas

  • Semantic models are "Bag-of-Events" models: Even the most advanced dense retrievers (like Qwen-Embed) performed similarly to or worse than the simple, keyword-based BM25. This suggests that "semantic" matching in these models is mostly about topical overlap, not structural or relational understanding.
  • Instructions aren't a silver bullet: Simply telling a model in the prompt to "consider the order of events" yielded almost no improvement. The limitation appears to be architectural rather than a lack of instruction following.
  • Order is a powerful ranking signal: The massive jump in performance when switching from Order-Agnostic to Order-Preserving alignment proves that temporal progression is a distinct, valuable signal that current models are leaving on the table.

Key Results

The quantitative results were striking. When using the MiniLM-L12 model as a base:

  • Order-Agnostic (OA) alignment achieved an MRR@10 of 0.55.
  • Order-Preserving (OP) alignment jumped to 0.70.
  • mAP@10 (Mean Average Precision) rose from 0.36 to 0.50.

As seen in the results table, the OP method consistently outperformed all standard retrievers and rerankers across all metrics.

Table 1: Test-set results showing that Order-Preserving (OP) configurations significantly outperform standard models and order-agnostic baselines.

Limitations and Future Work

While the results are promising, the authors note a few limitations:

  • Scale: The evaluation set is relatively small (66 queries) due to the high cost of high-quality temporal labeling.
  • Domain Specificity: The study focused on disaster reports; results might vary in domains with more complex or ambiguous timelines (like fiction).
  • Preprocessing Overhead: The method relies on an offline step to chronologically order document sentences, which currently requires an LLM.

The researchers suggest that future work should focus on building "temporal awareness" directly into the architecture of retrievers, rather than relying on post-processing alignment.

Summary updated

Abstract

Semantic retrieval models aim to match documents to a query's "meaning", yet it remains unclear whether they successfully encode temporal relations between events in their ranking mechanisms. We investigate this question by studying timeline-constrained document retrieval, where a query is an ordered sequence of natural-language event statements while a relevant document should match those events in a compatible temporal progression. We then test whether widely used dense retrievers and neural rerankers capture such constraints, and whether explicitly preserving temporal order improves ranking, using a controlled setting over a novel focused evaluation collection. Experiments show that state-of-the-art semantic retrievers and rerankers provide only limited improvements over lexical retrieval, suggesting that their semantic matching capabilities do not reliably capture temporal constraints. We then compare a timeline-aware method with both semantic matching and standard rerankers and find that it yields substantial improvements. This demonstrates that considering temporal order can provide a strong ranking signal in this setting, suggesting future research on improving temporal modeling in semantic retrievers.
BibTeX
@misc{eirew2026do,
  author = {Alon Eirew and Lidan Avisar and Kfir Bar and Ido Dagan},
  title = {{Do Retrieval Ranking Models Capture Event Order? A Study of Timeline-Constrained Retrieval}},
  year = {2026},
  eprint = {2611.04343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04343},
}
Figure