TL;DR
While Large Language Models (LLMs) are often viewed as universal solvers, they struggle to outperform smaller, specialized encoder-only models like RoBERTa in Temporal Relation Classification (TRC). This study identifies a fundamental "autoregressive bottleneck": LLMs' inherent focus on the end of a sequence hinders their ability to capture the global context necessary for determining event order.
The Challenge of Temporal Relation Classification
Temporal Relation Classification (TRC) involves determining the chronological sequence of events within a text—identifying whether Event A occurred before, after, or simultaneously with Event B. This capability is vital for narrative comprehension, automated summarization, and complex question-answering.
While encoder-only models like RoBERTa have long been the standard, the emergence of powerful LLMs has prompted a shift. However, the "black box" nature of many top-tier LLMs (like GPT-4) makes it difficult to diagnose their specific failures in temporal reasoning. This research benchmarks open-source LLMs to uncover the structural reasons behind the performance gap.
Experimental Setup
The researchers evaluated RoBERTa against a suite of LLMs, including Mistral 7B, Mixtral 8x7B, the Llama2 family, and GPT-3/3.5. They explored three interaction paradigms:
- In-Context Learning (ICL): Few-shot prompting for label prediction.
- Question Answering (QA) Prompts: Reframing classification as binary questions (QA1) or sequential context-building questions (QA2).
- Fine-Tuning: Applying LoRA (Low-Rank Adaptation) to Llama2 models to establish a performance upper bound.
To analyze model behavior, the team employed KernelShap, an explainability tool that identifies which tokens most influence the model's output.
The Autoregressive Bottleneck and Other Insights
The study's most significant finding relates to how models process information:
- The Autoregressive Bottleneck: Explainability analysis revealed that LLMs exhibit a strong "recency bias." Because they are trained for next-token prediction, they focus disproportionately on tokens at the end of the input sequence. In contrast, RoBERTa's encoder architecture allows it to weigh context more uniformly. Since temporal cues are often distributed throughout a sentence, the LLM's narrow focus is a major disadvantage.
- Prompting vs. Architecture: While sequential QA (QA2) improved LLM performance, it could not bridge the gap to RoBERTa. This suggests the issue is architectural rather than just a matter of prompt engineering.
- Knowledge Cutoffs: Llama2 70B occasionally outperformed RoBERTa on the TIMELINE dataset (covering 2020–2021 events). This is because RoBERTa’s 2019 pre-training makes it "blind" to terms like "COVID-19," whereas newer LLMs have these concepts in their vocabulary.
- Training Objectives: Masked Language Modeling (MLM) appears inherently better suited for capturing relational features than standard autoregressive pre-training.
Performance Benchmarks
- RoBERTa remains the state-of-the-art: It led with Micro-F1 scores of 87.6% on MATRES and 83.1% on TB-Dense.
- The LLM Gap: The best non-fine-tuned LLM (Llama2 70B using QA2) reached only 65.3% on MATRES—a 22-point lag.
- Fine-Tuning Limits: Even a fine-tuned Llama2 13B (84.3% F1) failed to surpass RoBERTa, despite its much larger parameter count and higher compute requirements.
- Class Imbalance: LLMs struggled significantly with rare relations like "Equal" (simultaneous), showing a heavy bias toward "Before" and "After."
Conclusion
The results indicate that for specialized tasks like TRC, bigger is not always better. The structural advantages of encoder-only models—specifically their ability to process full context without recency bias—make them more efficient and effective than current autoregressive LLMs. Future progress likely lies in hybrid architectures that combine the context-aware feature extraction of encoders with the reasoning power of LLMs.
Summary updated