Only One Relation Possible? Modeling the Ambiguity in Event Temporal Relation Extraction

Yutong Hu, Quzhe Huang, Yansong FengarXiv 2024 · DOI · arXiv

Introduction

Event Temporal Relation Extraction (ETRE) is a fundamental task in natural language processing that involves determining the chronological order of events mentioned in text—identifying whether one event occurred before, after, or during another. A persistent challenge in this field is the prevalence of the "Vague" label. In most benchmark datasets, a pair of events is labeled as Vague when the text lacks sufficient information for annotators to reach a consensus on a single specific relation.

Traditional modeling approaches treat "Vague" as a mutually exclusive, atomic category. However, this paper argues that such a treatment is fundamentally flawed. Ambiguity is rarely a total lack of information; rather, it is often a composition of multiple plausible relations. By forcing models to treat Vague as a distinct class, standard training regimes penalize models for detecting valid linguistic clues that point toward specific relations, thereby creating "confusion" in the learning process and ignoring rich latent information.

The METRE Framework

To address these limitations, the authors introduce METRE (Multi-label Event Temporal Relation Extraction). This framework shifts the paradigm from single-label classification to a multi-label estimation problem, where the model estimates the probability of every possible relation independently.

Multi-label Classification and Thresholding

Instead of applying a softmax function over all classes, METRE utilizes independent probabilities for each well-defined relation. The model employs a learnable, adaptive threshold ($T$) to determine which relations are "active." If the model's confidence for multiple relations exceeds this threshold, the resulting state is interpreted as Vague. This allows the model to maintain the possibility of multiple overlapping relations rather than being forced to choose a single, potentially incorrect, label.

Confusion Set (CS) Speculation

Because Vague instances in existing datasets do not come with ground-truth sub-labels, METRE employs a "speculation mechanism" during training to uncover latent relations. It constructs a "Confusion Set" for Vague instances based on two primary signals:

  1. Model Confidence (Top2): The two relations the model currently identifies as most probable.
  2. Linguistic Mapping: Based on the theory of endpoint uncertainty, the authors mapped specific relations to their likely "confusion counterparts." For example, a "Before" relation is frequently confused with "Include" when the end-point of an event is unclear.

By rewarding the model for identifying the relations within this speculated Confusion Set, the dynamic training objective allows METRE to extract signal from what was previously considered noise.

Experimental Evaluation

The model was evaluated across three major benchmarks representing different levels of ambiguity:

  • TB-Dense: A dataset characterized by high density, where nearly 50% of instances are labeled Vague.
  • MATRES: A dataset focusing specifically on event start-points, featuring a lower Vague proportion (~12%).
  • UDS-T: A dataset derived from Universal Decompositional Semantics, re-processed to define Vague labels based on actual annotator disagreement.

Key Findings

State-of-the-Art Performance

METRE achieved state-of-the-art results on the TB-Dense benchmark, reaching a 67.9% F1 score using a BERT-Base backbone. This represents a 2.6% F1 improvement over strong single-label baselines, demonstrating that modeling the internal structure of ambiguity leads to better overall classification accuracy.

Efficiency in Data-Scarce Scenarios

One of the most significant advantages of METRE is its performance in low-resource settings. Because the model can learn from the latent information hidden within Vague samples, it is far more efficient than traditional models when training data is limited. In few-shot scenarios (using 5% to 50% of available data), METRE outperformed baselines by an average of 7.1% F1.

Interpretability

Unlike "black box" models that simply output a Vague label, METRE provides a degree of interpretability. It can explain why a relation is considered ambiguous by surfacing the specific relations that crossed the activation threshold (e.g., "Vague because the evidence supports both Before and Include").

Conclusion and Future Work

METRE demonstrates that "Vague" should be treated as a state of uncertainty among multiple relations rather than a standalone category. By unboxing the ambiguity in temporal relations, the model not only improves performance but also provides a more nuanced understanding of how events relate to one another in natural language.

Future research directions include refining the "Confusion Relation" logic for datasets like MATRES that focus only on start-points, and developing more dynamic, learned methods for identifying the latent composition of ambiguous instances beyond current heuristic-based speculation.

Summary updated

Abstract

Event Temporal Relation Extraction (ETRE) aims to identify the temporal relationship between two events, which plays an important role in natural language understanding. Most previous works follow a single-label classification style, classifying an event pair into either a specific temporal relation (e.g., \textit{Before}, \textit{After}), or a special label \textit{Vague} when there may be multiple possible temporal relations between the pair. In our work, instead of directly making predictions on \textit{Vague}, we propose a multi-label classification solution for ETRE (METRE) to infer the possibility of each temporal relation independently, where we treat \textit{Vague} as the cases when there is more than one possible relation between two events. We design a speculation mechanism to explore the possible relations hidden behind \textit{Vague}, which enables the latent information to be used efficiently. Experiments on TB-Dense, MATRES and UDS-T show that our method can effectively utilize the \textit{Vague} instances to improve the recognition for specific temporal relations and outperforms most state-of-the-art methods.
BibTeX
@misc{hu2024only,
  author = {Yutong Hu and Quzhe Huang and Yansong Feng},
  title = {{Only One Relation Possible? Modeling the Ambiguity in Event Temporal Relation Extraction}},
  year = {2024},
  eprint = {2408.07353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.07353},
}
Figure