Introduction
Event Temporal Relation Extraction (ETRE) is a fundamental task in natural language processing that involves determining the chronological order of events mentioned in text—identifying whether one event occurred before, after, or during another. A persistent challenge in this field is the prevalence of the "Vague" label. In most benchmark datasets, a pair of events is labeled as Vague when the text lacks sufficient information for annotators to reach a consensus on a single specific relation.
Traditional modeling approaches treat "Vague" as a mutually exclusive, atomic category. However, this paper argues that such a treatment is fundamentally flawed. Ambiguity is rarely a total lack of information; rather, it is often a composition of multiple plausible relations. By forcing models to treat Vague as a distinct class, standard training regimes penalize models for detecting valid linguistic clues that point toward specific relations, thereby creating "confusion" in the learning process and ignoring rich latent information.
The METRE Framework
To address these limitations, the authors introduce METRE (Multi-label Event Temporal Relation Extraction). This framework shifts the paradigm from single-label classification to a multi-label estimation problem, where the model estimates the probability of every possible relation independently.
Multi-label Classification and Thresholding
Instead of applying a softmax function over all classes, METRE utilizes independent probabilities for each well-defined relation. The model employs a learnable, adaptive threshold ($T$) to determine which relations are "active." If the model's confidence for multiple relations exceeds this threshold, the resulting state is interpreted as Vague. This allows the model to maintain the possibility of multiple overlapping relations rather than being forced to choose a single, potentially incorrect, label.
Confusion Set (CS) Speculation
Because Vague instances in existing datasets do not come with ground-truth sub-labels, METRE employs a "speculation mechanism" during training to uncover latent relations. It constructs a "Confusion Set" for Vague instances based on two primary signals:
- Model Confidence (Top2): The two relations the model currently identifies as most probable.
- Linguistic Mapping: Based on the theory of endpoint uncertainty, the authors mapped specific relations to their likely "confusion counterparts." For example, a "Before" relation is frequently confused with "Include" when the end-point of an event is unclear.
By rewarding the model for identifying the relations within this speculated Confusion Set, the dynamic training objective allows METRE to extract signal from what was previously considered noise.
Experimental Evaluation
The model was evaluated across three major benchmarks representing different levels of ambiguity:
- TB-Dense: A dataset characterized by high density, where nearly 50% of instances are labeled Vague.
- MATRES: A dataset focusing specifically on event start-points, featuring a lower Vague proportion (~12%).
- UDS-T: A dataset derived from Universal Decompositional Semantics, re-processed to define Vague labels based on actual annotator disagreement.
Key Findings
State-of-the-Art Performance
METRE achieved state-of-the-art results on the TB-Dense benchmark, reaching a 67.9% F1 score using a BERT-Base backbone. This represents a 2.6% F1 improvement over strong single-label baselines, demonstrating that modeling the internal structure of ambiguity leads to better overall classification accuracy.
Efficiency in Data-Scarce Scenarios
One of the most significant advantages of METRE is its performance in low-resource settings. Because the model can learn from the latent information hidden within Vague samples, it is far more efficient than traditional models when training data is limited. In few-shot scenarios (using 5% to 50% of available data), METRE outperformed baselines by an average of 7.1% F1.
Interpretability
Unlike "black box" models that simply output a Vague label, METRE provides a degree of interpretability. It can explain why a relation is considered ambiguous by surfacing the specific relations that crossed the activation threshold (e.g., "Vague because the evidence supports both Before and Include").
Conclusion and Future Work
METRE demonstrates that "Vague" should be treated as a state of uncertainty among multiple relations rather than a standalone category. By unboxing the ambiguity in temporal relations, the model not only improves performance but also provides a more nuanced understanding of how events relate to one another in natural language.
Future research directions include refining the "Confusion Relation" logic for datasets like MATRES that focus only on start-points, and developing more dynamic, learned methods for identifying the latent composition of ambiguous instances beyond current heuristic-based speculation.
Summary updated