Abstract
In this paper, we propose a data-intensive approach for inferring sentence-internal temporal relations, a task critical for NLP applications such as multi-document summarization and question answering. Our method bypasses the need for expensive manual coding by exploiting overt temporal markers (e.g., "after", "while") as proxy labels in large, unannotated corpora. We evaluate our models on a pseudo-disambiguation task and compare their performance against both the TimeBank corpus and human subjects. Our results demonstrate that models trained on noisy, approximate data can achieve performance levels comparable to human agreement, suggesting a viable path for the semi-automatic creation of temporal annotations.
Introduction
Understanding the chronological order of events is a fundamental component of natural language understanding. While humans easily infer that "John kissed the girl he met at a party" implies the meeting preceded the kiss, teaching machines to resolve such implicit temporal links remains a significant challenge.
The primary obstacle in this field is the data bottleneck. Existing human-annotated datasets, such as TimeBank, are often too small to train robust machine learning models. Furthermore, manual temporal annotation is notoriously difficult; human annotators frequently struggle to agree on complex links, leading to low recall. This research addresses these issues by leveraging the abundance of "raw" text, using explicit linguistic markers as a supervision signal to learn the underlying logic of temporal relations.
Methodology
The Proxy Label Approach
The core innovation of this study is the use of temporal markers as self-contained labels. By extracting sentences containing explicit markers (e.g., "The results were announced after the market closed"), we can remove the marker and task a model with predicting it back. This "pseudo-disambiguation" task allows us to utilize massive datasets without manual intervention.
Experimental Tasks
We define two primary tasks for our models:
- Interpretation: Given two clauses, the model must identify the missing temporal marker from a set of candidates.
- Fusion: Given two clauses and a marker, the model must determine the correct syntactic structure—specifically, which clause is the "main" event and which is the "subordinate" event.
Model Architectures and Features
We compared several probabilistic models to determine the most effective way to represent temporal information:
- Conjunctive vs. Disjunctive Models: Conjunctive models analyze pairs of features across both clauses. While theoretically superior for capturing interactions, they often suffer from data sparsity. Disjunctive models, which treat clause features as independent, proved more robust in practice.
- Linguistic Abstraction: To improve generalization, we utilized "Temporal Signatures" (comprising tense, aspect, and voice) and semantic classes for verbs and nouns derived from WordNet and Levin classes. This allows the model to learn that "reporting" an event generally follows the event itself, regardless of the specific verb used.
- Syntactic Signatures: For the Fusion task, we incorporated features representing the complexity and structure of the parse tree.
- Ensemble Learning: We employed a "stacking" approach using a C5.0 decision tree to combine predictions from multiple sub-models, allowing the system to weigh different linguistic cues dynamically.
Datasets
Our primary training data was the BLLIP Corpus, a 30-million-word collection of Wall Street Journal text. From this, we extracted approximately 83,810 instances of sentence-internal temporal relations. For benchmarking, we used a subset of 40 pairs evaluated by 198 human volunteers and compared our findings against the gold-standard TimeBank corpus.
Results and Discussion
Task Performance
The ensemble model demonstrated significant improvements over baseline approaches. In the Interpretation Task, the model achieved 70.6% accuracy (69.1% F-score), vastly outperforming the word-based baseline of 48.2%. In the Fusion Task, the model was nearly perfect, achieving 97.3% accuracy, indicating that syntactic structure is a highly reliable signal for clause ordering.
Human-Level Performance
To assess the real-world utility of the model, we compared its performance to human agreement using the Kappa scale (which accounts for chance agreement). The ensemble model achieved a Kappa score of .413, while human subjects achieved .421. This suggests that the model is capturing linguistic nuances and ambiguities at a level nearly identical to that of human readers.
Key Insights
Our experiments revealed that linguistic abstraction—moving from raw words to verb classes—is essential for generalization. Furthermore, we found that simpler disjunctive models often outperform complex conjunctive ones when data for specific word combinations is sparse. Finally, the high accuracy in the Fusion task suggests that machines are particularly adept at identifying the "shape" of subordinate clauses relative to main clauses.
Conclusion and Future Work
This research demonstrates that sentence-internal temporal relations can be learned effectively from unannotated data using markers as proxies. While our current focus is limited to relations within a single sentence, the results provide a strong foundation for future work.
Future iterations will aim to address marker ambiguity (e.g., the temporal vs. causal use of "since") and extend the model to extra-sentential relations by incorporating broader paragraph context. Ultimately, this methodology could be used to "pre-annotate" large-scale corpora, significantly accelerating the development of high-quality temporal datasets for the broader NLP community.
Summary updated