TL;DR
This paper introduces a novel approach to temporal annotation in Chinese text, focusing on how syntactic, semantic, and discourse constraints can improve the accuracy and consistency of identifying events and their temporal relationships. The work highlights the challenges of temporal annotation, especially for complex relations, and proposes a more strategic selection of annotation targets to overcome these hurdles.
What Problem Does This Paper Address?
Temporal inference – understanding when events happen and how they relate to each other in time – is a crucial component of many natural language processing (NLP) tasks, including information extraction, question answering, and machine translation. While significant progress has been made in English, temporal inference in Chinese presents unique challenges. Chinese lacks the rich inflectional morphology that marks tense in English, making it harder to directly infer temporal information. Existing annotation efforts, like those for the TempEval campaigns, aim to create datasets for training and evaluating temporal inference systems. However, achieving high inter-annotator agreement, a key indicator of annotation quality and system reliability, remains a significant challenge, particularly for complex temporal relations. This paper addresses the difficulty of achieving consistent temporal annotation in Chinese by exploring how to better select what to annotate, guided by linguistic constraints.
Methods
The authors describe their work within the TempEval-2 evaluation campaign, which uses a multi-layered annotation approach. The core idea is to break down the complex task of temporal annotation into smaller, more manageable subtasks.
- Event and Time Expression Identification: The initial step involves identifying "event anchors" (verbs or nouns representing events) and time expressions (TIMEX3s).
- Temporal Relation Annotation: This is the more challenging part, focusing on relationships between:
- Events and time expressions.
- Events and the document creation time (DCT).
- Subordinating and subordinated events.
- Main events in adjacent sentences.
- Constraint-Based Annotation Target Selection: A key contribution is the proposed strategy for selecting which event-time pairs to annotate. Instead of annotating all possible pairs, they focus on identifying which events are modified by a given time expression, guided by syntactic scope. For event-event relations, they leverage semantic criteria like predicate-argument structure. This approach aims to reduce ambiguity and improve annotator agreement.
- Leveraging Existing Annotations: The authors utilize the Chinese Treebank and its associated predicate-argument structure annotations (Chinese Propbank) to inform their annotation process, particularly for identifying event-event relations.
Datasets
The dataset used in this experiment consists of 60 files taken from the Chinese Treebank, which are sourced from Xinhua newswire. This dataset is part of the TempEval-2 campaign, a multilingual temporal evaluation initiative. The paper does not introduce a new benchmark dataset but rather describes the annotation process for a subset of existing Chinese text.
Key Insights and Ideas
- Temporal Relation Annotation is Inherently Difficult: The paper strongly emphasizes that while identifying events and time expressions is relatively straightforward, annotating temporal relations between them is significantly more challenging, leading to lower inter-annotator agreement.
- Strategic Selection of Annotation Targets is Crucial: The core insight is that what is annotated is as important as how it's annotated. By using syntactic, semantic, and discourse constraints to guide the selection of annotation targets, the authors aim to improve consistency and reduce ambiguity.
- Syntactic Scope for Event-Time Relations: Focusing on the syntactic scope of time expressions for event-time relation annotation helps reduce "fuzzy" cases and aligns better with annotator intuition.
- Predicate-Argument Structure for Event-Event Relations: Leveraging predicate-argument structures provides a more robust way to identify subordinating and subordinated events, even when syntactic structures change.
- Challenges in Chinese Temporal Inference: The lack of grammatical tense marking in Chinese is highlighted as a significant factor contributing to the difficulty of temporal inference.
- Main Event Identification is Tricky: Even a seemingly simple task like identifying the "main event" in a sentence proves surprisingly difficult, with low inter-annotator agreement, especially in cases of direct quotes or coordination structures.
Key Results
The paper reports on the inter-annotator agreement (IAA) scores for various annotation tasks.
- High Agreement for Simpler Tasks: IAA scores for identifying event anchors (event-extent) and time expressions (timex-extent) are high, generally above 0.90.
- Lower Agreement for Temporal Relations: IAA scores for temporal relations are considerably lower, ranging from 0.53 to 0.75 for different relation types. This confirms the inherent difficulty of these tasks.
- Detailed Agreement Analysis (Table 2): For event-time and subordinating-subordinated event relations, the paper shows that annotators have higher agreement in assigning the temporal relation once a pair is selected, but lower agreement in selecting the pair itself. This reinforces the importance of the annotation target selection strategy.
Limitations and Future Work
- Limitations:
- The current annotation experiment has not yet been annotated with discourse structures, which the authors believe would further improve temporal annotation consistency, especially for inter-sentential relations.
- The reliance on syntactic criteria for identifying main events can lead to unintuitive results, particularly with direct quotes and coordination structures.
- The annotators, while trained, are not necessarily professional linguists, which may contribute to lower agreement on more complex tasks.
- Future Work:
- The authors suggest incorporating discourse structure information into the annotation process, either by annotating it separately or by integrating its key elements into the annotation guidelines.
- Developing more robust methods for identifying main events that account for discourse structure and semantic importance, rather than purely syntactic cues.
- Exploring how to automatically extract event-time and event-event pairs using syntactic and predicate-argument structures as constraints, to potentially aid annotators and improve consistency.
Summary updated