TL;DR
This paper introduces the ARRAU corpus and associated datasets, designed to advance anaphora resolution research by including a wider range of linguistic phenomena beyond simple coreference. The authors present preliminary results from a shared task using these datasets, offering a baseline for future research in this complex area of natural language understanding.
What Problem Does This Paper Address?
Understanding how language refers to entities and concepts (anaphora resolution) is crucial for building truly intelligent machines. While existing resources like the ONTONOTES corpus have been valuable, they have limitations. They often exclude important linguistic phenomena such as "bridging references" (e.g., "the engine" referring to a car previously mentioned) and "discourse deixis" (referring to abstract concepts not explicitly introduced by a noun phrase). Furthermore, ONTONOTES doesn't always treat all potential referring expressions (like expletives or singletons) as "markables" (the units of interest for annotation). This paper addresses the need for a more comprehensive resource that captures these broader aspects of anaphora, enabling more robust and nuanced language understanding systems.
Methods
The core contribution is the ARRAU corpus itself, which has been meticulously annotated to overcome the limitations of previous resources. Key methodological aspects include:
- Comprehensive Markable Definition: Unlike previous corpora, ARRAU treats all noun phrases (NPs) as markables, including expletives (e.g., "it" in "it is raining") and singletons (NPs that don't corefer with anything else). Non-referring markables are further sub-classified.
- Rich Anaphoric Relation Annotation: ARRAU annotates not only identity (coreference) but also:
- Bridging References: Capturing associative relationships between an anaphor and its antecedent.
- Discourse Deixis: Annotating references to abstract entities not directly introduced by nominal markables.
- Dataset Creation for CRAC 2018: The authors extracted three specific datasets from ARRAU to support the CRAC 2018 Shared Task, focusing on identity anaphora, bridging references, and discourse deixis.
- Evaluation Scripts: Customized evaluation scripts were developed to assess system performance on these tasks, supporting both gold-standard and predicted markables.
Datasets
The primary dataset discussed is the ARRAU corpus. It comprises four sub-corpora: RST (news text), TRAINS (task-oriented dialogues), PEAR (spoken narratives), and GNOME (medical and art history documents). For the CRAC 2018 Shared Task, three specific datasets were extracted from ARRAU, targeting:
- Identity Anaphora Resolution: Focusing on whether a markable is referring, discourse-new, or discourse-old, and identifying its antecedent.
- Bridging Reference Resolution: Identifying anchor entities and markables that bridging references refer to.
- Discourse Deixis: Identifying the unit (e.g., utterance or clause) that a discourse deixis refers to.
These datasets are designed to be more comprehensive than previous benchmarks by including a wider range of anaphoric phenomena and markable types.
Key Insights and Ideas
- Bridging and Discourse Deixis are Crucial: The paper strongly advocates for the importance of annotating and resolving bridging references and discourse deixis, arguing that they are essential for a complete understanding of how language refers.
- ARRAU Offers a More Inclusive View: By treating all NPs as markables, ARRAU provides a more comprehensive view of potential referential expressions, which is critical for developing robust systems.
- Markable Extraction is Easier in ARRAU: Preliminary results suggest that extracting markables from ARRAU is significantly easier than from ONTONOTES, likely due to ARRAU's broader definition of markables (including singletons and non-referring NPs).
- Domain Adaptation Matters: Experiments suggest that domain-specific training can significantly impact anaphora resolution performance, highlighting the need for diverse corpora like ARRAU.
Key Results
The paper presents preliminary results from the CRAC 2018 Shared Task, serving as a baseline.
- Markable Extraction: Systems achieved higher F1 scores for markable extraction on ARRAU (e.g., ~82.6% for an LSTM-based system on RST) compared to ONTONOTES (e.g., ~55.65% for CoreNLP on ONTONOTES, excluding singletons and non-referring NPs).
- Identity Anaphora (Task 1): Baseline results using the Stanford deterministic coreference resolver on the ARRAU RST subset showed a CONLL score of approximately 43.2% on predicted markables (excluding singletons and non-referring NPs), which is lower than typical scores on the CONLL 2012 dataset, suggesting the task's complexity or differences in evaluation. Including singletons improved scores significantly.
- Bridging Anaphora (Task 2): A rule-based and ML-based system from the University of Stuttgart participated. Results were comparable to previous work on similar corpora for RST and TRAINS domains but lower for the PEAR domain, though the small number of bridging references in PEAR limits strong conclusions.
- Discourse Deixis (Task 3): No systems participated in this task, indicating its nascent stage of research.
Limitations and Future Work
- Limited Participation in CRAC 2018: Notably, no systems participated in Task 1 (identity anaphora) and Task 3 (discourse deixis) of the CRAC 2018 Shared Task. This suggests that research in these specific areas, especially discourse deixis, is still developing.
- Nascent Research Areas: The paper acknowledges that research in bridging reference and discourse deixis is still limited, and the ARRAU corpus aims to facilitate this research.
- Future Work: The authors hope that making the ARRAU datasets available will encourage more researchers to work on these aspects of anaphora resolution, increasing the community and advancing the state of the art. The preliminary results are intended to serve as a baseline for future research.
Summary updated