Abstract
Jon Cai, Kristin Wright-Bettner, Zekun Zhao, Shafiuddin Rehan Ahmed, Abijith Trichur Ramachandran, Jeffrey Flanigan, Martha Palmer, James Martin. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).
2025.
LiDARR is a new web-based annotation tool designed to scale Abstract Meaning Representation (AMR) from single sentences to full documents. By integrating AI "copilots" for coreference resolution and text-to-graph alignment, it transforms the grueling task of building document-level knowledge graphs into a streamlined process of model-assisted verification.
Introduction: The Challenge of Document-Level AMR
Abstract Meaning Representation (AMR) has become a standard for capturing semantic structures—answering "who did what to whom"—within a single sentence. However, when moving from individual sentences to full documents, the semantic representation often breaks down into isolated islands. To create a cohesive "Document AMR," researchers must resolve coreferences, linking entities across sentence boundaries (e.g., connecting "the boy" in sentence one to "him" in sentence two).
Existing annotation tools for this task are often unintuitive, frequently requiring users to edit raw PENMAN code or manually link pairs of entities, a process that is both time-consuming and cognitively exhausting. Furthermore, document-level understanding requires handling implicit roles—arguments that are understood by the reader but not explicitly stated in a specific sentence. LiDARR aims to bridge these gaps with a user-friendly, AI-augmented interface that treats annotation as a verification task rather than a construction task.
The LiDARR System Architecture
LiDARR employs a modular architecture featuring three parallel panels designed to synchronize different views of the document data:
- Text Panel: Displays the raw source text with highlighted mentions.
- AMR Panel: Visualizes the graph structure and hierarchical semantic relations.
- Cluster Panel: Organizes coreferent mentions into manageable entity groups.
AI "Copilot" Support
The system’s primary innovation is its suite of integrated AI assistants:
- Alignment Models: Utilizing the LEAMR aligner, the tool automatically maps surface text tokens to specific nodes in the AMR graph.
- Coreference Resolution: State-of-the-art models like FastCoref and LingMess suggest entity clusters across the entire document.
- Implicit Role Filling: Using PropBank rolesets, LiDARR identifies missing arguments for predicates. This allows users to link a pronoun not just to another word, but to an unstated concept in the graph. For example, if a pilot "flies" in sentence one and "arrives in New York" in sentence two, the user can link "New York" to the implicit destination role of the "fly" predicate in the first sentence.
- LLM Interpreter: An optional GPT-4o integration provides natural language justifications for the AI's clustering logic, helping human annotators quickly identify potential model errors or "hallucinations."
Key Insights: Annotation as Correction
LiDARR shifts the annotation paradigm from "building a graph" to "correcting a model." By providing high-quality initial suggestions for alignments and clusters, the human workload shifts toward verification, which is significantly faster than manual entry.
A key finding from the authors' study is the effectiveness of the "Interactive Helper" mode. In this mode, the AI suggests the most likely next token for a cluster, guiding the annotator through the document. User studies indicated that experts specifically preferred this "lead-and-follow" relationship over static suggestions, finding it more effective for complex linguistic tasks.
Evaluation and Results
The authors evaluated LiDARR using the MS-AMR (Multi-Sentence AMR) corpus. Key findings include:
- Model Performance: When transferring surface-text coreference models to the AMR domain, the LingMess model outperformed FastCoref, achieving a CEAF_E F1 score of 44.22%.
- User Preference: In comparative testing, the "Interactive Helper" received a preference score of 11 points, nearly double the score of building from scratch (6 points) and significantly higher than using a static helper (7 points).
- Efficiency: The underlying coreference models proved highly efficient, completing clustering for long documents in under one second, ensuring a responsive user experience.
Limitations and Future Work
While LiDARR represents a significant step forward, the authors acknowledge several areas for improvement. Currently, the tool is optimized for English and may struggle with the nuances of other languages. It also faces challenges with "discourse deixis," where pronouns refer to entire previous events or paragraphs rather than single entities. Additionally, the use of LLM-based interpreters requires data to be sent to external APIs, which may be a concern for sensitive documents.
Future development will focus on refining the UI for "bridging relations" (linking related but non-identical entities, such as "the car" and "the engine") and exploring the tool's utility in educational settings to provide verifiable knowledge support.
Summary updated