TL;DR
Large Language Models (LLMs) are powerful for coreference resolution but often hallucinate, generating incorrect or extraneous text. This paper introduces "Efficient Constrained Decoding" (ECD), a novel method that significantly reduces hallucinations and improves computational efficiency by guiding LLMs to generate only necessary coreference information, achieving state-of-the-art results on the OntoNotes dataset.
What Problem Does This Paper Address?
Coreference resolution (CR) is a fundamental NLP task that involves identifying mentions referring to the same entity in a text. While LLMs have shown great promise in CR, especially in zero-shot and few-shot settings, a major challenge is "hallucination." This occurs when LLMs generate content not present in the original input, making evaluation difficult and reducing overall performance. Existing methods that try to mitigate this by reproducing the full document often struggle with this hallucination problem, especially with long and complex texts. The paper aims to address this by developing a low-hallucination and computationally efficient CR method for LLMs.
Methods
The paper proposes Efficient Constrained Decoding (ECD) for coreference resolution. This method builds upon the idea of constrained decoding, which guides LLM generation to adhere to specific rules or templates.
Here's how it works:
- Structured Prompting: Similar to previous work, the LLM is given a prompt with mentions marked and placeholders for cluster IDs.
- Template-Guided Generation: The LLM is constrained to generate output that matches a template. This template ensures that the model only fills in the designated cluster ID positions.
- Efficient Multi-Token Decoding: Unlike traditional constrained decoding that operates token-by-token, ECD allows the model to generate multiple tokens in a single step when dealing with template-defined regions. This significantly speeds up the decoding process.
- Low-Hallucination Focus: By strictly guiding the LLM to generate only cluster IDs and copy template text, ECD minimizes the opportunity for the model to hallucinate extraneous content.
The paper also revisits their previous method, Iterative Document Generation, which avoids full-text generation by predicting cluster IDs one mention at a time. While robust, this method is computationally inefficient due to repeated model invocations. ECD offers a more efficient alternative.
Datasets
The primary dataset used for training and evaluation is English OntoNotes 5.0. The authors also used CRAC (including its LIGHT, Penn Treebank, and TRAINS subsets) for further evaluation and to assess cross-domain robustness.
Key Insights and Ideas
- Hallucinations are a critical bottleneck: The paper strongly emphasizes that hallucinations are not just a minor nuisance but a fundamental issue hindering the reliable deployment of LLMs for CR.
- Input complexity exacerbates hallucinations: The authors hypothesize that LLMs struggle with long and structurally complex inputs, leading to attention dilution and a tendency to deviate from the source text.
- Constraining generation is key: Both Iterative Document Generation and Constrained Decoding are presented as effective strategies to mitigate hallucinations by limiting the model's generative freedom.
- Efficiency matters: While robustness is crucial, the paper highlights the need for computationally efficient methods, leading to the development of ECD.
- "Pass" metric can be misleading: The paper points out that alignment checks (like "Pass") can overestimate reliability if they don't consider semantic correctness, as seen in the TRAINS subset.
Key Results
- The proposed Efficient Constrained Decoding (ECD) method, implemented with Llama 3.1 8B, achieved state-of-the-art results on the English OntoNotes development set with a CoNLL F₁ score of 95.1.
- This performance surpasses previous state-of-the-art systems, including coref-T0 (94.8), while using significantly fewer parameters (8B vs. larger models).
- ECD achieved 100% rates in both "Pass" and "EM" (Exact Match) alignment checks on OntoNotes and CRACs datasets, indicating effective elimination of textual alignment hallucinations.
- ECD demonstrated superior computational efficiency, achieving the least inference time compared to other methods, while maintaining comparable training costs to direct decoding.
- The iterative approach, while robust, was found to be substantially more resource-intensive, requiring significantly longer training and inference times.
Limitations and Future Work
- Limited computational resources: The authors could not evaluate their methods on larger-scale language models, though they expect findings to generalize.
- Rare unalignable examples: The direct decoding variant (D) encountered unalignable examples, preventing exact CoNLL F₁ score computation in some cases.
- Special token limitations: While using special tokens for markers can improve robustness, their availability and consistent definition across LLMs remain a challenge.
- Future work: The paper suggests that further gains can be expected with larger model scales. The focus on efficient and low-hallucination methods for LLM-based CR is a promising direction for future research.
Summary updated