TL;DR
This paper introduces novel methods, Linear Sentence Linking (LSL) and Attentional Sentence Linking (ASL), to improve co-reference resolution by enabling neural models to learn dependencies between words across different sentences. By incorporating context from neighboring sentences, these approaches significantly boost the performance of end-to-end co-reference resolution systems.
What Problem Does This Paper Address?
Co-reference resolution, the task of identifying mentions that refer to the same real-world entity, is crucial for understanding text. Traditional neural models for this task often process sentences independently, treating them as separate batches. This approach, while efficient, fails to capture vital cross-sentence dependencies. For instance, pronouns often refer to entities mentioned in preceding sentences, and without considering this context, their representations are incomplete, hindering accurate co-reference prediction. Simply concatenating all sentences into one long sequence is computationally prohibitive and prone to vanishing/exploding gradients. This paper addresses this gap by proposing methods to effectively integrate information from context sentences into word representations without incurring prohibitive computational costs.
Methods
The authors propose two novel methods to learn cross-sentence dependency:
- Linear Sentence Linking (LSL): This is a direct approach where the initial states of a bidirectional LSTM are initialized with the encoded representations of adjacent sentences. Essentially, the model "looks back" at the previous sentence's output to inform the current sentence's processing.
- Attentional Sentence Linking (ASL): This method builds upon LSL by incorporating an attention mechanism and a context gate. It treats the previous sentence's word embeddings as an external memory module. An attention mechanism calculates a weighted summary of this memory, and a context gate dynamically decides how much to rely on this cross-sentence information versus the current sentence's input. This allows the model to selectively leverage relevant context.
Both methods aim to enrich word embeddings by considering information beyond the current sentence, feeding these enhanced representations into an existing end-to-end co-reference resolution model.
Datasets
The models were trained and evaluated on the English corpus of the CoNLL-2012 shared task.
Key Insights and Ideas
- Cross-sentence dependency is crucial: The paper strongly demonstrates that ignoring context from other sentences significantly limits the performance of co-reference resolution models, especially for pronouns.
- Attention is better than simple linking: The ASL model, which uses an attention mechanism to selectively draw information from context sentences, outperforms the simpler LSL model. This highlights the benefit of a more nuanced approach to integrating cross-sentence information.
- Context gate for dynamic integration: The context gate in ASL allows the model to adaptively decide how much to rely on external context, making the representation learning more robust.
- Improved pronoun resolution: The qualitative examples show that ASL is better at resolving pronouns whose antecedents are in different sentences, a common challenge for models that process sentences in isolation.
Key Results
The proposed models, particularly ASL, significantly improved the performance of the state-of-the-art end-to-end co-reference resolution model.
- The baseline model achieved an average F1 score of 67.2%.
- The E2E-CR + ASL model achieved an average F1 score of 67.8%, a 0.6% improvement.
- The E2E-CR + LSL model achieved an average F1 score of 67.4%.
These improvements, while seemingly small, are substantial in the context of co-reference resolution benchmarks. The paper also notes that ASL outperformed LSL, indicating the effectiveness of the attention mechanism.
Limitations and Future Work
- Computational Cost: While the proposed methods are more efficient than concatenating entire documents, they still introduce additional computation compared to processing sentences entirely independently.
- Generalizability: The authors suggest exploring whether their model can improve performance on other natural language processing tasks that involve multi-sentence inputs, such as reading comprehension, dialog generation, and sentiment analysis. This indicates a potential for broader applicability beyond co-reference resolution.
Summary updated