Introduction
In "pro-drop" languages such as Chinese, pronouns are frequently omitted when the context provides sufficient clarity. These omitted elements, known as Zero Pronouns (ZP), present a significant challenge for natural language processing. Zero Pronoun Resolution (ZPR) is the task of identifying the correct "antecedent" for these gaps from a set of candidate noun phrases (NP) in the surrounding text.
Traditional deep learning approaches to ZPR often suffer from two primary limitations. First, they typically utilize unidirectional attention, focusing only on how the ZP context relates to the candidate, thereby missing the bidirectional interaction necessary to refine both representations. Second, they treat ZPR as a binary classification problem, failing to account for the relative ranking between candidates or the fact that multiple candidates may belong to the same coreference chain.
This paper introduces HAN-PL, a Hierarchical Attention Network with Pairwise Loss, designed to address these shortcomings. By fostering bidirectional interaction and optimizing for relative ranking, HAN-PL achieves a new state-of-the-art performance on the standard benchmark.
The HAN-PL Framework
The HAN-PL model consists of two core innovations: a hierarchical attention mechanism for feature encoding and a pairwise loss function for training.
Hierarchical Attention Network (HAN)
The HAN component moves beyond independent encoding by implementing a two-layer attention structure:
- Bidirectional Attention Layer: This layer enables the model to learn interactive representations. It identifies which words in a candidate's context are most relevant to the ZP's context and vice versa, capturing the inherent dependency between the two.
- Self-Attention Layer: Following the bidirectional interaction, a self-attention layer further refines the features by highlighting the most informative parts of the learned representations before they reach the final resolution layer.
Pairwise Loss (PL)
To move away from the "classification trap," the authors propose a Pairwise Loss (PL) function. This approach is more robust for the imbalanced nature of ZPR data, where incorrect candidates significantly outnumber correct ones. The loss function is defined by two primary constraints:
- Pairwise-Margin Loss: This ensures that the resolution score for a correct antecedent exceeds that of an incorrect one by at least a predefined margin ($m$).
- Correct-Antecedent Similarity Constraint: Recognizing that multiple mentions may refer to the same entity (coreference chains), this constraint encourages the representations of all correct antecedents for a single ZP to be similar to one another.
Experimental Results
The model was evaluated on the Chinese portion of the OntoNotes 5.0 dataset, covering diverse sources such as Broadcast News (BN), Newswire (NW), and Web Blogs (WB).
HAN-PL achieved an overall F-score of 60.2%, representing a significant 2.9% improvement over previous state-of-the-art models. The model showed particular strength in structured sources like Broadcast News (+5.6%) and Newswire (+2.4%). Ablation studies confirmed the importance of the model's components: removing the self-attention layer resulted in a 1.6% performance drop, while replacing the pairwise-margin loss with standard cross-entropy led to a sharp 3.7% decrease in F-score.
Discussion and Conclusion
The success of HAN-PL highlights the importance of bidirectional interaction and ranking-based optimization in ZPR. By treating the zero pronoun as a context-dependent entity and focusing on the relative likelihood of candidates, the model achieves superior resolution accuracy.
However, challenges remain in informal or short-form texts. Performance was lower in Magazines (MZ) and Web Blogs (WB) due to limited contextual information. Additionally, Telephone Conversations (TC) presented difficulties due to conversational noise and filler words. Future work may focus on integrating noise-filtering mechanisms or leveraging global coreference information more explicitly to improve resolution in these challenging domains.
Summary updated