TL;DR
This paper introduces a novel self-training approach for pronoun interpretation that achieves performance competitive with state-of-the-art supervised models, despite using no manually annotated training data. This suggests that large-scale, unsupervised learning can be a viable alternative to expensive human annotation for complex NLP tasks.
What Problem Does This Paper Address?
Pronoun interpretation, also known as anaphora resolution, is the task of identifying the noun phrase (antecedent) that a pronoun refers to. Traditionally, building high-performing systems for this task has required large amounts of manually annotated data, which is time-consuming and expensive to create. This paper addresses the challenge of achieving strong pronoun interpretation performance without relying on such annotated datasets, exploring the potential of self-training from raw text.
Methods
The core of the proposed system is a Maximum Entropy (MaxEnt) classifier, a standard machine learning technique for classification tasks. What's novel is how this classifier is trained. Instead of using human-labeled data, the system iteratively trains itself:
- Initial Model Bootstrapping: The algorithm first creates a preliminary training set by identifying pronouns where only one plausible antecedent can be found based on simple heuristics (like gender/number agreement and proximity). For ambiguous cases, it randomly assigns "True" or "False" labels with a certain probability.
- Iterative Refinement: The MaxEnt model is trained on this initial data. Then, the model itself is used to re-label the training data. It assigns "True" to the antecedent it predicts with the highest probability and "False" to others. This process is repeated, retraining the model with the newly labeled data, until the model's predictions stabilize.
- Feature Set: The system utilizes a standard set of 40 "soft features" that capture linguistic information such as gender/number agreement, distance between pronoun and antecedent, grammatical role, and linguistic form of the antecedent. These features are computed using the output of a shallow parser (SRI's TEXTPRO).
Datasets
- Training Data: The self-trained system uses raw text from the newswire segment of the Topic Detection and Tracking (TDT-2) corpus.
- Evaluation Data: The system is evaluated on a blind test set of 762 annotated third-person pronouns from the ACE (Automatic Content Extraction) program's February 2002 evaluation set. These pronouns refer to entities categorized as PERSONS, ORGANIZATIONS, GEOPOLITICALENTITIES, LOCATIONS, and FACILITIES.
Key Insights and Ideas
- Self-Training Efficacy: The most significant insight is that a pronoun interpretation system can be trained effectively using only raw data and a standard set of features, achieving performance remarkably close to supervised systems.
- Noisy Data is Manageable: Despite the inherent noise in self-acquired training data (due to the simplifying assumptions and potential parsing errors), the resulting model is robust enough to select correct antecedents.
- Feature Generality: The success of the self-trained system, even with relatively small amounts of training data, suggests that the chosen feature set is general and powerful enough to capture important linguistic cues for coreference.
- Potential for Scale: The paper posits that with even larger raw corpora and more fine-grained features, self-training could potentially surpass supervised methods, which are limited by the cost of manual annotation.
Key Results
- Hobbs Baseline: 68.8% accuracy.
- Self-Trained System: 73.4% accuracy.
- Supervised System: 75.7% accuracy (trained on the same number of manually annotated pronouns).
The self-trained system outperforms the traditional Hobbs baseline by 4.6% and comes within 2.3% of the supervised system. The iterative training process converged rapidly, typically within 8-14 iterations.
Limitations and Future Work
- Simplifying Assumptions: The self-training algorithm makes a simplifying assumption that each pronoun has exactly one correct antecedent. This is not always true, as multiple antecedents can exist, or antecedents might be misparsed or missed by hard constraints.
- Future Work: The authors suggest exploring variants of the algorithm, such as using model probabilities to dictate the likelihood of positive outcomes rather than a strict winner-take-all approach. They also highlight the potential of combining self-training with a larger set of features and massive raw corpora to potentially surpass supervised models. Further investigation into the nature of the acquired training data is also planned.
Summary updated