TL;DR
This paper tackles the challenge of training statistical parsers with less manual annotation by exploring "co-training," where two parsers learn from each other's outputs. They introduce and evaluate novel "example selection" methods that balance choosing accurate parses with selecting those that offer the most learning utility for the other parser, leading to better parser performance.
What Problem Does This Paper Address?
State-of-the-art statistical parsers, crucial for understanding sentence structure, typically require massive amounts of manually annotated training data (like the Penn Treebank). Creating this data is expensive and time-consuming. This bottleneck limits the widespread adoption and development of parsers. The paper addresses this by investigating methods to automate the annotation process, specifically focusing on "bootstrapping" techniques that allow parsers to learn from unlabeled data.
Methods
The core of the paper lies in example selection for co-training and corrected co-training frameworks.
- Co-training: This is a semi-supervised learning approach where two parsers, ideally with different "views" of the data, are trained on a small seed set. They then iteratively label unlabeled data for each other. The key challenge is that these machine-generated labels can be noisy.
- Corrected Co-training: A variation where a human annotator reviews and corrects the machine-labeled examples before they are added to the training set. This aims to reduce the human effort required.
- Example Selection Methods: The paper proposes and evaluates three novel selection methods:
- Sabove-n: Selects sentences where the "teacher" parser's score (confidence in its parse) is above a certain threshold
n. This prioritizes accuracy. - Sdiff-n: Selects sentences where the difference in scores between the teacher and "student" parser is above a threshold
n. This prioritizes "training utility" – sentences where the student parser is likely to learn the most from the teacher's (potentially more accurate) parse. - Sint-n: Selects sentences where the teacher's parse is highly scored and the student's parse is poorly scored. This attempts to balance both accuracy and training utility.
The effectiveness of these methods is evaluated using two scoring functions: an "oracle" (perfect knowledge of parse correctness) and a practical "conditional probability" (parser confidence).
Datasets
- Training Data:
- Seed Data: 1,000 sentences from Section 2 of the Penn Treebank (Wall Street Journal corpus).
- Unlabeled Data Pool: Approximately 38,000 sentences from Sections 2-21 of the Penn Treebank (stripped of annotations).
- Test Data: Unseen sentences from Section 23 of the Penn Treebank.
- Development Data: Section 0 of the Penn Treebank was used for parameter tuning.
No new benchmarks were introduced; the study uses existing, standard datasets for parsing evaluation.
Key Insights and Ideas
- Training Utility Matters: The paper strongly suggests that prioritizing "training utility" (selecting sentences that offer the most learning potential for the student parser) is more effective than solely focusing on the accuracy of the teacher's labels, even if it means introducing more errors into the training data.
- Sdiff-n Outperforms: The
Sdiff-nmethod, which focuses on training utility, consistently achieved better parser performance thanSabove-n(accuracy-focused) in co-training experiments. - Balancing Act is Promising: The
Sint-nmethod, aiming to balance accuracy and utility, shows promise, especially in the corrected co-training scenario, suggesting that a combined approach can be beneficial. - Scoring Function Impact: The choice of scoring function significantly impacts performance. While an oracle scoring function provides ideal conditions, practical scoring functions (like conditional probability) lead to degraded performance, highlighting the challenge of relying on parser confidence alone.
- Corrected Co-training Benefits: In the corrected co-training setup, selection methods that consider training utility can reduce the number of sentences a human annotator needs to check, even if they require slightly more constituent corrections.
Key Results
- Co-training:
- Using the oracle scoring function (
fF-score),Sdiff-10%(training utility) achieved the best parser performance, even with a training data accuracy of around 85%.
- Using the oracle scoring function (
- With a stricter control (95% accuracy),
Sdiff-10%still outperformedSabove-90%andSint-30%in the long run. - Using the practical
fprobscoring function,Sdiff-30%andSint-30%showed a 5% improvement over a parser trained only on seed data, whileSabove-70%did not. - Corrected Co-training:
Sdiff-10%trained a better parser thanSabove-90%using fewer training examples.
Sint-30%showed the fastest initial improvement rate but ran out of data before convergence.- In terms of human effort,
Sdiff-10%required fewer sentences to be checked thanSabove-90%for comparable parsing performance.
Limitations and Future Work
- Scoring Function Accuracy: The paper acknowledges that practical scoring functions (like conditional probability) are imperfect estimates of parse accuracy. Developing better scoring functions that more accurately reflect parse correctness is a key area for future work.
- Data Size for Sint-n: The
Sint-nmethod showed promise but was limited by the available unlabeled data. Future work could explore its benefits with much larger unlabeled datasets. - Approximating Agreement: The paper mentions that directly maximizing parser agreement on unlabeled data (as in some theoretical co-training work) is computationally too intensive for parsers. Developing efficient ways to approximate this is a future direction.
- Partial Parses: The authors suggest that using partial parses as training data might make co-training more effective.
- Comparison to Single Learners: The paper focuses on multi-parser co-training. Future work will involve comparing corrected co-training with other active learning methods applied to single learners.
- Human Effort Trade-offs: While
Sdiff-10%reduced the number of sentences to check in corrected co-training, it required slightly more constituent corrections. Further investigation into the trade-offs for human annotators is suggested.
Summary updated