Introduction
Supervised learning in Natural Language Processing (NLP) is often bottlenecked by the high cost of manual annotation. This is particularly true for complex linguistic frameworks like Head-Driven Phrase Structure Grammar (HPSG). Unlike simple text classification, HPSG parsing requires expert linguists to select the correct structure from a vast space of potential parses—often dozens or hundreds for a single sentence.
Active Learning (AL) offers a solution by prioritizing the most informative examples for labeling. However, traditional AL focuses on single-model uncertainty. This paper investigates whether an ensemble of models can more effectively guide the learning process to reduce annotation effort.
Methodology
The researchers explored several strategies to optimize annotation efficiency:
- Ensemble-based Uncertainty Sampling: Instead of relying on a single model's entropy, the authors utilized a "Product Model" ensemble. This ensemble combines probabilities from three distinct models trained on different feature sets (configurational, ngram, and conglomerate).
- Lowest Best Probability (LBP): A simplified AL method that selects sentences where the model's top-ranked choice has the lowest absolute probability, indicating low confidence.
- Discriminant Cost: Moving beyond "unit cost" (per sentence), the authors introduced "Discriminant Cost." This metric measures the actual number of binary decisions an annotator must make to identify the correct parse, providing a more realistic assessment of human effort in complex grammar selection.
Experimental Setup
The study utilized the Redwoods Treebank, based on the HPSG English Resource Grammar (ERG). The dataset consists of 5,302 sentences with an average of 58.0 possible parses per sentence, highlighting the significant cognitive load placed on human annotators.
Results and Discussion
The experimental results demonstrate that ensemble methods significantly outperform single-model approaches in both accuracy and efficiency.
Key Findings:
- 73% Cost Reduction: The best ensemble-based AL method reached a high level of accuracy with 73% less effort than a single model using random sampling.
- Ensemble Superiority: A striking observation was that an ensemble model (LL-PROD) trained on randomly sampled data often outperformed a single model (LL-CONFIG) using sophisticated active learning. This suggests that the underlying model architecture is as critical as the data selection strategy.
- Efficiency Gains: To reach 77% accuracy, the Product Uncertainty Sampling method required only 6,410 discriminant decisions, whereas random sampling required 13,800—effectively doubling annotator productivity.
- Simplicity vs. Complexity: The new LBP method performed nearly as well as more mathematically complex entropy-based methods, suggesting that simple confidence metrics can yield substantial gains in practical engineering applications.
Conclusion
By combining ensemble learning with active learning and realistic cost metrics, the researchers demonstrated a path toward significantly more efficient linguistic annotation. The use of a "committee" of models to identify informative examples allows for state-of-the-art parsing performance with a fraction of the manual labor typically required for complex grammars like HPSG.
Summary updated