Introduction: The Problem of Weight Undertraining
Discriminative models like Conditional Random Fields (CRFs) are powerful tools in Natural Language Processing (NLP). However, they often suffer from a phenomenon known as "weight undertraining." This occurs when a few highly predictive features "drown out" many weaker but useful ones during the optimization process.
When a model is trained, it seeks the path of least resistance to minimize error. If a "strong" feature (e.g., a specific word in a dictionary) is highly indicative of a label, the optimization process assigns it a high weight. Consequently, the model stops learning from "weaker" features, such as character prefixes or surrounding context. This leads to a lack of robustness: if the strong feature is missing or noisy at test time, the model fails because it never properly learned the supporting evidence provided by the weaker features.
Methodology: Feature Bagging
To combat weight undertraining, the authors propose Feature Bagging. Unlike traditional bagging, which creates ensembles based on data subsets, feature bagging trains separate "expert" models on different subsets of features.
The researchers explored four combination strategies for these experts, including per-sequence and per-transition mixtures and products of experts. By isolating features into separate "bags," the model is forced to optimize weights for each subset independently, preventing strong features from overshadowing weaker ones.
Experimental Setup and Results
The method was evaluated on two standard NLP benchmarks: CoNLL 2003 (Named Entity Recognition) and CoNLL 2000 (Noun-Phrase Chunking). Features were manually split into "baseline" (e.g., word identity) and "specialized" (e.g., character n-grams, lexicons) bags.
Feature bagging consistently outperformed standard CRFs trained on the full feature set. On the NER task, the combined CRF achieved an F1 score of 86.61%, a significant improvement over the 85.45% achieved by a single CRF using all features. This represents a substantial 8.32% error reduction.
Key Insights and Conclusion
The success of feature bagging highlights an important optimization bottleneck: standard training algorithms often fail to reach optimal weight configurations when presented with all features at once. Feature bagging acts as a powerful regularizer, ensuring the model remains robust by attending to a wider array of signals.
While manual bagging was used in this study, future work may focus on automated feature splitting and reducing the computational overhead of training multiple models. Overall, feature bagging provides a robust framework for training discriminative models with large, diverse feature sets, achieving state-of-the-art results on complex sequence labeling tasks.
Summary updated