Reducing Weight Undertraining in Structured Discriminative Learning

Charles Sutton, Michael Sindelar, Andrew McCallum

Introduction: The Problem of Weight Undertraining

Discriminative models like Conditional Random Fields (CRFs) are powerful tools in Natural Language Processing (NLP). However, they often suffer from a phenomenon known as "weight undertraining." This occurs when a few highly predictive features "drown out" many weaker but useful ones during the optimization process.

When a model is trained, it seeks the path of least resistance to minimize error. If a "strong" feature (e.g., a specific word in a dictionary) is highly indicative of a label, the optimization process assigns it a high weight. Consequently, the model stops learning from "weaker" features, such as character prefixes or surrounding context. This leads to a lack of robustness: if the strong feature is missing or noisy at test time, the model fails because it never properly learned the supporting evidence provided by the weaker features.

Figure 1: Accuracy vs. Alpha illustrating the impact of weight undertraining when a strong feature is removed.

Methodology: Feature Bagging

To combat weight undertraining, the authors propose Feature Bagging. Unlike traditional bagging, which creates ensembles based on data subsets, feature bagging trains separate "expert" models on different subsets of features.

The researchers explored four combination strategies for these experts, including per-sequence and per-transition mixtures and products of experts. By isolating features into separate "bags," the model is forced to optimize weights for each subset independently, preventing strong features from overshadowing weaker ones.

Experimental Setup and Results

The method was evaluated on two standard NLP benchmarks: CoNLL 2003 (Named Entity Recognition) and CoNLL 2000 (Noun-Phrase Chunking). Features were manually split into "baseline" (e.g., word identity) and "specialized" (e.g., character n-grams, lexicons) bags.

Feature bagging consistently outperformed standard CRFs trained on the full feature set. On the NER task, the combined CRF achieved an F1 score of 86.61%, a significant improvement over the 85.45% achieved by a single CRF using all features. This represents a substantial 8.32% error reduction.

Table 1: F1 scores for the NER task comparing single CRF baselines with the combined model.

Key Insights and Conclusion

The success of feature bagging highlights an important optimization bottleneck: standard training algorithms often fail to reach optimal weight configurations when presented with all features at once. Feature bagging acts as a powerful regularizer, ensuring the model remains robust by attending to a wider array of signals.

While manual bagging was used in this study, future work may focus on automated feature splitting and reducing the computational overhead of training multiple models. Overall, feature bagging provides a robust framework for training discriminative models with large, diverse feature sets, achieving state-of-the-art results on complex sequence labeling tasks.

Summary updated

Abstract

Discriminative probabilistic models are very popular in NLP because of the latitude they afford in designing features. But training involves complex trade-offs among weights, which can be dangerous: a few highly-indicative features can swamp the contribution of many individually weaker features, causing their weights to be undertrained. Such a model is less robust, for the highly-indicative features may be noisy or missing in the test data. To ameliorate this weight undertraining, we introduce several new feature bagging methods, in which separate models are trained on subsets of the original features, and combined using a mixture model or a product of experts. These methods include the logarithmic opinion pools used by Smith et al. (2005). We evaluate feature bagging on linear-chain conditional random fields for two natural-language tasks. On both tasks, the feature-bagged CRF performs better than simply training a single CRF on all the features.
BibTeX
@inproceedings{sutton2006reducing,
  author = {Charles Sutton and Michael Sindelar and Andrew McCallum},
  title = {{Reducing Weight Undertraining in Structured Discriminative Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure