Abstract
Named Entity Recognition (NER) systems are typically optimized for overall F1-score, which may not align with specific user requirements for high precision or high recall. We present a simple "extractor tweaking" method for sequential classifiers like Conditional Random Fields (CRFs) and Hidden Markov Models (HMMs). By adjusting the weight of the "Outside" label, users can precisely navigate the precision-recall trade-off. Evaluation across three corpora demonstrates that this single-parameter adjustment effectively generates smooth performance curves and allows for targeted optimization of user-defined metrics without retraining the entire model.
Introduction
Named Entity Recognition (NER)—the task of identifying names, organizations, and locations in text—is rarely perfect. Standard models like CRFs are typically trained to maximize the overall likelihood of the data. While this often results in a high F1 score, it does not account for varying user preferences. In the real world, different tasks have different requirements: fact extraction for database population requires high precision, whereas anonymizing medical records or providing candidates for human review requires high recall. Standard sequential models lack an inherent "volume knob" to adjust these trade-offs, as simple thresholding is mathematically complex in structured prediction and often only allows for decreasing recall.
Methodology: Extractor Tweaking
The proposed approach involves directly manipulating the weight of the "Outside" (O) label. In sequential models, the system uses dynamic programming (such as the Viterbi algorithm) to find the most likely sequence of labels. The authors introduced a specific feature, $f_0$, which is active (1) if a token is labeled as "Outside an entity" and inactive (0) otherwise.
By "tweaking" the weight ($w_0$) of this feature:
- Large negative values penalize the "Outside" label, forcing the model to label more tokens as entities, thereby boosting recall.
- Large positive values reward the "Outside" label, making the model more conservative and boosting precision.
To find the optimal $w_0$ for a specific user-defined metric (like $F_\beta$, where $\beta$ weights recall relative to precision), the authors utilized a Gauss-Newton line search. This algorithm iteratively approximates the performance curve to find the optimal weight on a labeled validation dataset.
Experimental Evaluation
The method was evaluated on three distinct corpora to test robustness across different writing styles:
- MUC-6: Formal Wall Street Journal news articles.
- Enron: Emails from the Enron corpus (meetings and calendars).
- Mgmt-Groups: Emails from a CMU management course, testing generalization on a different mix of names.
Key Results and Discussion
The "tweaking" method successfully allowed the models to peak at the user's desired $\beta$ value. For example, when optimizing for $F_2$ (weighting recall twice as much as precision), the resulting model achieved its highest $F_2$ score on the test set.
As shown in the Precision-Recall curves, the method provides a smooth, predictable transition between the two extremes. On the MUC-6 dataset using a CRF, the system could move from a baseline of 76% recall / 93% precision to a high-recall setting of 91% recall / 65% precision, or a high-precision setting of 53% recall / 100% precision.
Key Insights
- The Power of One Parameter: Adjusting just $w_0$ is sufficient to generate a nearly complete and smooth precision-recall curve, eliminating the need to retrain models or invent complex confidence measures.
- Optimization Generalizes: Weights optimized on training data remain effective on unseen test data, suggesting the tweak captures fundamental model uncertainty.
- Token vs. Entity Smoothness: Optimizing for token-level performance is significantly smoother than entity-level performance. Entity-level metrics are more "jittery" because a single token shift can invalidate an entire multi-word entity extraction.
Conclusion
While the objective function for NER performance is not perfectly smooth, the Gauss-Newton search provides an effective approximation for tuning systems to user preferences. This approach is theoretically applicable to any structured learning task with a "background" or "null" class, such as part-of-speech tagging or image segmentation.
Summary updated