On the Role of Lexical Features in Sequence Labeling

Yoav Goldberg, Michael ElhadadEMNLP 2009 · OpenReview

Abstract

We use the technique of SVM anchoring to demonstrate that lexical features extracted from a training corpus are not necessary to obtain state of the art results on tasks such as Named Entity Recognition and Chunking. While standard models require as many as 100K distinct features, we derive models with as little as 1K features that perform as well or better on different domains. These robust reduced models indicate that the way rare lexical features contribute to classification in NLP is not fully understood. Contrastive error analysis (with and without lexical features) indicates that lexical features do contribute to resolving some semantic and complex syntactic ambiguities -- but we find this contribution does not generalize outside the training corpus. As a general strategy, we believe lexical features should not be directly derived from a training corpus but instead, carefully inferred and selected from other sources.

Introduction

In Natural Language Processing (NLP), sequence labeling tasks like Named Entity Recognition (NER) and chunking traditionally rely on high-dimensional feature spaces. Conventional wisdom suggests that including every unique word form (lexical features) from the training corpus is essential for state-of-the-art performance. However, because language follows a Zipfian distribution, the vast majority of these features are extremely rare, appearing only once or twice. This paper investigates whether these rare lexical features provide genuine linguistic information or merely serve as "mechanical anchors" that allow models to overfit the training data.

Methodology: Anchored SVM Learning

Standard Support Vector Machines (SVMs) often experience a significant drop in performance when rare features are pruned. This is because SVMs use these features to find a separating hyperplane in difficult, ambiguous cases. To isolate the informational value of these features, we employ Anchored SVM Learning.

In this approach, we add a unique, artificial "anchor" feature to every training instance, ensuring the data is perfectly separable by construction. During testing, these anchors are removed. This technique allows for aggressive feature pruning—removing any feature appearing fewer than $k$ times—without the catastrophic loss of separability that plagues standard soft-margin SVMs.

Experimental Results

We evaluated our approach on standard benchmarks, including CoNLL 2002 Dutch NER and CoNLL 2000 NP Chunking.

Named Entity Recognition

For the NER task, the standard model utilized over 186,000 features. By applying Anchored SVMs and pruning features that appeared fewer than 1,000 times, we reduced the feature set by 99.4% to just 1,207 features. Remarkably, the F-score only decreased from 90.92 to 90.10, demonstrating that the vast majority of lexical features are redundant for this task.

Performance of Anchored SVM vs. Soft-Margin SVM on NER under aggressive pruning.

NP Chunking

Similar results were observed in NP Chunking. Pruning features with a frequency less than 100 reduced the dimensionality from approximately 93,000 to 1,100 features. The Anchored SVM maintained a stable performance, dipping only slightly from 94.08 to 93.83.

Comparison of Soft-Margin and Anchored SVM performance on NP Chunking across various pruning levels.

Generalization and Domain Variation

To test the robustness of these reduced models, we conducted out-of-domain evaluations using Wikipedia articles and LiveJournal blog posts. We found that fully lexicalized models are often brittle; they perform well on news text (similar to the training data) but struggle with the stylistic variations of blogs. In contrast, pruned models—which rely on more robust features like Part-of-Speech (POS) tags and suffixes—frequently outperformed their high-dimensional counterparts on out-of-domain data.

Conclusion

Our findings suggest that the reliance on rare lexical features in sequence labeling is often a "lexical crutch" that aids training-set separability at the expense of generalization. While lexical information is occasionally necessary for resolving deep syntactic or semantic ambiguities, current methods of direct extraction from training corpora fail to capture this in a generalizable way. We advocate for a strategy of "strategic lexicalization," where lexical information is carefully inferred from external, broad-coverage sources rather than being blindly extracted from limited training sets.

Summary updated

Abstract

We use the technique of SVM anchoring to demonstrate that lexical features extracted from a training corpus are not necessary to obtain state of the art results on tasks such as Named Entity Recognition and Chunking. While standard models require as many as 100K distinct features, we derive models with as little as 1K features that perform as well or better on different domains. These robust reduced models indicate that the way rare lexical features contribute to classification in NLP is not fully understood. Contrastive error analysis (with and without lexical features) indicates that lexical features do contribute to resolving some semantic and complex syntactic ambiguities -- but we find this contribution does not generalize outside the training corpus. As a general strategy, we believe lexical features should not be directly derived from a training corpus but instead, carefully inferred and selected from other sources.
BibTeX
@inproceedings{goldberg2009role,
  author = {Yoav Goldberg and Michael Elhadad},
  title = {{On the Role of Lexical Features in Sequence Labeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  url = {https://openreview.net/forum?id=rk4dzMz_WH},
}
Figure