Abstract: Discriminative models have been of interest in the NLP community in recent years. Previous research has shown that they are advantageous over generative models. In this paper, we investigate how different objective functions and optimization methods affect the performance of the classifiers in the discriminative learning framework. We focus on the sequence labelling problem, particularly POS tagging and NER tasks. Our experiments show that changing the objective function is not as effective as changing the features included in the model.
Introduction
In the early 2000s, the Natural Language Processing (NLP) community began a significant shift from generative models, such as Hidden Markov Models (HMMs), toward discriminative models like Conditional Random Fields (CRFs). While discriminative models are better equipped to handle overlapping features and avoid the "label bias" problem, they introduce new complexities for researchers. Specifically, one must choose an appropriate loss function, determine the granularity of optimization (sequential vs. pointwise), and select an efficient optimization algorithm. This study investigates whether these mathematical choices significantly impact final performance or if researchers should prioritize feature engineering.
Methodology
The researchers designed a comparative framework using a 2x2 matrix of objective functions based on two criteria:
- Scale: Exponential loss (standard in Boosting) vs. Logarithmic loss (standard for Maximum Likelihood).
- Manner: Sequential (treating the whole sequence as one unit) vs. Pointwise (optimizing marginal probabilities of individual labels).
This resulted in four distinct loss functions, denoted as $\mathcal{F}^1$ through $\mathcal{F}^4$.
To optimize these functions, the study compared exact gradient-based methods with approximate methods like the Perceptron algorithm and Boosting. These were evaluated on two benchmarks: Part-of-Speech (POS) tagging using the Penn TreeBank and Named Entity Recognition (NER) using the CoNLL 2002 Spanish corpus. Three feature sets ($S1, S2, S3$) were tested, ranging from simple bigrams to complex sets incorporating morphological information and context windows.
Key Results and Insights
The most striking finding is the dominance of feature engineering over the choice of objective function. While the four loss functions vary in their theoretical properties—for instance, exponential loss is more sensitive to noise—their impact on final accuracy was marginal.
As shown in the NER results, moving from the basic $S1$ feature set to the enhanced $S3$ set resulted in a massive jump in F1 scores (from ~60% to ~73%). In contrast, the variation between different loss functions for a given feature set was often less than 1.5%. Similar stability was observed in POS tagging, where accuracy hovered between 94.5% and 95.7% across all configurations.
Other notable insights include:
- Sparsity: Boosting produced much sparser classifiers, using only about 10% of available features, which leads to faster inference at test time.
- Perceptron Performance: The Perceptron algorithm proved to be a remarkably strong baseline, performing nearly as well as exact CRF methods while being computationally cheaper.
Conclusion
The study concludes that while the choice of loss function and optimization method involves interesting theoretical trade-offs regarding noise sensitivity and model sparsity, they are secondary to the quality of the feature set. For practitioners in sequence labeling, the highest returns on effort come from richer feature engineering rather than fine-tuning the mathematical objective. Future work may explore the application of Support Vector Machines (SVMs) and kernel methods to further enhance discriminative sequence learning.
Summary updated