Abstract
In high-stakes domains such as finance and medicine, the cost of a false positive in Named Entity Recognition (NER) often outweighs the cost of a missed entity. Standard NER models typically optimize for the $F_1$ score, treating precision and recall with equal weight. This paper presents a neural semi-Markov Structured Support Vector Machine (SSVM) designed to prioritize high precision during the training phase. By operating on segments rather than individual tokens and utilizing a loss-augmented inference objective, the proposed model significantly improves performance on complex, multi-word entities.
Introduction
Named Entity Recognition (NER) is a fundamental task in information extraction. While the $F_1$ score—the harmonic mean of precision and recall—is the standard metric for evaluation, it assumes symmetric costs for false positives and false negatives. In industrial applications like those at Bloomberg, a false positive (e.g., misidentifying a company name in a trade signal) can lead to significant financial repercussions, whereas a false negative might only result in a missed opportunity.
Current approaches to high-precision NER often rely on post-hoc thresholding of model confidence at inference time. However, these methods do not allow the model to learn precision-oriented features during training. This research addresses this gap by integrating precision requirements directly into the training objective using a Structured Support Vector Machine (SSVM) framework.
Methodology
The proposed architecture integrates three core components to achieve high-precision performance:
Neural Feature Extraction
The model utilizes a Bi-directional Long Short-Term Memory (BiLSTM) network combined with character-level Convolutional Neural Networks (CNNs). The CNNs capture sub-word features such as prefixes and suffixes, while the BiLSTM encodes the global context of the sentence.
Semi-Markov Modeling
Unlike standard token-level models (e.g., BIO tagging), the semi-Markov approach operates on variable-length segments. This allows the model to assign scores to entire phrases (e.g., "New York") rather than individual words. This segment-level view is crucial for precision, as it enables the model to evaluate the risk of a multi-word sequence as a single entity.
Structured SVM Objective
The model employs an SSVM objective with loss-augmented inference. This framework allows for the injection of custom costs into the training process. By modifying the standard Hamming loss to penalize false positives on specific target classes more heavily, the model learns to be more "cautious" and prioritize precision.
Experimental Results
The model was evaluated on the CoNLL 2003 English dataset, focusing on Person (PER), Organization (ORG), Location (LOC), and Miscellaneous (MISC) entities.
Precision-Recall Trade-off
The semi-Markov SSVM demonstrates a superior precision-recall trade-off compared to standard Conditional Random Field (CRF) baselines, particularly for the Organization (ORG) class. As illustrated in the Precision-Recall curves, the semi-Markov SSVM consistently maintains higher recall at high precision levels.
Performance on Multi-Word Mentions
A key finding is the model's robustness in identifying long, multi-token mentions. For ORG entities consisting of three or more tokens, the semi-Markov SSVM achieved a recall of 83.52% at a fixed high-precision level, significantly outperforming the thresholded CRF baseline (75.57%). This suggests that the segment-level scoring effectively handles the complexity of longer entity names.
Discussion and Key Insights
- Phrase-Level Penalties: Token-level models often fail to capture the holistic risk of a phrase. The semi-Markov approach allows the model to reject a sequence if the overall phrase-level score does not meet the precision requirement.
- Class Confusion Mitigation: The precision-tuned model showed a reduced tendency to misclassify "MISC" entities (e.g., titles or events) as major classes like ORG or PER, effectively reducing "identity theft" between entity types.
- Training vs. Inference Tuning: While inference-time thresholding is common, training-time optimization provides a more robust foundation for handling variable-length and complex entities.
Conclusion and Future Work
The neural semi-Markov SSVM provides a principled way to incorporate high-precision requirements into NER training. While the model excels at phrase-level precision, it occasionally suffers from "larger predicted span" errors, where adjacent non-entity words are included in the prediction. Future research will explore direct optimization for metrics like $F_\beta$, the combination of training-time and inference-time techniques, and methods to mitigate span-length errors.
Summary updated