Introduction
Sequence labeling tasks, such as Named Entity Recognition (NER), are fundamental to natural language processing. Traditionally, these tasks have been addressed using Conditional Random Fields (CRFs), which model dependencies between adjacent word-level labels. While CRFs are effective at capturing local transitions, they often struggle to model the holistic features of multi-word segments (e.g., "Barack Hussein Obama").
To address this, semi-Markov CRFs (SCRFs) were introduced to treat segments as the primary unit of labeling. However, standard SCRFs frequently ignore internal word-level transitions during training. This paper proposes the Hybrid semi-Markov Conditional Random Field (HSCRF), a novel architecture that unifies word-level and segment-level perspectives to improve labeling accuracy across entities of varying lengths.
The HSCRF Architecture
The core innovation of this work is the HSCRF layer, which derives segment scores from underlying word-level label scores. Unlike traditional SCRFs that calculate segment scores directly from a single representation, the HSCRF utilizes a shared Bi-directional LSTM (Bi-LSTM) backbone to generate word representations.
For each word within a potential segment, the model constructs a feature vector consisting of the Bi-LSTM representation, a boundary feature (the difference between the start and end word representations), and a positional embedding. This allows the model to benefit from fine-grained word-level supervision while maintaining the global context of the segment.
Joint Training and Decoding
The proposed model, LM-BLSTM-JNT, employs a shared Bi-LSTM backbone that feeds into two parallel output layers: a standard CRF and the new HSCRF.
- Joint Training: The loss functions for both the CRF and HSCRF layers are minimized simultaneously. This joint training acts as a regularizer, forcing the Bi-LSTM to learn robust representations that satisfy both local word-to-word and global segment-to-segment dependencies.
- Joint Decoding: The authors implement a "naive joint decoding" strategy. The model generates label sequences from both the CRF and HSCRF layers and then selects the final result based on which model yields the lowest combined negative log-likelihood (NLL) when cross-evaluated.
Experimental Evaluation
The model was evaluated on the CoNLL 2003 English NER dataset. To maintain efficiency, the maximum segment length was capped at 6 words, covering over 99.5% of the training data. The results demonstrate that the hybrid approach successfully captures the complementary strengths of both architectures.
As shown in the performance analysis, CRFs typically excel at identifying short entities (1-2 words), while SCRFs are more effective for longer entities (3+ words). The joint model (LM-BLSTM-JNT) achieves the best of both worlds, outperforming baselines across all entity lengths. Specifically, it achieved a state-of-the-art mean F1 score of 91.38% and a maximum F1 score of 91.53% on the CoNLL 2003 task without the use of external knowledge like gazetteers.
Conclusion
The Hybrid semi-Markov CRF provides a powerful framework for sequence labeling by bridging the gap between word-level and segment-level modeling. By integrating word-level information into the segment scoring process and utilizing joint decoding, the model achieves superior performance, particularly on complex, multi-word entities. Future work may explore more sophisticated integration of probability distributions and the inclusion of large-scale pre-trained transformers to further enhance these results.
Summary updated