Hybrid semi-Markov CRF for Neural Sequence Labeling

Zhi-Xiu Ye, Zhen-Hua Ling

Introduction

Sequence labeling tasks, such as Named Entity Recognition (NER), are fundamental to natural language processing. Traditionally, these tasks have been addressed using Conditional Random Fields (CRFs), which model dependencies between adjacent word-level labels. While CRFs are effective at capturing local transitions, they often struggle to model the holistic features of multi-word segments (e.g., "Barack Hussein Obama").

To address this, semi-Markov CRFs (SCRFs) were introduced to treat segments as the primary unit of labeling. However, standard SCRFs frequently ignore internal word-level transitions during training. This paper proposes the Hybrid semi-Markov Conditional Random Field (HSCRF), a novel architecture that unifies word-level and segment-level perspectives to improve labeling accuracy across entities of varying lengths.

The HSCRF Architecture

The core innovation of this work is the HSCRF layer, which derives segment scores from underlying word-level label scores. Unlike traditional SCRFs that calculate segment scores directly from a single representation, the HSCRF utilizes a shared Bi-directional LSTM (Bi-LSTM) backbone to generate word representations.

Architecture of the HSCRF model showing the flow from word embeddings to segment scores.

For each word within a potential segment, the model constructs a feature vector consisting of the Bi-LSTM representation, a boundary feature (the difference between the start and end word representations), and a positional embedding. This allows the model to benefit from fine-grained word-level supervision while maintaining the global context of the segment.

Joint Training and Decoding

The proposed model, LM-BLSTM-JNT, employs a shared Bi-LSTM backbone that feeds into two parallel output layers: a standard CRF and the new HSCRF.

  1. Joint Training: The loss functions for both the CRF and HSCRF layers are minimized simultaneously. This joint training acts as a regularizer, forcing the Bi-LSTM to learn robust representations that satisfy both local word-to-word and global segment-to-segment dependencies.
  2. Joint Decoding: The authors implement a "naive joint decoding" strategy. The model generates label sequences from both the CRF and HSCRF layers and then selects the final result based on which model yields the lowest combined negative log-likelihood (NLL) when cross-evaluated.

Experimental Evaluation

The model was evaluated on the CoNLL 2003 English NER dataset. To maintain efficiency, the maximum segment length was capped at 6 words, covering over 99.5% of the training data. The results demonstrate that the hybrid approach successfully captures the complementary strengths of both architectures.

Performance comparison of different models across various entity lengths.

As shown in the performance analysis, CRFs typically excel at identifying short entities (1-2 words), while SCRFs are more effective for longer entities (3+ words). The joint model (LM-BLSTM-JNT) achieves the best of both worlds, outperforming baselines across all entity lengths. Specifically, it achieved a state-of-the-art mean F1 score of 91.38% and a maximum F1 score of 91.53% on the CoNLL 2003 task without the use of external knowledge like gazetteers.

Conclusion

The Hybrid semi-Markov CRF provides a powerful framework for sequence labeling by bridging the gap between word-level and segment-level modeling. By integrating word-level information into the segment scoring process and utilizing joint decoding, the model achieves superior performance, particularly on complex, multi-word entities. Future work may explore more sophisticated integration of probability distributions and the inclusion of large-scale pre-trained transformers to further enhance these results.

Summary updated

BibTeX
@inproceedings{ye2018hybrid,
  author = {Zhi-Xiu Ye and Zhen-Hua Ling},
  title = {{Hybrid semi-Markov CRF for Neural Sequence Labeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=Sk-LVpgdZr},
}
Figure