TL;DR
This paper introduces a novel two-stage framework for Ancient Chinese Named Entity Recognition (NER) that enforces valid entity tag transitions using a "Logits-Constrained" mechanism. This approach, built upon the GujiRoBERTa model, significantly improves accuracy, especially in complex scenarios, by directly integrating structural constraints into the model's decision-making process, outperforming traditional methods like CRFs.
What Problem Does This Paper Address?
Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and classify entities like people, places, and organizations within text. However, applying NER to Ancient Chinese presents unique challenges due to the language's complex semantic properties and often ambiguous word meanings. This ambiguity can lead to errors in predicting the sequence of entity tags (e.g., the start, middle, or end of an entity). Traditional methods often rely on external components like Conditional Random Fields (CRFs) or Bidirectional LSTMs (BiLSTMs) to enforce these tag sequence rules, but these can be complex, introduce interference, and struggle with low-resource or label-sparse scenarios. The paper aims to address these limitations by proposing a simpler, yet effective, method to ensure structurally valid entity tag sequences.
Methods
The proposed framework is a two-stage process:
- Stage 1: Contextual Encoding with GujiRoBERTa: The core of the first stage uses GujiRoBERTa, a pre-trained language model specifically optimized for traditional Chinese texts. This model generates contextualized embeddings for each word, capturing nuanced linguistic patterns. A simple linear projection layer then transforms these embeddings into initial label logits (scores for each possible entity tag). Crucially, this stage trains using standard cross-entropy loss without explicit transition modeling.
- Stage 2: Logits-Constrained Decoding: This is the novel part of the framework. Instead of relying on external models, it introduces a "Logits-Constrained" (LC) mechanism. This mechanism uses a pre-defined constraint matrix that encodes valid transitions between entity tags (e.g., a "Begin" tag can only be followed by a "Middle" or "End" tag for the same entity type). During inference, this matrix is used to dynamically mask invalid transitions in the predicted logits. This "masked autoregressive refinement" ensures that the output sequence adheres to the defined structural constraints without requiring any additional trainable parameters, making it differentiable and efficient.
The authors explicitly state their design philosophy avoids stacking complex components like CRFs or BiLSTMs to maintain model simplicity and avoid interference patterns during learning.
Datasets
The experiments were conducted on the EvaHan 2025 datasets, which are specifically designed for Ancient Chinese NER. The paper mentions three training sets:
- trainset_a: Annotated with 6 NER categories.
- trainset_b: Annotated with 3 NER categories (a subset of trainset_a).
- trainset_c: Annotated with 6 NER categories.
All sets also include a "O" label for non-entity tokens. The tagging scheme used is BMES (Begin, Middle, End, Single). Due to the subset relationship between trainset_a and trainset_b, the dataset effectively includes 37 classification labels.
The paper also discusses a "hybrid Dataset B" created by integrating annotated data from Dataset A, increasing the sample size by 229%.
Key Insights and Ideas
- Simplicity is Key: The paper advocates for a "less is more" approach, demonstrating that a well-designed constraint mechanism integrated directly into the model can outperform complex, multi-component systems.
- Direct Constraint Integration: The Logits-Constrained (LC) framework's strength lies in its ability to directly enforce structural rules (like valid tag transitions) within the model's decision boundary, eliminating the need for heuristic post-processing or separate constraint models.
- BiLSTM Detrimental: Surprisingly, the inclusion of BiLSTM modules consistently degraded performance across all datasets. The authors speculate this is due to disruption of the pre-trained model's attention patterns and potential for parameter updates to get trapped in local optima.
- CRF Limitations with Scale: While CRFs are effective for simpler scenarios, their performance degrades as the number of entity labels increases, likely because manually designed transition matrices struggle to cover high-dimensional state spaces.
- Data-Driven Model Selection: The paper proposes an empirically derived formula to guide model selection based on label cardinality (L) and sentence count (N), offering practical advice on when to use LC-only versus CRF+LC.
Key Results
- The proposed Logits-Constrained framework with GujiRoBERTa achieved state-of-the-art results on the EvaHan 2025 datasets.
- The LC framework showed a significant advantage in scenarios with six or more labels (Datasets A/C), achieving an average F1 improvement of 1.95%.
- On Dataset C, which has a complex entity distribution, the LC mechanism improved the F1 score from 0.8067 to 0.8262 (+2.95%).
- The introduction of BiLSTM led to an average F1 degradation of -3.8%.
- Expanding the dataset size (hybrid Dataset B) led to a 1.33% increase in F1 score.
- The combined CRF+LC framework yielded optimal performance, surpassing individual implementations, especially on the hybrid dataset.
Limitations and Future Work
- Manual Constraint Matrix: The current Logits-Constrained matrix is based on manual BMES rules, which might not generalize perfectly and is sensitive to the initial token's accuracy.
- Inference Overhead: The two-stage pipeline introduces some additional inference overhead compared to end-to-end models.
- Sentence Segmentation Sensitivity: Performance is dependent on sentence segmentation quality, making it vulnerable to errors in unpunctuated or irregular historical texts.
Future work could explore:
- Adaptive constraint learning.
- Unified architectures to address the two-stage pipeline.
- More robust methods for handling unpunctuated or irregular texts.
Summary updated