Abstract
We present a new toolkit—NiuParser—for Chinese syntactic and semantic analysis. It can handle a wide range of Natural Language Processing (NLP) tasks in Chinese, including word segmentation, part-of-speech tagging, named entity recognition, chunking, constituent parsing, dependency parsing, and semantic role labeling. The NiuParser system runs fast and shows state-of-the-art performance on several benchmarks. Moreover, it is very easy to use for both research and industrial purposes. Advanced features include the Software Development Kit (SDK) interfaces and a multi-thread implementation for system speed-up.
Introduction
Processing the Chinese language presents unique challenges compared to English, primarily due to the lack of explicit word boundaries (whitespace) and a highly flexible underlying syntactic structure. While established NLP toolkits like Stanford CoreNLP or FudanNLP exist, they often lack full optimization for Chinese nuances or fail to provide a completely integrated, high-performance pipeline. Researchers and developers frequently encounter compatibility issues and efficiency bottlenecks when stitching together disparate tools for word segmentation, POS tagging, and parsing.
NiuParser addresses these challenges by offering a comprehensive, "all-in-one" C++ toolkit designed specifically for Chinese NLP. It is the first unified package to integrate seven core tasks—from basic word segmentation to complex semantic role labeling—providing industrial-grade speed through multi-threading and optimized statistical models while maintaining state-of-the-art accuracy for academic research.
System Architecture and Methods
NiuParser is built on a pipeline architecture where the output of each stage serves as the input for the subsequent one. This integrated approach reduces error propagation and technical debt. The system leverages various machine learning models to optimize performance across different tasks.
Sequence Labeling
For foundational tasks such as word segmentation, part-of-speech (POS) tagging, named entity recognition (NER), and chunking, NiuParser utilizes Conditional Random Fields (CRF). To mitigate "Out-of-Vocabulary" (OOV) issues—such as dates, emails, and URLs—the toolkit integrates a regular expression engine and a dictionary-based preprocessing layer. This hybrid approach combines statistical robustness with rule-based precision.
Syntactic and Semantic Analysis
For syntactic analysis, NiuParser employs transition-based shift-reduce parsers. The dependency parser is available in multiple versions, including one utilizing a Neural Network classifier and another using an Averaged Perceptron with beam search. Semantic Role Labeling (SRL) is treated as a two-stage classification problem (Identification and Classification) using Maximum Entropy models to identify relationships between predicates and their arguments.
System Optimizations
To ensure industrial readiness, the toolkit supports multi-threading and provides a Software Development Kit (SDK) compatible with both Windows and Linux, facilitating seamless integration into production environments.
Evaluation and Datasets
The performance of NiuParser was evaluated using several standard benchmarks:
- Penn Chinese Treebank (CTB) 5.1: Used for word segmentation, POS tagging, chunking, and constituent parsing.
- OntoNotes: Used for Named Entity Recognition, covering 18 distinct entity types.
- Xue (2008) & Zhang and Clark (2011): Utilized for evaluating semantic role labeling and dependency parsing, respectively.
Key Results and Discussion
NiuParser achieves state-of-the-art or near-state-of-the-art results across all supported tasks. Notably, it maintains high accuracy while delivering exceptional processing speeds.
As shown in the table above, word segmentation reaches a 97.3% F1 score at a speed of 45,000 characters per second. While the system is generally lightweight, dependency parsing is identified as the most resource-intensive task, requiring up to 2.9GB of memory.
The success of NiuParser highlights several key insights:
- Integration Benefits: Supporting seven distinct functions in a single package significantly streamlines NLP workflows.
- Hybrid Modeling: Combining statistical methods with handcrafted rules and large dictionaries improves robustness in real-world scenarios.
- Efficiency: Transition-based parsing provides an optimal balance between speed and accuracy for Chinese syntactic analysis.
Conclusion and Future Work
NiuParser represents a significant advancement in Chinese NLP tooling, offering a fast, accurate, and easy-to-use solution for both researchers and industry professionals. Future development will focus on improving chunking accuracy (currently at 81.1%) and optimizing the memory footprint of the dependency parsing module. Additionally, the team plans to explore Recurrent Neural Networks (RNNs) to further enhance the speed and accuracy of constituent parsing and integrate dependency-based semantic role labeling.
Summary updated