VnCoreNLP: A Vietnamese Natural Language Processing Toolkit

Thanh Vu, Dat Quoc Nguyen, Dai Quoc Nguyen, Mark Dras, Mark Johnson

Abstract

We present an easy-to-use and fast toolkit, namely VnCoreNLP—a Java NLP annotation pipeline for Vietnamese. Our VnCoreNLP supports key natural language processing (NLP) tasks including word segmentation, part-of-speech (POS) tagging, named entity recognition (NER) and dependency parsing, and obtains state-of-the-art (SOTA) results for these tasks. We release VnCoreNLP to provide rich linguistic annotations to facilitate research work on Vietnamese NLP. Our VnCoreNLP is open-source under GPL v3.

Introduction

While the NLP community has seen a surge in tools for English, such as Stanford CoreNLP, Vietnamese NLP research remained fragmented for years. Existing tools were often specialized for single tasks, outdated, or—in the case of newer neural-based toolkits—prohibitively slow for large-scale production use, with some processing only 300 words per second.

Vietnamese presents unique linguistic challenges, most notably that white space is not a reliable word delimiter. In Vietnamese, white space separates syllables, and a single word can consist of multiple syllables (e.g., "học sinh" means "student"). This makes word segmentation a critical and difficult first step for all downstream tasks. VnCoreNLP addresses these issues by providing a unified, high-performance pipeline.

System Architecture

VnCoreNLP is built as a Java pipeline where an Annotation object is passed through various components. Rather than relying on heavy deep learning architectures, the authors opted for optimized, feature-based models to maximize speed without sacrificing accuracy.

The VnCoreNLP pipeline architecture, showing the flow from raw text to annotated output via an Annotation object.

The pipeline includes:

  • Word Segmentation: Uses a transformation rule-based learning model.
  • POS Tagging: Employs MarMoT, a generic CRF-based framework known for efficiency in morphological tagging.
  • NER: Utilizes a dynamic feature induction model that automatically optimizes feature combinations.
  • Dependency Parsing: Implements a greedy transition-based parsing model with selectional branching.

The entire toolkit is packaged into a single .jar file, simplifying deployment and eliminating dependency conflicts.

Key Insights and Performance

A major finding of this work is that well-engineered feature-based models can outperform complex neural architectures like BiLSTM-CRF for less-resourced languages like Vietnamese, while being significantly faster.

Comparison of NER performance and speed. VnCoreNLP achieves the highest F1 score while being significantly faster than neural baselines.

By avoiding heavy neural networks, VnCoreNLP achieves a processing speed of approximately 18,000 words per second for NER—roughly 10 times faster than the best neural baseline and 60 times faster than previous Vietnamese toolkits.

Furthermore, VnCoreNLP was evaluated in a "realistic" NER scenario. Unlike standard evaluations that assume perfect word segmentation, VnCoreNLP was tested on its ability to handle syllable-to-word merging itself, proving its robustness for real-world data.

Results Summary

VnCoreNLP established new State-of-the-Art (SOTA) benchmarks across all four core tasks:

  • NER: Achieved an F1-score of 88.55%.
  • Dependency Parsing: Achieved a Labeled Attachment Score (LAS) of 73.39% (with gold POS tags).
  • Word Segmentation: Reached an F1-score of 97.90%.
  • POS Tagging: Achieved an accuracy of 95.88%.

Conclusion

VnCoreNLP provides a fast, accurate, and easy-to-use solution for Vietnamese NLP. By prioritizing efficient feature-based models, it offers a practical tool for both researchers and developers working with large-scale Vietnamese text data.

Availability: VnCoreNLP is open-source and available on GitHub at https://github.com/vncorenlp/VnCoreNLP.

Summary updated

Abstract

We present an easy-to-use and fast toolkit, namely VnCoreNLP---a Java NLP annotation pipeline for Vietnamese. Our VnCoreNLP supports key natural language processing (NLP) tasks including word segmentation, part-of-speech (POS) tagging, named entity recognition (NER) and dependency parsing, and obtains state-of-the-art (SOTA) results for these tasks. We release VnCoreNLP to provide rich linguistic annotations to facilitate research work on Vietnamese NLP. Our VnCoreNLP is open-source under GPL v3, and available at: this https URL
BibTeX
@inproceedings{vu2018vncorenlp,
  author = {Thanh Vu and Dat Quoc Nguyen and Dai Quoc Nguyen and Mark Dras and Mark Johnson},
  title = {{VnCoreNLP: A Vietnamese Natural Language Processing Toolkit}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=SkVWiQ-_br},
}
Figure