Attentive Neural Network for Named Entity Recognition in Vietnamese

Kim Anh Nguyen, Ngan Dong, Cam-Tu Nguyen

TL;DR

This paper introduces VNER, an attentive neural network for Vietnamese Named Entity Recognition (NER). By combining character-based language models, pre-trained word embeddings, and an attention mechanism, VNER achieves state-of-the-art results on benchmark Vietnamese NER datasets, outperforming both traditional feature-based and other neural models.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like people, organizations, locations) within text. While traditional NER systems often rely on hand-crafted features and domain-specific resources, which are costly to create and difficult to reuse, neural network approaches have emerged as a powerful alternative. However, for languages like Vietnamese, which have complex morphology and lack extensive linguistic resources, developing effective NER systems remains challenging. This paper aims to address this by proposing a novel neural network architecture that can automatically learn features and effectively handle the nuances of Vietnamese text for NER.

Methods

The proposed VNER model is built upon a neural architecture comprising four main layers:

  • Word Layer: This layer creates a rich representation for each word by concatenating two components:
  • Character-based word embeddings: A neural language model (LM) is used to encode character sequences within a word. This LM is trained on unannotated text and uses forward and backward LSTMs to predict the next word based on character sequences, capturing better lexical information than just spelling.
  • Pre-trained word embeddings: Standard pre-trained word embeddings are also incorporated.
  • Highway Units: Four highway units are employed to enhance the output of the language model, allowing for non-linear transformations and better feature propagation.
  • Encoder Layer: A bidirectional Long Short-Term Memory (LSTM) network is used to encode the entire input sentence. This layer captures contextual information by processing the sentence from both left-to-right and right-to-left.
  • Attention Layer: This layer allows the model to selectively focus on relevant parts of the encoder's output when generating predictions. It computes attention weights based on the decoder's hidden state and the encoder's outputs, producing a context vector that summarizes the most important information for the current decoding step.
  • Decoder Layer: Another LSTM network acts as the decoder. It takes the encoder's output, the previous decoder state, and the attention context vector as input to predict the entity tags.
  • CRF Layer: Finally, a Conditional Random Field (CRF) layer is applied on top of the decoder's outputs to model dependencies between adjacent entity tags, ensuring more coherent and accurate label sequences. The model is trained jointly, optimizing both the language model and the NER task.

Datasets

The VNER model was evaluated on two benchmark datasets for Vietnamese NER:

  • VLSP-2016 NER dataset: This dataset includes four entity types (LOC, ORG, PER, MISC) and provides additional annotations like word segmentation, part-of-speech (POS) tags, and chunking tags.
  • VLSP-2018 NER dataset: This dataset also contains four entity types (LOC, ORG, PER, MISC) but crucially, it lacks additional annotations like word segmentation or POS tags. It also includes nested entities, where one entity can be contained within another.

For VLSP-2016, a development set was created by randomly sampling from the training data.

Key Insights and Ideas

  • Character-level LM for Vietnamese: The use of a character-level neural language model is particularly effective for Vietnamese, a language where word segmentation can be ambiguous and character sequences carry significant lexical meaning. This approach helps capture richer word representations.
  • Attention Mechanism's Impact: The attention mechanism plays a crucial role in improving performance. Experiments showed a significant drop in F1 score when the attention layer was disabled, highlighting its importance in focusing on relevant contextual information.
  • End-to-End Learning: The model's ability to learn directly from raw text without relying on hand-crafted features or external linguistic resources makes it more adaptable and easier to deploy across different domains.
  • Handling Nested Entities: The VNER model demonstrates capability in handling nested entities by defining joint tags that combine level-1 and level-2 entity information, a significant challenge in NER.

Key Results

  • State-of-the-Art Performance: VNER achieved state-of-the-art results on both VLSP-2016 and VLSP-2018 datasets, outperforming both feature-based CRF models and other neural network models.
  • VLSP-2016: VNER achieved an F1 score of 95.33% when using annotated POS and chunking tags, surpassing feature-based CRF (93.93%), VnCoreNLP (92.91%), and vie-ner-lstm (92.05%). Even without POS/chunking tags, it achieved a comparable result to feature-based CRF.
  • VLSP-2018: For single-layer entities, VNER achieved an F1 score of 77.52%, outperforming feature-based CRF (76.63%) and ZA-NER (74.00%). The model without attention scored 73.23%. For nested entities, VNER achieved an F1 score of 77.88%, outperforming feature-based CRF by over 3%.

Limitations and Future Work

  • Computational Cost: The paper notes that while neural models are powerful, they can be complex and require large amounts of training data and time.
  • Future Work: The authors suggest that further research could explore more advanced architectures and potentially incorporate additional linguistic information if available, though the current model demonstrates strong performance without it. The paper also implicitly suggests that extending this approach to other low-resource languages could be a promising direction.

Summary updated

Abstract

We propose an attentive neural network for the task of named entity recognition in Vietnamese. The proposed attentive neural model makes use of character-based language models and word embeddings to encode words as vector representations. A neural network architecture of encoder, attention, and decoder layers is then utilized to encode knowledge of input sentences and to label entity tags. The experimental results show that the proposed attentive neural network achieves the state-of-the-art results on the benchmark named entity recognition datasets in Vietnamese in comparison to both hand-crafted features based models and neural models.
BibTeX
@inproceedings{nguyen2019attentive,
  author = {Kim Anh Nguyen and Ngan Dong and Cam-Tu Nguyen},
  title = {{Attentive Neural Network for Named Entity Recognition in Vietnamese}},
  booktitle = {RIVF},
  year = {2019},
  url = {https://openreview.net/forum?id=znnU1pNBBt},
}
Figure