Empirical Evaluation of Character-Based Model on Neural Named-Entity Recognition in Indonesian Conversational Texts

Kemal Kurniawan, Samuel LouvanEMNLP 2018

TL;DR

This paper tackles the challenge of Named Entity Recognition (NER) in noisy, informal Indonesian conversational texts, where out-of-vocabulary (OOV) words are rampant. They demonstrate that neural models leveraging character-level embeddings significantly outperform traditional word-based models, especially in handling OOV words, achieving up to a 15 F1 point improvement.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and categorize named entities (like people, organizations, locations) in text. While NER has been extensively studied on well-structured data like news articles, its application to conversational texts, especially in languages like Indonesian, presents unique challenges. Conversational texts are inherently noisy, informal, and prone to numerous word variations, leading to a high rate of Out-of-Vocabulary (OOV) words. Traditional word-based NER models struggle with OOV words because they cannot represent them effectively. This paper addresses this gap by empirically evaluating character-based neural models for NER specifically on Indonesian conversational data, aiming to understand their effectiveness in handling OOV challenges.

Methods

The authors employ neural sequence labeling models, specifically building upon the architecture proposed by Rei et al. (2016). The core of their methodology lies in comparing two word representation strategies:

  • Word-level embeddings: Standard approach where each word is represented by a learned vector.
  • Character-level embeddings: This approach uses character embeddings, which are then combined (e.g., through concatenation or attention mechanisms) to form a representation for each word. This allows the model to infer representations for OOV words by leveraging their sub-word (character) structure, thus capturing orthographic similarities.

They experiment with a concatenation model, where character embeddings are concatenated to form word representations, and an attention model.

Datasets

The study utilizes two custom, manually annotated Indonesian conversational datasets:

  • SMALL-TALK: Contains 16K conversational messages from a chatbot service, focusing on small talk. It includes 6 entities: DATETIME, EMAIL, GENDER, LOCATION, PERSON, and PHONE.
  • TASK-ORIENTED: Comprises 72K task-oriented imperative messages (e.g., flight booking, food delivery) from another service. It includes 4 entities: EMAIL, LOC, PER, and PHONE.

These datasets are noted for their high OOV rates, with SMALL-TALK exceeding 50% OOV. The authors mention they cannot publish these datasets due to proprietary reasons.

Key Insights and Ideas

  • Character is King for OOV: The most significant insight is the superior performance of character-based models in handling OOV words. This is crucial for noisy conversational data where OOV words are prevalent.
  • Robustness to High OOV: Character models demonstrate remarkable robustness, maintaining performance even when OOV rates approach 100%, a feat that word-based models cannot achieve.
  • Beyond Word-Level Limitations: The paper highlights that simply assigning a single vector to OOV words is suboptimal. Character-level representations allow models to exploit morphemic similarities, leading to better generalization.
  • Task-Specific Entity Sets: The difference in entity sets between SMALL-TALK and TASK-ORIENTED datasets reflects the practical needs of different conversational agents, with SMALL-TALK requiring more personal information entities.

Key Results

  • Overall Improvement: Character models (concatenation and attention) outperform word embedding-only models by up to 4 F1 points on the overall test set.
  • OOV Performance Boost: For entities containing OOV words, character models achieve an improvement of as high as 15 F1 points on the SMALL-TALK dataset.
  • GENDER Entity Success: On the SMALL-TALK dataset, the concatenation model achieves 40 F1 points for GENDER, while the word embedding-only model recognizes none.
  • Robustness: Performance degradation for character models is minimal even at OOV rates as high as 90%, whereas word embedding-only models start to drop significantly around 70% OOV.
  • Confusion Matrix Analysis: Both models struggle with PERSON and LOCATION in SMALL-TALK, but character models show a reduction in false negatives. In TASK-ORIENTED, both models have similar performance, with a tendency to partition long location entities into smaller parts, which the authors deem reasonable.

Limitations and Future Work

  • Speed: The authors acknowledge that the LSTM used for composing character embeddings can be slow. They point to recent work using CNNs for character composition as a potential speed improvement.
  • Dataset Publication: The proprietary nature of the datasets prevents public access, limiting reproducibility for other researchers.
  • Future Directions:
    • Exploring character trigrams as subword units.
  • Investigating hybrid approaches, such as using regex-based classifiers for entities with clear patterns (like PHONE and EMAIL) alongside neural models for person and location names.

Summary updated

BibTeX
@inproceedings{kurniawan2018empirical,
  author = {Kemal Kurniawan and Samuel Louvan},
  title = {{Empirical Evaluation of Character-Based Model on Neural Named-Entity Recognition in Indonesian Conversational Texts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure