TL;DR
This paper argues that traditional Vietnamese Natural Language Processing (NLP) approaches, which rely on word segmentation based on syllable structure, are linguistically flawed and hinder performance. They introduce ViWordFormer, a novel Transformer-based model that directly learns Vietnamese phrasal lexemes without explicit segmentation, achieving state-of-the-art results on various Vietnamese NLP tasks.
What Problem Does This Paper Address?
For decades, Vietnamese NLP has operated under the assumption that Vietnamese words are formed by concatenating syllables, leading to a heavy reliance on word segmentation as a crucial preprocessing step. This assumption, however, lacks strong linguistic grounding and has led to a proliferation of complex segmentation algorithms. The authors contend that this approach creates artificial complexities and can obscure the true semantic units within Vietnamese text. They aim to challenge this paradigm by proposing a linguistically informed, segmentation-free approach to understanding Vietnamese text.
Methods
The core of the paper is the introduction of ViWordFormer, a Transformer-based model designed to learn Vietnamese word formation and representation without explicit word segmentation.
- Linguistic Rationale: The authors draw upon two linguistic views of Vietnamese:
- Morpheme-Based Linguistic View: Words are formed from morphemes, where each morpheme is a meaningful unit, typically one syllable. This view underpins traditional word segmentation.
- Word-Based Linguistic View: Words and morphemes are identical in Vietnamese. This view suggests that words are the fundamental meaningful units, and segmentation is unnecessary. The paper advocates for this view.
- ViWordFormer Architecture:
- Attention Module: Standard Transformer self-attention mechanism to model relationships between input tokens.
- Phrasal Lexeme Module: This is the novel component. It introduces a "Phrasal score" (P) that augments the attention scores. This score is designed to capture the probability of words belonging to the same phrasal lexeme, based on a bilinear function that models the semantic relationship between adjacent words. This module aims to constrain attention to focus on linguistically coherent units.
- Co-Text Module: To address ViWordFormer's difficulty in forming larger phrasal lexemes in deeper layers, this module maintains the "co-text" (information about phrasal lexemes) from shallower layers, allowing deeper layers to build upon this information.
Datasets
The paper evaluates ViWordFormer and baselines on two types of Vietnamese NLP tasks:
- Sentence Classification:
- UIT-VSFC (Vietnamese Sentiment Analysis)
- UIT-ViCTSD (Vietnamese Constructive Detection)
- UIT-VIOCD (Vietnamese Toxic Detection)
- ViHSD (Vietnamese Hate Speech Detection)
- Sequential Labeling (Named Entity Recognition - NER):
- PhoNER (Vietnamese Named Entity Recognition)
- ViHOS (Vietnamese Hate Speech Spans Detection)
The authors note that PhoNER was chosen for NER because it offers a syllable-based version, allowing for experiments without requiring prior word segmentation, unlike other available NER datasets.
Key Insights and Ideas
- Word Segmentation is Not Necessarily Crucial: The paper strongly argues, based on linguistic theory and empirical results, that word segmentation is not an indispensable preprocessing step for Vietnamese NLP.
- Linguistic Views Matter: The distinction between the morpheme-based and word-based linguistic views is critical for understanding the challenges and opportunities in Vietnamese NLP. The paper champions the word-based view.
- Phrasal Lexemes as the True Units: The authors propose that Vietnamese text is better understood by modeling "phrasal lexemes" – linguistically coherent units that may span multiple traditional "words" or morphemes – rather than relying on syllable-based segmentation.
- Transformer Architecture Adaptable: The Transformer architecture, with modifications like the Phrasal Lexeme and Co-Text modules, can effectively learn these phrasal lexemes without explicit segmentation.
Key Results
- First Scenario (Segmentation vs. No Segmentation): Baselines using the word-based linguistic view (treating space-separated tokens as words) performed comparably to, and sometimes better than, those using the morpheme-based view (with explicit segmentation). This empirically supports the idea that explicit segmentation does not significantly improve performance.
- Second Scenario (ViWordFormer vs. Baselines): ViWordFormer achieved state-of-the-art results on most datasets for both sentence classification and sequential labeling tasks, outperforming Transformer, RNN-based, and CNN-based baselines.
- Co-Text Module Effectiveness: An ablation study showed that ViWordFormer's performance significantly dropped without the Co-Text module, highlighting its importance in maintaining phrasal information across layers.
- Visualization: Attention maps from ViWordFormer demonstrated its ability to capture phrasal relations and semantic dependencies, identifying meaningful phrases like "tâm thần" (mental disorder) and "hoang tưởng" (paranoid) even in early layers.
Limitations and Future Work
- Linear Assumption: The paper acknowledges that the bilinear function used to model semantic structure ($\mu$) is a linear assumption, which might not fully capture the complexity of phrasal relations in Vietnamese.
- Task Scope: Current experiments are limited to sentence classification and sequential labeling.
- Future Directions:
- Explore ViWordFormer on other Vietnamese NLP tasks like machine translation, text summarization, and reading comprehension.
- Investigate more sophisticated kernel functions for the Phrasal Lexeme module to better approximate semantic structures.
- Develop better pre-trained language models for Vietnamese that incorporate linguistic theories and the concept of phrasal lexemes.
Summary updated