TL;DR
This paper tackles the under-researched problem of Complex Word Identification (CWI) in Vietnamese, a crucial first step for Vietnamese Text Simplification. They found that word frequency is a strong predictor of complexity, achieving 86.87% accuracy, and also explored the subjectivity of CWI through human annotation.
What Problem Does This Paper Address?
Text Simplification (TS) aims to make text more accessible to a wider audience, including those with reading difficulties or non-native speakers. While TS has seen significant progress in languages like English, Spanish, and Japanese, it remains a relatively unexplored area for Vietnamese. A key bottleneck in TS, particularly Lexical Simplification, is identifying "complex" words that need simplification. This paper focuses on the Vietnamese-specific Complex Word Identification (CWI) task, aiming to build foundational work for future Vietnamese TS systems.
Methods
The authors explored two primary approaches for CWI in Vietnamese:
- Frequency Threshold: This method leverages the observation that less frequent words are often more complex. They trained a model by sorting words by frequency and testing different cutoff points to determine the optimal threshold for classifying words as simple or complex.
- Support Vector Machines (SVM) Classifier: This approach uses a machine learning model with multiple features. In addition to frequency, they incorporated features like the number of syllables, number of characters, and number of characters with diacritics. The SVM was trained using the scikit-learn library with a radial basis function kernel.
Datasets
The study utilized three Vietnamese corpora, each with different characteristics:
- READABILITY (Luong et al., 2020): A corpus of approximately 3 million words from the literature domain, constructed for Vietnamese text readability research.
- CLUSTER (Tran et al., 2020): A dataset of news articles from various topics, aggregated by Google News.
- CLASSIFICATION (Hoang et al., 2007): A large corpus of news articles categorized into 27 predefined topics.
For their experiments, they created simple and complex word lists by removing stopwords, proper nouns, and invalid words from these corpora.
Key Insights and Ideas
- Frequency is King (Even in Vietnamese): The most striking finding is that word frequency is a remarkably strong indicator of complexity in Vietnamese, mirroring observations in English. This suggests a universal principle in CWI across languages.
- The 10% Rule: For larger corpora, the top 10% most frequent words can reliably be considered "simple," providing a practical heuristic for CWI.
- Subjectivity of Complexity: Human annotation revealed significant variability in what annotators consider complex, even among native speakers. This highlights the inherent subjectivity of the task and the need for robust annotation guidelines.
- Surface Features Aren't Enough: While features like word length and diacritics seem intuitive for predicting complexity, the SVM approach using these features did not significantly outperform the simple frequency-based method, suggesting that deeper linguistic or contextual features might be needed.
Key Results
- Frequency Threshold Performance: The frequency-based approach achieved a mean accuracy of 86.87% across the three datasets. The CLASSIFICATION corpus, being the largest and most consistent (news articles), yielded the highest accuracy at 95.3%.
- SVM Performance: The SVM classifier showed only marginal improvements in recall compared to the frequency threshold, but did not significantly improve accuracy or F1 scores. In some cases, it even led to a slight decline in precision and F1 on the READABILITY dataset.
- Human Annotation: When allowing for majority agreement (2 out of 3 annotators), human annotation accuracy jumped from 43.7% to 82.4%, with precision reaching 100%. This indicates that while individual judgments vary, a consensus among annotators aligns well with the data's complexity.
Limitations and Future Work
- Dataset Limitations: The paper acknowledges potential issues with the datasets, such as acronyms and ambiguous words (e.g., "TP") that can have multiple meanings depending on context. The presence of foreign words, especially English, in some corpora also adds noise.
- Surface Features: The SVM experiment suggests that surface-level features alone are not sufficient to significantly improve CWI performance beyond frequency.
- Future Work: The authors suggest several avenues for future research:
- More Salient Features: Exploring features beyond pronunciation, such as sense count, synonym count, and word type (e.g., loan words).
- Contextual Information: Incorporating sentence context to aid in word complexity identification.
- Diverse Human Annotators: Expanding the pool of annotators to include a wider range of backgrounds and expertise to better reflect diverse user needs and construct more personalized simplification models.
Summary updated