TL;DR
This paper tackles the challenge of automatically identifying Chavacano, a Philippine Creole Spanish, from related languages using character-level Convolutional Neural Networks (CNNs). They achieve a respectable 93% accuracy, demonstrating that character-level features are sufficient for this task and paving the way for practical applications like social media analysis for disaster response.
What Problem Does This Paper Address?
Language Identification (LI) is crucial for many Natural Language Processing (NLP) tasks, from machine translation to web crawling. While LI for major languages is well-established, low-resource languages, especially creoles, remain under-researched due to limited digital resources. Chavacano, a Spanish-based creole spoken in the Philippines, is often misidentified as Spanish by automated systems due to its significant Spanish lexicon. This paper aims to build an LI model that can accurately distinguish Chavacano from its related languages (Spanish, Portuguese, Cebuano, and Hiligaynon), moving beyond geographical proximity as a basis for similarity and focusing on linguistic creolization.
Methods
The authors employ a character-level Convolutional Neural Network (CNN) for language identification. This approach treats text as a raw signal, allowing the CNN to automatically learn relevant character-level features, eliminating the need for manual feature engineering like n-grams. The CNN architecture involves:
- Convolutional Layers: These layers apply filters of varying widths (2-6 characters) to extract local patterns (character n-grams) from the input text.
- Pooling Layer: Max pooling is used to down-sample the feature maps, retaining the most important features detected by the convolutional filters.
- Dense Layer: A fully-connected layer with dropout is used for further processing.
- Softmax Layer: This final layer outputs probabilities for each of the five language classes.
The model is trained using categorical cross-entropy loss and optimized with the Adam optimizer.
Datasets
The study utilizes a mixed-domain corpus comprising:
- Chavacano: Collected from print sources and online platforms.
- Cebuano and Hiligaynon: Sourced from the PH-MNMT corpus, including web-scraped articles and bible translations.
- Spanish and Portuguese: Primarily from the DSL Corpus Collection (news articles) and also from bible translations.
The dataset consists of 107,500 sentences in total, with 21,500 sentences for each of the five languages (Chavacano, Cebuano, Hiligaynon, Spanish, and Portuguese). The data is split into training (90,000 sentences), validation (10,000 sentences), and testing (7,500 sentences) sets using stratified sampling.
Key Insights and Ideas
- Creolization as a Discriminator: The paper highlights that linguistic creolization, rather than just geographical proximity, can be a strong basis for distinguishing similar languages.
- Character-Level CNN Efficacy: The study confirms that character-level CNNs are effective in capturing the nuances of language, even for closely related languages and creoles, without requiring explicit n-gram feature extraction.
- Overfitting Concerns: The authors found that increasing filter width, number of filters, or training epochs beyond a certain point did not necessarily improve accuracy and could lead to overfitting, indicated by irregular training behavior and divergence between training and validation losses.
- Orthographic Clues: The analysis suggests that orthographic differences, particularly the absence of diacritics in Chavacano compared to Spanish and Portuguese, play a role in misclassifications.
Key Results
The best-performing model, achieved with 10 filters, a filter width of 5, and trained for 20 epochs, yielded a validation accuracy of 93.76%. The final test accuracy was 93%.
The error analysis revealed that:
- Chavacano is most frequently confused with Cebuano, followed by Portuguese, Spanish, and Hiligaynon.
- Hiligaynon and Cebuano are often confused with each other.
- Short sentence fragments and the lack of diacritics in Chavacano contribute to misclassifications.
Limitations and Future Work
Limitations: * The study is limited to related Creole languages and does not explore broader dialectal LI. * Only CNNs were used as the model architecture. * The mixed-domain nature of the training data might influence performance.
Future Work: * Experiment with other LI algorithms (SVM, Transformers). * Investigate the impact of different dataset domains on performance. * Develop models for identifying Chavacano in natural settings, including code-switched text and shorter sentences (e.g., from social media). * Improve classification accuracy for practical applications like social media sensors for disaster response. * Further research into the orthographical similarities and differences between Chavacano and its related languages.
Summary updated