Abstract
While automatic language identification (LID) is highly effective for distinct languages, discriminating between closely related languages and dialects remains a significant challenge. This paper evaluates traditional sparse models against dense neural networks in the context of the Discriminating between Similar Languages (DSL) 2017 shared task. We compare a hierarchical two-layer Support Vector Machine (SVM), a flat SVM, and a multi-task Continuous Bag of Words (CBOW) neural network. Our findings demonstrate that traditional models utilizing character and word $n$-grams significantly outperform neural architectures for this task, with our hierarchical SVM system securing second place overall.
Introduction
Language identification is often considered a solved problem in natural language processing when dealing with typologically distinct languages. However, the task becomes considerably more complex when distinguishing between varieties such as Bosnian, Croatian, and Serbian, or different regional dialects of Spanish and French. These varieties often share substantial vocabulary and orthographic conventions, requiring models to capture subtle linguistic nuances. This study details our participation in the DSL 2017 shared task, where we aimed to build a robust system for 14 language varieties across 6 groups.
Methods
We investigated three primary architectures to address the challenge of similar language discrimination:
- Hierarchical SVM (SVM-1): A two-layer approach where the first linear SVM identifies the broad language group, and the second layer consists of specialized SVMs for intra-group classification. Features included character $n$-grams (1-6) and word $n$-grams (1-2) with tf-idf weighting.
- Flat SVM (SVM-2): A single-layer linear SVM trained to classify all 14 varieties simultaneously using the same feature set as SVM-1.
- Multi-task CBOW (NN): A neural network using a Continuous Bag of Words architecture. To incorporate hierarchical information, we employed multi-task learning, requiring the network to predict both the specific variety and the broader language group.
Dataset
The experiments were conducted on the DSL Corpus Collection (DSLCC), comprising short newspaper texts. The dataset includes 14 varieties organized into six groups: Bosnian-Croatian-Serbian (BCS), Spanish (Argentine, Peninsular, Peruvian), Farsi (Persian, Dari), French (Canadian, Hexagonal), Indonesian/Malay, and Portuguese (Brazilian, European). Each language provided 18,000 training and 2,000 development instances.
Results and Analysis
The hierarchical SVM (SVM-1) emerged as the top-performing model, achieving an overall accuracy of 0.9254. As shown in the performance breakdown by language group, the traditional SVM models consistently outperformed the neural network across all categories.
The performance gap was most notable in the Spanish and BCS groups. While the Indonesian-Malay group was identified with near-perfect accuracy (0.9880), the BCS group proved the most challenging, with an accuracy of 0.8579 for the best model.
Analysis of the confusion matrix for the hierarchical SVM reveals the specific difficulties in dialect discrimination. The Bosnian-Croatian-Serbian group exhibits significant overlap, with Bosnian frequently serving as a "middle ground" confused with both Croatian and Serbian.
Discussion and Conclusion
Our results suggest that for tasks involving high-dimensional but sparse features—such as character $n$-grams in short texts—traditional linear models like SVMs remain superior to dense neural networks. The hierarchical structure of SVM-1 provided a marginal accuracy boost but offered significant computational advantages by reducing the feature space during the second stage of classification. While neural networks might close this gap with extensive hyperparameter tuning or more complex architectures, the efficiency and performance of sparse models make them a compelling choice for discriminating between similar languages.
Summary updated