Abstract
Social media messages' brevity and unconventional spelling pose a challenge to language identification. We introduce a hierarchical model that learns character and contextualized word-level representations for language identification. Our method performs well against strong baselines, and can also reveal code-switching.
Introduction
Language Identification (LID) is often considered a "solved" problem for long, formal documents. However, social media presents unique challenges: tweets are short, filled with unconventional spellings, emojis, hashtags, and URLs. Furthermore, users often mix languages (code-switching) or use closely related languages that are difficult to distinguish. Traditional LID tools often rely on n-grams and require heavy preprocessing to strip out "noise." Researchers at the University of Washington developed C2V2L ("character to vector to language"), a hierarchical neural model that handles this noise natively and understands the hierarchical structure of language—from characters to words to full sentences.
Methods
The C2V2L model consists of two main stages trained end-to-end:
- Char2vec (The Word Builder): Instead of using a fixed vocabulary, the model looks at the Unicode character sequence of each word. It uses a Convolutional Neural Network (CNN) to extract features from these characters, followed by a residual network layer to produce a continuous "word vector." This allows the model to handle typos and rare words gracefully.
- Sentence-level RNN (The Context Builder): These word vectors are fed into a bidirectional LSTM (Long Short-Term Memory) network. This layer looks at the words in context (both forwards and backwards) to decide the language of the entire tweet.
Crucially, the model requires zero preprocessing. It doesn't strip URLs, hashtags, or casing; it learns which tokens are informative and which are "noise" through its internal gating mechanisms.
Datasets and Evaluation
The model was evaluated on two primary datasets:
- TweetLID: A challenging dataset of ~15,000 tweets in six languages from the Iberian Peninsula (Spanish, Portuguese, Catalan, Galician, English, and Basque), including "ambiguous" and "code-switched" labels.
- Twitter70: A larger, more diverse dataset covering 70 languages from various language families.
Key Insights
A major advantage of the model is its ability to share information between related languages. By analyzing the LSTM's input gates, the authors observed that the model naturally learned to ignore non-informative tokens like URLs and hashtags. Furthermore, because the model generates a language prediction for every word before averaging them, it can detect code-switching without architectural changes.
In a T-SNE visualization of the model's language embeddings, related languages (such as Slavic or North Germanic groups) naturally clustered together, indicating that the model captures linguistic similarities.
Key Results
C2V2L achieved state-of-the-art results on the TweetLID dataset with a macro-averaged F1 score of 77.1, outperforming previous shared-task winners. On the 70-language Twitter70 task, the model achieved a 91.2 F1 score. While a smoothed 5-gram character model slightly outperformed it (93.8), C2V2L remained highly competitive without relying on the massive feature tables required by n-gram models. Additionally, the model's performance improved when trained on a mix of domains (e.g., including Wikipedia data), demonstrating robustness to unconstrained data.
Conclusion
While powerful, the model still struggles with extremely similar language pairs, such as Danish vs. Norwegian or Bosnian vs. Croatian. Future work involves further adapting the C2V2L architecture specifically for word-level code-switching analysis. The authors also suggest that while the model is powerful, smoothed character n-gram models remain a surprisingly tough-to-beat baseline for many LID tasks.
Summary updated