Hierarchical Character-Word Models for Language Identification

Aaron Jaech, George Mulcaire, Shobhit Hathi, Mari Ostendorf, Noah A. Smith

Abstract

Language Identification (LID) for short, informal social media text remains a significant challenge due to brevity, slang, and frequent code-switching. We present C2V2L (Character to Vector to Language), a hierarchical neural network architecture that builds word representations directly from Unicode characters using Convolutional Neural Networks (CNNs) and processes them in context via a Bidirectional LSTM. Our model achieves state-of-the-art performance on multilingual Twitter datasets, demonstrating robustness to noise and the ability to perform word-level language detection.

Introduction

While Language Identification is often considered a solved problem for long, formal documents, social media presents unique hurdles. Tweets are characterized by brevity, unconventional spellings, emojis, and hashtags. Furthermore, users frequently mix multiple languages within a single post, a phenomenon known as code-switching. Traditional n-gram models often struggle with these short bursts of informal text, especially when distinguishing between closely related languages like Spanish and Galician.

This research introduces a model that avoids manual feature engineering and aggressive text cleaning, instead learning directly from raw character sequences to capture both morphology and syntax.

Methods: The C2V2L Architecture

The C2V2L model operates through a two-stage hierarchical process designed to bridge the gap between character-level noise and sentence-level context.

  1. Char2vec (Word Representation): The model treats Unicode characters as the primary input. It employs two layers of CNNs to capture local character patterns (e.g., trigrams). A Residual Network layer then refines these features into a single "word vector." This allows the model to derive the meaning of a word even if it contains typos or has never been seen in the training set.
  2. Sentence-level RNN (Contextual Analysis): The resulting sequence of word vectors is fed into a Bidirectional LSTM. This layer analyzes words in relation to their neighbors, capturing the linguistic "flow." A final softmax layer outputs the probability of the tweet belonging to a specific language.
The C2V2L architecture, showing the hierarchical flow from character embeddings to word vectors and sentence-level classification.

By explicitly recognizing word boundaries, C2V2L learns how characters form words (morphology) and how those words form sentences (syntax), a distinction often lost in models that treat a tweet as a single character string.

Experimental Results

The model was evaluated on two primary datasets: TweetLID (six Iberian languages, including code-switched data) and Twitter70 (70 languages to test scalability).

On the TweetLID dataset, C2V2L achieved a macro-averaged F1 score of 76.2, outperforming both the previous shared task winners and standard tools like langid.py. The model showed particular strength in handling "ambiguous" and "code-switched" categories, which are traditionally difficult for LID systems.

Performance comparison on the TweetLID dataset, where C2V2L achieves the highest average F1 score across various languages.

In broader testing on the 70-language Twitter70 dataset, C2V2L maintained high performance (91.2 F1), significantly bettering langid.py (87.9). Additionally, word-level experiments in Spanish-English code-switching scenarios yielded over 94% F1, proving the model's utility for granular linguistic analysis.

Discussion and Conclusion

A key insight from this work is the model's inherent robustness to noise. The LSTM's input gates automatically learn to ignore non-informative tokens like URLs and punctuation. Furthermore, by training on all languages simultaneously, the network learns a "universal" character representation that aids in identifying related languages.

Despite these successes, challenges remain in distinguishing between highly mutually intelligible languages in extremely short contexts, such as Bosnian vs. Croatian. Future work will focus on refining the architecture for non-whitespace segmented languages (like Chinese) and further optimizing for complex code-switching environments.

Summary updated

Abstract

Social media messages' brevity and unconventional spelling pose a challenge to language identification. We introduce a hierarchical model that learns character and contextualized word-level representations for language identification. Our method performs well against strong base- lines, and can also reveal code-switching.
BibTeX
@inproceedings{jaech2016hierarchical,
  author = {Aaron Jaech and George Mulcaire and Shobhit Hathi and Mari Ostendorf and Noah A. Smith},
  title = {{Hierarchical Character-Word Models for Language Identification}},
  booktitle = {SocialNLP @ EMNLP},
  year = {2016},
  url = {https://openreview.net/forum?id=tbbxijVw2o},
}
Figure