Abstract
Standard language identification (LID) systems often struggle with ambiguous "false friends"—words like velo, which means "veil" in Spanish but "bike" in French. This paper introduces preference-aware LID, a method that leverages a user’s historical language patterns to disambiguate text. By employing a clever unsupervised training strategy, this approach achieves a massive 22% accuracy boost on challenging, ambiguous cases without requiring labeled user data.
Introduction: The Challenge of Ambiguity
Language Identification is the critical first step in most NLP pipelines. If the language is misidentified, downstream tasks like translation or search fail. Short, ambiguous inputs are a major hurdle; for instance, a user typing "basket" might intend the English word for a container or the French word for a sneaker.
Current LID models are typically "one-size-fits-all," treating every input identically regardless of the user. To resolve linguistic ambiguity, models require context. While a user's history is an excellent proxy for intent, a significant data gap exists: there are no large-scale, public datasets pairing specific users with their historical language distributions and intended labels for ambiguous text.
Methodology
The authors propose two architectures to integrate user "traits" (historical language distributions) into a Transformer-based LID model:
- Revision-Based Model: A "plug-and-play" approach where the model's standard probability output is multiplied by the user's historical language distribution. While intuitive, this can lead to "overconfidence," potentially causing the model to ignore the input text if a user has a very strong language preference.
- Representation-Based Model: A more sophisticated approach that learns "language embeddings." A user's preference is represented as a weighted sum of these embeddings and fed into the model's final layer as an inductive bias. This allows the model to learn an optimal balance between the input text and the user's history.
Unsupervised Training Strategy
To train these models without labeled user data, the authors synthesized "pseudo-users." They sampled training sets from a standard LID corpus based on specific language distributions. Label smoothing was applied during sampling to prevent the model from becoming overly biased toward a user's primary language, ensuring it continues to "listen" to the input text.
Datasets and Evaluation
- Training Data: A massive collection from Wikipedia, W2C, Common Crawl, and Tatoeba covering 21 languages (5 million samples per language).
- U-LID (New Benchmark): The first user-labeled LID test set, containing 11,031 samples mined from Alibaba Translate logs where users manually corrected system predictions. This dataset specifically targets "hard" cases where standard models fail.
- KB-21: A standard LID benchmark used to ensure the new models maintain accuracy on "normal," non-ambiguous text.
Key Results
The preference-aware models demonstrated significant performance gains, particularly on ambiguous text. On the U-LID dataset, the baseline Transformer achieved ~67% accuracy, while the preference-aware models reached 88-89%—an improvement of over 20 percentage points.
Crucially, performance on the standard KB-21 benchmark remained high (~93%), proving that adding user context does not degrade the model's ability to handle clear-cut cases. Furthermore, even the "Revision-based" model without additional training (applying the logic only at inference) reached ~84% on U-LID, highlighting the strength of the user preference signal.
Discussion and Future Work
The study highlights that a user's past behavior is a powerful implicit signal for resolving linguistic "deadlocks." While the Revision-based model is effective with ample history, the Representation-based approach is more robust when data is sparse.
Future research will need to address the "cold start" problem for new users, manage privacy and latency in production environments, and account for dynamic preferences where a user's language choice might shift based on task or location.
Summary updated