TL;DR
This paper explores the surprisingly effective task of identifying a web page's language using only its URL, bypassing the need to download and parse page content. The authors demonstrate that machine learning models, particularly those using word-based features, can achieve high accuracy (up to 96% F-measure) in this challenging domain, even outperforming human evaluators in some cases.
What Problem Does This Paper Address?
When crawling the web, especially for search engines or to meet specific language quotas, it's often desirable to know a page's language before downloading its content. Downloading pages in an undesired language wastes bandwidth and resources. Traditional language identification relies on analyzing the page's text, but this paper tackles the problem of inferring language solely from the URL itself. This is particularly challenging because URLs are often short, may contain non-linguistic elements (like technical terms or random strings), and can be intentionally designed to appear generic (e.g., using .com domains).
Methods
The core of the paper involves applying various machine learning techniques to classify URLs. The authors experimented with:
- Feature Extraction:
- Word-based features: URLs were tokenized into meaningful strings (after removing common URL components and short strings). The frequency of these tokens was used as features.
- N-gram features: Sequences of three characters (trigrams) were extracted from the tokens. This is a standard technique for text classification and was adapted here.
- Custom-made features: This included features derived from top-level domain (TLD) country codes (e.g.,
.defor German), presence in common dictionaries (like OpenOffice), and even custom-trained dictionaries based on the training data. - Machine Learning Algorithms:
- Naive Bayes (NB)
- Decision Trees (DT)
- Relative Entropy (RE)
- Maximum Entropy (ME)
- A simple baseline using only TLDs (ccTLD).
- Classifier Combination: The authors also explored combining multiple classifiers to improve precision or recall.
Datasets
The study utilized three primary datasets:
- Open Directory Project (ODP): Approximately 150k URLs per language, categorized by language in the ODP.
- Search Engine Results (SER): Roughly 100k URLs per language obtained from Microsoft's Live Search, using language-specific search options and additional filtering.
- Web Crawl (WC): A manually labeled random sample of 1,260 URLs from a large web crawl. This dataset was used for testing and human evaluation.
Key Insights and Ideas
- URLs are surprisingly informative: The paper demonstrates that even without page content, URLs contain enough linguistic cues to achieve high language identification accuracy.
- Word-based features excel: Surprisingly, simple word (token) frequencies from the URL outperformed more complex n-gram approaches, especially when sufficient training data was available. This suggests that specific tokens within URLs are strong indicators of language.
- Human performance is a benchmark, but not always the best: While human evaluators performed reasonably well (F-measure of 0.75), the best machine learning models surpassed them (F-measure up to 0.92 on the same dataset). This highlights the power of systematic feature extraction and learning.
- TLDs are a weak signal alone: Relying solely on country-code TLDs (like
.deor.fr) yields high precision but very low recall, making it insufficient for comprehensive language identification. - Domain name memorization is a factor, but not the whole story: The performance of word-based features is partly due to learning common domain names. However, the models still perform well even when domain names aren't seen in training, indicating they learn more general linguistic patterns.
- Training on content can hurt: Counter-intuitively, including page content during training decreased performance. This is attributed to strong URL signals (like country codes) being diluted by common words in the page content that also appear in other languages.
Key Results
- The best performing models achieved an average F-measure of around 90% across datasets and languages.
- On the search engine results dataset, the F-measure reached 96%.
- Recall for the best models was consistently high, around 93% on average.
- Human evaluators achieved an average F-measure of 75% with a recall of 67%.
- The simple ccTLD heuristic achieved an F-measure of 68% with a recall below 60%.
- English was found to be the most difficult language to classify, while Italian was the easiest.
Limitations and Future Work
- Language bias: The study focused on five languages (English, German, French, Spanish, Italian). The performance on other languages, especially those with less distinct URL patterns or different writing systems, is not discussed.
- URL structure variations: The paper acknowledges that the performance might be affected by the specific structure of URLs and the presence of non-linguistic elements.
- Future Work: The authors suggest exploring the use of hyperlink structure (in-links) to further improve language identification, as pages in the same language tend to link to each other. They also propose investigating what other types of information can be deduced from URLs beyond language.
Summary updated