TL;DR
This paper introduces a "transitive translation model" to improve cross-language information retrieval (CLIR) by leveraging web anchor text. It addresses the challenge of translating obscure or new terms by using an intermediate language to bridge gaps where direct translation from anchor text is insufficient.
What Problem Does This Paper Address?
Cross-language information retrieval (CLIR) aims to let users search for information in one language and retrieve documents in another. A major hurdle for CLIR, especially in practical web search, is the lack of comprehensive and up-to-date bilingual dictionaries, particularly for new terminology and proper nouns. Existing methods rely on dictionaries or parallel corpora, both of which have limitations: dictionaries are often incomplete, and large parallel corpora are scarce. While anchor text mining offers a promising approach by treating web links and their associated text as live bilingual resources, it struggles when direct translations don't co-occur frequently enough in anchor text for a specific language pair. This paper tackles this limitation by proposing a method to infer translations indirectly through a third, intermediate language.
Methods
The core contribution is a transitive translation model that combines direct and indirect translation strategies.
- Direct Translation Model: This builds upon the authors' previous work, using web anchor text to extract translation equivalents. It relies on a probabilistic inference model to estimate the likelihood of a translation candidate appearing alongside the source term in anchor texts pointing to the same web page. The model considers the authority of web pages (derived from link structures) to boost the confidence of translations found on more authoritative sites.
- Indirect Translation Model: This is the novel addition. When direct translation fails or is unreliable (e.g., infrequent co-occurrence), this model translates the source query term into an intermediate language (like English, which is common for bridging many language pairs). It then seeks the translation of this intermediate term into the target language. The assumption is that translations in the intermediate language are more readily available and reliable.
- Transitive Translation Model: This model integrates the direct and indirect approaches. It first attempts direct translation. If the confidence score for direct translation is below a threshold, it falls back to the indirect translation method.
Datasets
The paper uses several corpora for experiments:
- Traditional Chinese Web Pages: 1,980,816 pages collected from Taiwan.
- Simplified Chinese Web Pages: 2,179,171 pages collected from China.
- Anchor-text set corpora: Subsets of the above pages were filtered to create corpora containing anchor texts that link to pages with terms in specific language pairs (e.g., Traditional Chinese and English, Simplified Chinese and English, Traditional Chinese and Simplified Chinese).
- Query Logs: Popular query terms were collected from two Taiwanese search engines (Dreamer and GAIS). After filtering, 258 query terms were selected as the primary test set. Traditional Chinese translations of these terms were manually determined to form the source query set.
Key Insights and Ideas
- Bridging the "translation gap" with intermediate languages: The most significant insight is that using an intermediate language can unlock translations that are impossible to find through direct anchor text mining, especially for less common language pairs or obscure terms.
- English as a universal bridge: The paper highlights English's role as a highly probable and effective intermediate language due to its prevalence on the web.
- Anchor text authority matters: The probabilistic model's reliance on web page authority (derived from link structures) is crucial for improving translation confidence.
- Combining methods yields better results: The transitive model, by intelligently combining direct and indirect approaches, significantly improves translation coverage compared to direct methods alone.
Key Results
- The transitive translation model showed significant improvements over the direct translation model.
- For the Traditional Chinese to Simplified Chinese translation task, the top-5 inclusion rate (percentage of queries for which a correct translation was found within the top 5 results) for the direct model was 51.2%.
- The indirect translation model improved this to 60.5%.
- The transitive translation model further boosted the top-5 inclusion rate to 62.0%.
- A combination of the transitive model and a lexicon lookup achieved the highest top-5 inclusion rate of 67.8%.
- Qualitatively, the paper provides examples like translating "西門子" (Siemens) from Traditional Chinese to Simplified Chinese via the intermediate English term "Siemens," which was not directly available through anchor text mining.
Limitations and Future Work
- Reliance on intermediate language availability: The effectiveness of the indirect translation model hinges on the availability of reliable translations in the intermediate language.
- Computational cost: Mining large web corpora and performing transitive translations can be computationally intensive.
- Ambiguity: While the model aims to resolve ambiguity, it's an ongoing challenge in translation.
The authors suggest further investigation into improving the translation accuracy and efficiency of the proposed model. They also note that while the transitive model is particularly useful for language pairs with insufficient anchor texts, exploring other intermediate languages and refining the confidence threshold for switching between direct and indirect translation are potential areas for future work.
Summary updated