Panlingual Lexical Translation via Probabilistic Inference

Mausam, Stephen Soderland, Oren EtzioniAAAI 2010 · DOI

Abstract

Researchers at the University of Washington have introduced PanDictionary, a massive, sense-distinguished multilingual dictionary covering over 1,000 languages. By representing existing dictionaries as a "translation graph" and utilizing probabilistic inference to identify "translation circuits," the system successfully inferred millions of new translations absent from any single source. This approach outperforms the English Wiktionary by up to 24 times in scale while maintaining comparable precision.

Introduction: The Data Bottleneck in Translation

In an increasingly globalized world, machine translation (MT) remains a luxury for the majority of the world's 7,000 languages. Statistical MT services like Google Translate support only a small fraction of these languages because they rely on massive "aligned corpora"—identical texts in two different languages—which do not exist for most language pairs.

The fundamental building block for translation is a bilingual dictionary. However, with 49 million possible language pairs, the vast majority of dictionaries simply do not exist. Even when they do, they are often plagued by polysemy: the phenomenon where a single word has multiple meanings. For example, the English word "spring" can refer to a season, a mechanical coil, or a leap. Simple automated translation often suffers from "lexical drift," where a translation path might inadvertently jump from one sense to another (e.g., translating "spring" as a season into a second language, which then translates it back as a coil in a third).

A translation graph illustrating polysemy for the word 'spring' across multiple languages and senses.

Methodology: From Graphs to Circuits

The researchers proposed a three-step pipeline to address these challenges:

  1. The Translation Graph: They aggregated over 630 freely available web dictionaries into a unified graph. In this graph, nodes represent words or phrases, and edges represent translations. Crucially, each edge is assigned a "Sense ID."
  2. Managing Sense ID Inflation: Since different source dictionaries use different IDs for the same concept, the team developed algorithms to merge redundant IDs and identify when they represented the same underlying meaning.
  3. Probabilistic Inference via Translation Circuits: To infer new translations reliably, the system looks for translation circuits. While simple transitivity (if A=B and B=C, then A=C) is risky due to polysemy, a closed loop (if A translates to B, B to C, and C back to A) provides strong probabilistic evidence that all nodes share the same sense.

Two primary algorithms were evaluated: uSENSEUNIFORMPATHS (uSP), which uses random walks to find circuits, and SENSEUNIFORMPATHS (SP). The latter is a more sophisticated version that identifies "ambiguity sets"—instances where words share multiple senses—and prunes those paths to prevent incorrect inferences.

Key Results and Evaluation

The resulting PanDictionary contains over 80,000 distinct senses and translations across more than 1,000 languages. To verify accuracy, the authors recruited native speakers of 14 languages to manually validate thousands of inferred translations.

Comparison of direct vs. inferred translations across different algorithms and precision levels.

Key findings include:

  • Massive Scale: At a precision of 0.9, PanDictionary is 4.5 times larger than the English Wiktionary. At a precision of 0.7, it is 24 times larger.
  • Inference Efficiency: The inference algorithms produced four times as many pairwise translations as were present in the union of all input dictionaries (at 0.8 precision).
  • Language Reach: PanDictionary provides at least 1,000 distinct words for 107 languages, more than doubling the reach of Wiktionary (which hits that threshold for only 49 languages).
  • Precision and Recall: The SP algorithm doubled the number of baseline translations while maintaining a high precision of 0.95.

Discussion and Future Work

While PanDictionary represents a major leap in lexical translation, it focuses on words and phrases rather than full-sentence translation. It does not currently handle grammar, syntax, or the fluid context of speech. Additionally, the system is dependent on its source dictionaries; it cannot "invent" a sense if it is entirely missing from the underlying web data.

Future applications for PanDictionary include serving as a foundation for "knowledge-based" MT systems and enabling cross-lingual image searches, allowing users to find visual results for specific concepts (like a "spring coil") regardless of the language used for the query.

Summary updated

BibTeX
@inproceedings{mausam2010panlingual,
  author = {Mausam and Stephen Soderland and Oren Etzioni},
  title = {{Panlingual Lexical Translation via Probabilistic Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2010},
  doi = {10.1609/aaai.v24i1.7703},
}
Figure