Abstract
Speakers of many different languages use the Internet. A common activity among these users is uploading images and associating these images with words (in their own language) as captions, filenames, or surrounding text. We use these explicit, monolingual, image-to-word connections to successfully learn implicit, bilingual, word-to-word translations. Bilingual pairs of words are proposed as translations if their corresponding images have similar visual features. We generate bilingual lexicons in 15 language pairs, focusing on words that have been automatically identified as physical objects. The use of visual similarity substantially improves performance over standard approaches based on string similarity: for generated lexicons with 1000 translations, including visual information leads to an absolute improvement in accuracy of 8-12% over string edit distance alone.
Introduction
Bilingual lexicon induction—the task of finding word translations across languages—is a cornerstone of machine translation and cross-lingual search. Traditionally, researchers rely on parallel corpora (identical texts translated into two languages). However, parallel data is scarce for most of the world's language pairs. Existing alternatives use monolingual data to find translations based on orthographic similarity (e.g., apple and apfel), contextual similarity, or temporal similarity.
These methods often struggle with rare words or languages that are not closely related. This research proposes using the universality of visual information as a bridge. By leveraging the fact that a "candle" looks the same whether it is called candle (English) or vela (Spanish), the system can find correspondences that text-only methods miss.
Methodology
The proposed pipeline consists of four main stages:
- Identifying Physical Objects: Since abstract concepts like justice lack consistent visual representations, the system first identifies nouns likely to be physical objects using web N-gram patterns like "picture of a [word]."
- Image Acquisition: For each candidate word, the system automatically queries Google Image Search and downloads the top 20 thumbnails.
- Feature Extraction: Images are converted into mathematical vectors using two techniques: Color Histograms (global color distribution) and SIFT Keypoints (local, scale-invariant features like corners or textures). These are clustered into a "visual vocabulary" (bag-of-visual-words).
- Similarity Scoring and Fusion: To compare two words, the system compares their sets of images using the AVGMAX function—averaging the best-matching image pairs. This visual score is then fused with Normalized Edit Distance (NED), which measures spelling similarity.
Experimental Setup
The researchers evaluated the system using two primary datasets:
- 500-word lists: A "clean" set of physical object words across 15 language pairs (including English, Spanish, German, French, Italian, and Dutch).
- 20,000-word lists: A larger, noisier set of words expanded using distributional similarity.
- Evaluation: Results were benchmarked against Google Translate and in-house electronic dictionaries.
Results and Discussion
The study found that visual features provide a signal entirely independent of linguistics, making the approach particularly promising for unrelated language pairs where spelling-based methods fail.
- Accuracy Boost: On the 500-word test, combining SIFT, Color, and NED achieved 48% Top-1 accuracy, a significant improvement over using NED alone (37.3%).
- Large-Scale Performance: In the 20,000-word experiment, adding visual features led to an 8-12% absolute improvement in precision for the top 1,000 proposed translations.
- Complementary Strengths: Visual similarity and orthographic similarity (NED) are highly complementary. Visuals help identify translations for words that look nothing alike (e.g., English fishhook vs. Spanish anzuelo), while NED handles words that might have ambiguous or noisy image results.
Conclusion
While the method is currently limited to concrete nouns and depends on the quality of image search engines, it demonstrates that visual information is a powerful tool for bilingual lexicon induction. Future work aims to move beyond simple objects to more complex concepts through improved image annotation and integration with other semantic indicators to build even more robust bilingual correspondences.
Summary updated