Introduction
This paper introduces a framework for Language-Agnostic Visual-Semantic Embeddings (LAVSE) designed for seamless cross-modal retrieval across multiple languages. By replacing traditional word-based dictionaries with a novel character-level embedding approach called LIWE (Language-Invariant Word Embeddings), the authors have developed a model that is significantly more memory-efficient, robust to linguistic noise, and capable of outperforming state-of-the-art methods while operating up to 30x faster.
Challenges in Cross-Modal Retrieval
Traditional cross-modal retrieval—finding images from text or vice-versa—typically relies on pre-trained word embeddings like GloVe or Word2Vec. This dependency introduces three primary hurdles:
- Memory Bloat: Storing massive dictionaries for multiple languages requires substantial VRAM.
- Language Barriers: Most models are language-specific; adding new languages often necessitates expensive translation or retraining from scratch.
- Brittleness: Word-level models struggle with "Out-of-Vocabulary" (OOV) terms, including slang, informal language, and simple typos.
The LAVSE framework aims to create a language-invariant space where images and descriptions in any language (e.g., English, German, or Japanese) map to the same mathematical neighborhood.
The LAVSE Framework
The proposed framework consists of three primary components that enable efficient, language-agnostic retrieval:
- LIWE (Language-Invariant Word Embeddings): Instead of static lookups, LIWE constructs word embeddings on the fly by concatenating character embeddings and processing them through fully-connected layers. This allows the model to learn the latent meaning of character sequences, making it robust to typos and requiring approximately 50x less memory than standard word vectors.
- Self-Attentive Image Encoder: The model utilizes a Faster R-CNN for object detection followed by a Non-Local Module (NLM). The NLM captures global relationships between detected objects (e.g., a "ball" and a "player") without the heavy computational cost associated with the cross-modal attention layers used in previous state-of-the-art models.
- Hybrid Loss Function: To stabilize character-level training, the authors implemented a curriculum learning strategy. The loss function shifts from "Sum of Hinges" (considering all errors) to "Max of Hinges" (focusing on the hardest errors) as training progresses.
Experimental Setup and Datasets
The model was evaluated on several major benchmarks:
- MS COCO & Flickr30k: Standard English datasets for image-caption retrieval.
- Multi30k: A Flickr30k extension featuring German translations.
- YJ Captions 26K: A Japanese dataset, utilized in a transliterated phonetic Roman alphabet format to facilitate character-level training.
Results and Insights
The LAVSE framework demonstrates that focusing on characters rather than words allows the model to naturally handle linguistic variations and discover shared structural patterns across languages.
Key performance metrics include:
- State-of-the-Art Accuracy: On the MS COCO dataset, the LIWE model achieved a Recall@1 of 73.2% for image-to-text retrieval, surpassing previous leaders like SCAN (70.9%).
- Efficiency: The method is roughly 30x faster at encoding data and calculates similarity matrices up to three orders of magnitude faster than cross-attention-based baselines.
- Multilingual Success: In bilingual experiments (English-German and English-Japanese), the model successfully mapped both languages into a shared space, often outperforming models specifically tuned for a single language.
Conclusion and Future Work
While the model performs exceptionally well on English, German, and Japanese, the authors suggest that future research should explore more diverse language families with non-Roman scripts. Additionally, the language-invariant approach could be extended beyond retrieval to generative tasks such as cross-lingual image captioning or visual question answering.
Summary updated