Language-Agnostic Visual-Semantic Embeddings

Jonatas Wehrmann, Maurício Armani Lopes, Douglas M. Souza, Rodrigo C. Barros

Introduction

This paper introduces a framework for Language-Agnostic Visual-Semantic Embeddings (LAVSE) designed for seamless cross-modal retrieval across multiple languages. By replacing traditional word-based dictionaries with a novel character-level embedding approach called LIWE (Language-Invariant Word Embeddings), the authors have developed a model that is significantly more memory-efficient, robust to linguistic noise, and capable of outperforming state-of-the-art methods while operating up to 30x faster.

Challenges in Cross-Modal Retrieval

Traditional cross-modal retrieval—finding images from text or vice-versa—typically relies on pre-trained word embeddings like GloVe or Word2Vec. This dependency introduces three primary hurdles:

  1. Memory Bloat: Storing massive dictionaries for multiple languages requires substantial VRAM.
  2. Language Barriers: Most models are language-specific; adding new languages often necessitates expensive translation or retraining from scratch.
  3. Brittleness: Word-level models struggle with "Out-of-Vocabulary" (OOV) terms, including slang, informal language, and simple typos.

The LAVSE framework aims to create a language-invariant space where images and descriptions in any language (e.g., English, German, or Japanese) map to the same mathematical neighborhood.

The LAVSE Framework

The proposed framework consists of three primary components that enable efficient, language-agnostic retrieval:

  • LIWE (Language-Invariant Word Embeddings): Instead of static lookups, LIWE constructs word embeddings on the fly by concatenating character embeddings and processing them through fully-connected layers. This allows the model to learn the latent meaning of character sequences, making it robust to typos and requiring approximately 50x less memory than standard word vectors.
  • Self-Attentive Image Encoder: The model utilizes a Faster R-CNN for object detection followed by a Non-Local Module (NLM). The NLM captures global relationships between detected objects (e.g., a "ball" and a "player") without the heavy computational cost associated with the cross-modal attention layers used in previous state-of-the-art models.
  • Hybrid Loss Function: To stabilize character-level training, the authors implemented a curriculum learning strategy. The loss function shifts from "Sum of Hinges" (considering all errors) to "Max of Hinges" (focusing on the hardest errors) as training progresses.
The LAVSE architecture showing the image encoder with NLM and the character-level text processing for multiple languages.

Experimental Setup and Datasets

The model was evaluated on several major benchmarks:

  • MS COCO & Flickr30k: Standard English datasets for image-caption retrieval.
  • Multi30k: A Flickr30k extension featuring German translations.
  • YJ Captions 26K: A Japanese dataset, utilized in a transliterated phonetic Roman alphabet format to facilitate character-level training.

Results and Insights

The LAVSE framework demonstrates that focusing on characters rather than words allows the model to naturally handle linguistic variations and discover shared structural patterns across languages.

Retrieval performance (R@1) for Caption and Image retrieval tasks over training iterations, comparing LIWE against baselines.

Key performance metrics include:

  • State-of-the-Art Accuracy: On the MS COCO dataset, the LIWE model achieved a Recall@1 of 73.2% for image-to-text retrieval, surpassing previous leaders like SCAN (70.9%).
  • Efficiency: The method is roughly 30x faster at encoding data and calculates similarity matrices up to three orders of magnitude faster than cross-attention-based baselines.
  • Multilingual Success: In bilingual experiments (English-German and English-Japanese), the model successfully mapped both languages into a shared space, often outperforming models specifically tuned for a single language.

Conclusion and Future Work

While the model performs exceptionally well on English, German, and Japanese, the authors suggest that future research should explore more diverse language families with non-Roman scripts. Additionally, the language-invariant approach could be extended beyond retrieval to generative tasks such as cross-lingual image captioning or visual question answering.

Summary updated

Abstract

This paper proposes a framework for training language-invariant cross-modal retrieval models. We also introduce a novel character-based word-embedding approach, allowing the model to project similar words across languages into the same word-embedding space. In addition, by performing cross-modal retrieval at the character level, the storage requirements for a text encoder decrease substantially, allowing for lighter and more scalable retrieval architectures. The proposed language-invariant textual encoder based on characters is virtually unaffected in terms of storage requirements when novel languages are added to the system. Our contributions include new methods for building character-level-based word-embeddings, an improved loss function, and a novel cross-language alignment module that not only makes the architecture language-invariant, but also presents better predictive performance. We show that our models outperform the current state-of-the-art in both single and multi-language scenarios. This work can be seen as the basis of a new path on retrieval research, now allowing for the effective use of captions in multiple-language scenarios. Code is available at https://github.com/jwehrmann/lavse.
BibTeX
@inproceedings{wehrmann2019language,
  author = {Jonatas Wehrmann and Maurício Armani Lopes and Douglas M. Souza and Rodrigo C. Barros},
  title = {{Language-Agnostic Visual-Semantic Embeddings}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00590},
}
Figure