Language-agnostic visual-semantic embeddings

Jônatas Wehrmann, Douglas Matos de Souza, Mauricio Armani, Rodrigo C. BarrosOpenReview 2021 · OpenReview

Abstract

This paper proposes a framework for training language-invariant cross-modal retrieval models. We also introduce a novel character-based word-embedding approach, allowing the model to project similar words across languages into the same word-embedding space. In addition, by performing cross-modal retrieval at the character level, the storage requirements for a text encoder decrease substantially, allowing for lighter and more scalable retrieval architectures. The proposed language-invariant textual encoder based on characters is virtually unaffected in terms of storage requirements when novel languages are added to the system. Our contributions include new methods for building character-level-based word-embeddings, an improved loss function, and a novel cross-language alignment module that not only makes the architecture language-invariant, but also presents better predictive performance. We show that our models outperform the current state-of-the-art in both single and multi-language scenarios. This work can be seen as the basis of a new path on retrieval research, now allowing for the effective use of captions in multiple-language scenarios.

Introduction

Most current Visual-Semantic Embedding (VSE) models—designed to facilitate bidirectional search between images and text—are heavily English-centric. This bias stems from the scarcity of high-quality, large-scale datasets for languages other than English. Traditional attempts to bridge this gap often rely on machine translation modules, which are computationally expensive, slow, and prone to error propagation.

This research addresses the challenge of developing a multimodal retrieval system that operates across multiple languages without requiring massive labeled datasets for each. The goal is to project images and sentences from any language into a single, shared multimodal embedding space where semantically similar concepts are clustered together, regardless of the source language.

Methodology: Cross-Language Multimodal Retrieval (CLMR)

The authors propose the CLMR (Cross-Language Multimodal Retrieval) framework, which shifts the focus from word-level to character-level processing to achieve language invariance.

General framework for visual-semantic embedding (VSE) mapping images and text into a shared space.

Core Components

  • Character-Level Encoders (CharCNN, CHAIN-VSE, LIWE): By processing raw characters instead of whole words, these models reduce the semantic gap and increase robustness to noise, such as typos.
  • LIWE (Language-Invariant Word Embeddings): This novel approach projects similar words from different languages into the same vector space. By leveraging shared character patterns or roots, the system can recognize that words in different languages refer to the same underlying concept.
  • ADAPT Module: A "query-aware" visual representation module. Unlike static image embeddings, ADAPT uses the text query to filter the most relevant visual features, acting as a highly efficient attention mechanism.
Detailed architecture of the character-level encoders and cross-language alignment module.

Optimization

The framework utilizes an Exponentially Weighted Ranking Loss ($J_{WE}$). While standard loss functions often struggle with "hard negatives," this new function provides stable gradients early in training and exponentially increases the weight of hard-contrastive pairs as the model matures, leading to superior convergence.

Datasets

The research was validated using several large-scale benchmarks:

  • MS COCO: The standard benchmark for English image-text matching.
  • YJCaptions: A Japanese dataset for which the authors generated custom splits specifically for retrieval tasks.
  • Cross-lingual corpora: Utilized to align representations across disparate languages.

Key Insights and Results

Moving from word-level to character-level encoding provides a massive win for scalability. Storage requirements for the text encoder remain virtually unchanged even as new languages are added. Remarkably, these character-level models can outperform standard LSTM and GRU-based models while using up to 1,000x fewer parameters.

The proposed models achieved state-of-the-art (SOTA) performance across all tested scenarios. The new loss function alone accounted for up to a 20% improvement in recall values compared to previous standard approaches. Furthermore, the ADAPT module provides the benefits of attention-based strategies with significantly faster training and inference speeds.

Discussion and Future Work

While the paper demonstrates impressive results for language-agnosticism, the primary focus remains on languages that share similar alphabets or structures. Future work may extend these methods to more diverse linguistic families.

The impact of this work extends beyond retrieval into broader multimodal tasks like Visual Question Answering (VQA) and Text-to-Image synthesis. By leveraging existing large-scale English datasets, this framework helps democratize AI, enabling researchers to train high-quality models in their native languages.

Summary updated

Abstract

This paper proposes a framework for training language-invariant cross-modal retrieval models. We also introduce a novel character-based word-embedding approach, allowing the model to project similar words across languages into the same word-embedding space. In addition, by performing cross-modal retrieval at the character level, the storage requirements for a text encoder decrease substantially, allowing for lighter and more scalable retrieval architectures. The proposed language-invariant textual encoder based on characters is virtually unaffected in terms of storage requirements when novel languages are added to the system. Our contributions include new methods for building character-level-based word-embeddings, an improved loss function, and a novel cross-language alignment module that not only makes the architecture language-invariant, but also presents better predictive performance. We show that our models outperform the current state-of-the-art in both single and multi-language scenarios. This work can be seen as the basis of a new path on retrieval research, now allowing for the effective use of captions in multiple-language scenarios. Code is available at https://github. com/jwehrmann/lavse.
BibTeX
@misc{wehrmann2021language,
  author = {Jônatas Wehrmann and Douglas Matos de Souza and Mauricio Armani and Rodrigo C. Barros},
  title = {{Language-agnostic visual-semantic embeddings}},
  year = {2021},
  url = {https://openreview.net/forum?id=wsWsruHa0AA},
  howpublished = {OpenReview},
}
Figure