Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval

Xinyu Zhou, Fan Li, Shasha Chen, Min Cao, Ahtamjan Ahmat, Qiaoming Zhu, Mang YearXiv 2026 · DOI · arXiv

TL;DR

Researchers have introduced MultiPedes, the first massive multilingual dataset (10 million pairs) for text-based person retrieval, alongside ViSeLA, a framework that treats images as universal semantic anchors. By moving away from English-centric "bridge" models, ViSeLA achieves state-of-the-art performance and ensures that person search is equally effective across English, Chinese, French, and German.

What Problem Does This Paper Address?

Text-Based Person Retrieval (TBPR) is the task of finding a specific person in a massive gallery of images using only a natural language description (e.g., "A woman in a teal t-shirt and black pants"). This is critical for real-world surveillance and public safety where a query image might not exist.

While English-only TBPR has seen significant progress, multilingual TBPR has lagged behind due to two major hurdles:

  1. Data Scarcity: There are no large-scale, high-quality datasets for training models on pedestrian descriptions in multiple languages.
  2. English-Centric Bias: Current multilingual models often use English as a "pivot" language. They align non-English text to English text, and then to images. This forces the linguistic structures of English onto other languages, leading to suboptimal performance for non-English users.
Figure 1: (a) Multilingual TBPR, where person images are re- trieved with texts in multiple languages. (b) Existing method adopts a shared text encoder for all languages and relies on an English-centered knowledge transfer paradigm for cross- lingual cross-modal alignment. (c) Our ViSeLA framework introduces Se-L module for multilingual learning and Vi-A module for vision-grounded cross-modal alignment.

Methods

The authors tackle these problems with a two-pronged approach: a new data pipeline and a novel model architecture.

1. The MultiPedes Dataset Pipeline

To build a high-quality dataset without manual labeling of millions of images, the authors developed a multilingual quality-aware data construction pipeline.

  • Consensus-Guided Filtering: They used multiple "domain-specific evaluators" (specialized models) to vote on whether an image and its English description actually matched. If the models disagreed, a general-purpose Vision Question Answering (VQA) model acted as a tie-breaker.
  • Scalable Translation: Only the high-quality "vetted" pairs were then translated into Chinese, French, and German using Large Language Models (LLMs), ensuring the semantic meaning remained consistent across all versions.
Figure 2: Multilingual quality-aware data construction pipeline consisting of two stages: (1) data curation with consensus-guided filtering, and (2) multilingual text translation. In the first stage, consensus-guided filtering leverages both domain-specific and domain-agnostic evaluators to identify and remove noisy samples from large-scale image–text pairs collected from public datasets, resulting in higher-quality data. In the second stage, large language models are employed to perform machine translation at scale, generating multilingual descriptions and constructing the MultiPedes dataset.

2. The ViSeLA Framework

The Vision-grounded Semantic Learning and Alignment (ViSeLA) framework introduces two key modules:

  • Se-L (Semantic-aware multilingual Learning): This uses a Mixture-of-Experts (MoE) architecture. Instead of routing tokens based on language identity, a "semantic-aware router" sends tokens to specific experts based on their meaning. This forces the model to learn that "shirt" (English) and "hemd" (German) represent the same concept.
  • Vi-A (Vision-grounded Cross-modal Alignment): This module uses "semantic probes"—learnable queries that look for specific visual cues (like a red backpack or blue shoes). It aligns multilingual text directly to these visual anchors in the image, bypassing the need for an English "middleman."
Overview of the ViSeLA framework, highlighting the Se-L and Vi-A modules.

Datasets

  • MultiPedes (New): 10 million image-text pairs across English, Chinese, French, and German.
  • Evaluation Benchmarks: The model was tested on multilingual versions of standard benchmarks: CUHK-PEDES(M), ICFG-PEDES(M), and UFineBench(M).

Key Insights and Ideas

  • Images as the Universal Language: By using the image as the semantic anchor (Vi-A module), the model treats all languages as equals. This mitigates the "English dominance" found in previous models.
  • Semantic Routing > Language Routing: In MoE architectures, it's common to have "language-specific" experts. This paper argues that "concept-specific" experts are better for retrieval, as they naturally cluster equivalent terms from different languages together in the feature space.
  • Quality over Quantity: The consensus-guided filtering stage proved that cleaning up existing noisy datasets is more effective for pretraining than simply scraping more low-quality data.
Figure 3: Comparison of COMET-Wiki score distributions between MultiPedes and COCO-35L [23]. The scores are av- eraged over three target languages for each sample.

Key Results

  • State-of-the-Art (SOTA): ViSeLA consistently outperformed the previous leading method (Bi-IRRA) across all multilingual benchmarks.
  • Balanced Gains: Unlike previous models that saw huge gains in English but marginal improvements in other languages, ViSeLA showed uniform performance boosts. For example, on CUHK-PEDES(M), it improved Rank@1 accuracy by roughly +0.30% across all four languages simultaneously.
  • English-Only Excellence: Even when tested on traditional English-only tasks, ViSeLA outperformed specialized English models, proving that multilingual training actually enriches the model's overall semantic understanding.

Limitations and Future Work

  • Linguistic Diversity: While the paper covers four major languages, it does not yet address low-resource languages or those with significantly different scripts/structures (e.g., Arabic or Hindi).
  • Computational Cost: The Mixture-of-Experts (MoE) approach, while effective, increases the model's parameter count, which could impact deployment on edge devices (like smart cameras).
  • Future Directions: The authors suggest that this vision-grounded approach could be extended to other multilingual multimodal tasks beyond person retrieval, such as general image captioning or visual grounding.

Summary updated

Abstract

Multilingual Text-Based Person Retrieval (TBPR) aims to retrieve individuals with texts in multiple languages. Despite its practical importance, it has been overlooked by existing research, which mainly focuses on English-only retrieval. The only prior attempt at multilingual TBPR suffers from (1) the absence of a large-scale, multilingual pretraining dataset and (2) a suboptimal model architecture for multilingual text encoding and cross-modal alignment, which employs a shared encoder that overlooks linguistic heterogeneity and relies on English as a pivot for cross-modal alignment, introducing English-centric bias. For this, we contribute MultiPedes, the first large-scale, pedestrian-centric multilingual pretraining dataset. It contains 10 million image-text pairs across four languages, constructed by a multilingual quality-aware data construction pipeline. We also propose ViSeLA (Vision-grounded Semantic Learning and Alignment), a novel framework featuring: (1) a semantic-aware multilingual learning module that extends the mixture-of-experts architecture by routing language-invariant semantics to specialized experts for improved multilingual feature learning; and (2) a vision-grounded cross-modal alignment module that employs a semantic probe to capture fine-grained semantics across modalities, enabling direct cross-modal alignment without relying on a pivot language. Extensive experiments on both multilingual and English-only TBPR datasets demonstrate the effectiveness of ViSeLA. The dataset and code are available at https://github.com/Flame-Chasers/ViSeLA
BibTeX
@misc{zhou2026vision,
  author = {Xinyu Zhou and Fan Li and Shasha Chen and Min Cao and Ahtamjan Ahmat and Qiaoming Zhu and Mang Ye},
  title = {{Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval}},
  year = {2026},
  eprint = {2611.15328},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15328},
}
Figure