TL;DR
Researchers have introduced MultiPedes, the first massive multilingual dataset (10 million pairs) for text-based person retrieval, alongside ViSeLA, a framework that treats images as universal semantic anchors. By moving away from English-centric "bridge" models, ViSeLA achieves state-of-the-art performance and ensures that person search is equally effective across English, Chinese, French, and German.
What Problem Does This Paper Address?
Text-Based Person Retrieval (TBPR) is the task of finding a specific person in a massive gallery of images using only a natural language description (e.g., "A woman in a teal t-shirt and black pants"). This is critical for real-world surveillance and public safety where a query image might not exist.
While English-only TBPR has seen significant progress, multilingual TBPR has lagged behind due to two major hurdles:
- Data Scarcity: There are no large-scale, high-quality datasets for training models on pedestrian descriptions in multiple languages.
- English-Centric Bias: Current multilingual models often use English as a "pivot" language. They align non-English text to English text, and then to images. This forces the linguistic structures of English onto other languages, leading to suboptimal performance for non-English users.
Methods
The authors tackle these problems with a two-pronged approach: a new data pipeline and a novel model architecture.
1. The MultiPedes Dataset Pipeline
To build a high-quality dataset without manual labeling of millions of images, the authors developed a multilingual quality-aware data construction pipeline.
- Consensus-Guided Filtering: They used multiple "domain-specific evaluators" (specialized models) to vote on whether an image and its English description actually matched. If the models disagreed, a general-purpose Vision Question Answering (VQA) model acted as a tie-breaker.
- Scalable Translation: Only the high-quality "vetted" pairs were then translated into Chinese, French, and German using Large Language Models (LLMs), ensuring the semantic meaning remained consistent across all versions.
2. The ViSeLA Framework
The Vision-grounded Semantic Learning and Alignment (ViSeLA) framework introduces two key modules:
- Se-L (Semantic-aware multilingual Learning): This uses a Mixture-of-Experts (MoE) architecture. Instead of routing tokens based on language identity, a "semantic-aware router" sends tokens to specific experts based on their meaning. This forces the model to learn that "shirt" (English) and "hemd" (German) represent the same concept.
- Vi-A (Vision-grounded Cross-modal Alignment): This module uses "semantic probes"—learnable queries that look for specific visual cues (like a red backpack or blue shoes). It aligns multilingual text directly to these visual anchors in the image, bypassing the need for an English "middleman."
Datasets
- MultiPedes (New): 10 million image-text pairs across English, Chinese, French, and German.
- Evaluation Benchmarks: The model was tested on multilingual versions of standard benchmarks: CUHK-PEDES(M), ICFG-PEDES(M), and UFineBench(M).
Key Insights and Ideas
- Images as the Universal Language: By using the image as the semantic anchor (Vi-A module), the model treats all languages as equals. This mitigates the "English dominance" found in previous models.
- Semantic Routing > Language Routing: In MoE architectures, it's common to have "language-specific" experts. This paper argues that "concept-specific" experts are better for retrieval, as they naturally cluster equivalent terms from different languages together in the feature space.
- Quality over Quantity: The consensus-guided filtering stage proved that cleaning up existing noisy datasets is more effective for pretraining than simply scraping more low-quality data.
Key Results
- State-of-the-Art (SOTA): ViSeLA consistently outperformed the previous leading method (Bi-IRRA) across all multilingual benchmarks.
- Balanced Gains: Unlike previous models that saw huge gains in English but marginal improvements in other languages, ViSeLA showed uniform performance boosts. For example, on CUHK-PEDES(M), it improved Rank@1 accuracy by roughly +0.30% across all four languages simultaneously.
- English-Only Excellence: Even when tested on traditional English-only tasks, ViSeLA outperformed specialized English models, proving that multilingual training actually enriches the model's overall semantic understanding.
Limitations and Future Work
- Linguistic Diversity: While the paper covers four major languages, it does not yet address low-resource languages or those with significantly different scripts/structures (e.g., Arabic or Hindi).
- Computational Cost: The Mixture-of-Experts (MoE) approach, while effective, increases the model's parameter count, which could impact deployment on edge devices (like smart cameras).
- Future Directions: The authors suggest that this vision-grounded approach could be extended to other multilingual multimodal tasks beyond person retrieval, such as general image captioning or visual grounding.
Summary updated