FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification

Zhen Sun, Lei Tan, Yunhang Shen, Chengmao Cai, Xing Sun, Pingyang Dai, Liujuan Cao, Rongrong Ji

TL;DR

FlexiReID introduces a flexible multimodal person re-identification (Re-ID) framework capable of handling arbitrary combinations of four distinct modalities—RGB, infrared, sketch, and text—for pedestrian retrieval. By leveraging an Adaptive Mixture-of-Experts (MoE) mechanism and a novel cross-modal query fusion module, it seamlessly handles missing modalities in real-world deployment scenarios while achieving state-of-the-art performance.

What Problem Does This Paper Address?

Traditional person re-identification (Re-ID) systems rely heavily on standard RGB camera images. However, in real-world surveillance, RGB data degrades significantly due to poor lighting, night conditions, or physical occlusions. Cross-modal Re-ID attempts to solve this by incorporating alternative modalities like infrared (IR) images, hand-drawn sketches, or textual descriptions.

The core limitation of existing cross-modal Re-ID methods is their rigidity. They are typically engineered for specific, fixed pairs (e.g., only text-to-RGB or only IR-to-RGB). In practical scenarios, an investigator might possess a combination of inputs—such as a text description and a quick sketch. Current systems cannot dynamically combine these arbitrary inputs or gracefully handle situations where certain modalities are missing, creating a significant barrier to practical deployment.

Methods

The authors propose FlexiReID, a unified framework built on top of a pre-trained CLIP (ViT-B/16) dual-encoder architecture. To prevent parameter explosion, all three visual modalities (RGB, IR, sketch) share a single image encoder, while a text encoder processes textual descriptions.

To effectively manage these diverse inputs, FlexiReID introduces two novel components:

  1. Adaptive Expert Allocation Mixture of Experts (AEA-MoE): Traditional MoE architectures use a fixed "Top-K" routing mechanism, activating a set number of expert sub-networks regardless of the input. FlexiReID replaces this with an adaptive router governed by a confidence threshold. If the primary expert is highly confident, only that expert is activated. If confidence is low, additional experts are dynamically recruited in descending order of confidence until the threshold is cleared. An entropy-based adaptive loss ensures the system uses the minimum necessary number of experts, optimizing computational efficiency.
  2. Cross-Modal Query Fusion (CMQF): To handle arbitrary query combinations, this module uses a cross-attention mechanism where the query features for one modality are modulated by the sum of the other two. Crucially, it deploys learnable embedding features to substitute for any missing modalities. This allows the network to output a structurally consistent, unified fused feature vector whether the user provides one, two, or all three query modalities.
The network architecture of the proposed FlexiReID framework.

Datasets

Evaluating a truly flexible multi-modal system requires a dataset containing all four modalities simultaneously. Because no such benchmark existed, the authors constructed CIRS-PEDES by expanding four popular Re-ID datasets:

  • CUHK-PEDES, ICFG-PEDES, and RSTPReid: These datasets originally contained only RGB images and text descriptions. The authors used StyleGAN3 to generate corresponding sketch modalities and InfraGAN to synthesize infrared images.
  • SYSU-MM01: This dataset originally featured RGB and IR images. The authors used StyleGAN3 to generate sketches and leveraged GPT-4 to generate detailed textual descriptions from the RGB images.

Key Insights and Ideas

  • Dynamic Resource Allocation: The success of AEA-MoE demonstrates that different modalities and feature complexities require varying amounts of model capacity. Fixed Top-K routing either wastes compute on straightforward inputs or underperforms on highly complex, ambiguous ones.
  • Imputing the Missing via Embeddings: Instead of training separate models for every mathematical combination of inputs ($2^3 - 1 = 7$ modes), using learnable embeddings to "fill in the blanks" allows a single fusion transformer to maintain structural integrity during inference.
  • Modality Synergy: The paper reinforces that multi-modal inputs provide complementary features (e.g., IR provides structural outlines in low light, while text provides high-level semantic colors), drastically reducing ambiguity in matching.

Key Results

FlexiReID was evaluated across seven distinct retrieval tasks ($T\rightarrow R$, $S\rightarrow R$, $IR\rightarrow R$, and their combinations) against various state-of-the-art (SOTA) baselines:

  • Single-Modality Excellence: FlexiReID achieved SOTA or near-SOTA performance even when restricted to single-query modalities. For example, on the ICFG-PEDES text-to-RGB ($T\rightarrow R$) task, it achieved a Rank-1 accuracy of 61.34%.
  • The Power of Combination: The framework's performance scales dramatically when modalities are combined. On the CUHK-PEDES dataset, moving from a Text-only query (69.20% Rank-1) to a combined Text + Sketch + Infrared query ($T+S+IR\rightarrow R$) boosted the Rank-1 accuracy to 88.23%.
  • Routing Strategy Validation: In ablation studies, the proposed adaptive routing mechanism (83.90% average Rank-1) significantly outperformed traditional Top-K routing (80.58% average Rank-1).

Limitations and Future Work

  • Synthetic Data Reliance: A primary limitation is that the CIRS-PEDES dataset relies heavily on synthetically generated sketches and infrared images via generative adversarial networks (GANs). Real-world sensors (like actual thermal cameras) may introduce noise and artifacts not captured by these generative models.
  • Future Directions: The authors suggest that this methodology opens up new avenues for modality-agnostic retrieval. Future extensions could include adapting the framework for video-based Re-ID or incorporating additional real-world sensor data like depth maps or audio.

Summary updated

Abstract

Multimodal person re-identification (Re-ID) aims to match pedestrian images across different modalities. However, most existing methods focus on limited cross-modal settings and fail to support arbitrary query-retrieval combinations, hindering practical deployment. We propose FlexiReID, a flexible framework that supports seven retrieval modes across four modalities: rgb, infrared, sketches, and text. FlexiReID introduces an adaptive mixture-of-experts (MoE) mechanism to dynamically integrate diverse modality features and a cross-modal query fusion module to enhance multimodal feature extraction. To facilitate comprehensive evaluation, we construct CIRS-PEDES, a unified dataset extending four popular Re-ID datasets to include all four modalities. Extensive experiments demonstrate that FlexiReID achieves state-of-the-art performance and offers strong generalization in complex scenarios.
BibTeX
@inproceedings{sun2025flexireid,
  author = {Zhen Sun and Lei Tan and Yunhang Shen and Chengmao Cai and Xing Sun and Pingyang Dai and Liujuan Cao and Rongrong Ji},
  title = {{FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=dewR2augg2},
}
Figure