Abstract
Practical cloud-edge deployment of Cross-Modal Person Re-identification (CM-ReID) faces significant hurdles due to the fragmented nature of specialized models for diverse modalities. While Multi-Modal Large Language Models (MLLMs) offer a path toward unification, adapting them for efficient edge deployment remains a challenge. We present MLLMEmbed-ReID, a unified framework that leverages a powerful cloud-based MLLM as a teacher to train a lightweight, edge-native student. By identifying a "low-rank phenomenon" in the MLLM's feature space, we introduce a novel Adaptive SVD Distillation strategy. This approach employs Principal Component Mapping (PCM) and Feature Relation (FR) losses to preserve critical discriminative information. Our framework achieves state-of-the-art performance across multiple CM-ReID benchmarks, providing a robust solution for deploying high-level intelligence on resource-constrained devices. The code and models will be open-sourced soon.
Introduction
Person Re-identification (ReID) is a critical task in surveillance and security, requiring the matching of individuals across non-overlapping camera views. Modern requirements have evolved toward Cross-Modal ReID (CM-ReID), where queries and gallery images may belong to different modalities, such as RGB, infrared (IR), sketch, or text descriptions.
Current deployment strategies are often fragmented, requiring separate, heavy models for each modality pair (e.g., IR-to-RGB vs. Text-to-RGB). While MLLMs possess the inherent cross-modal understanding to unify these tasks, their massive scale prevents direct edge deployment. Previous attempts at distillation, such as VQA-based retrieval or textual distillation, often suffer from hallucination issues or inefficient knowledge transfer.
Methodology
The MLLMEmbed-ReID framework consists of a two-tier cloud-edge architecture designed for unified cross-modal alignment and efficient knowledge transfer.
Cloud-Based Teacher: Unified MLLM
We adapt a foundational MLLM (Qwen2-VL) into a unified ReID backbone. Instead of traditional task-specific heads, we utilize instruction-based prompting to guide the model. For instance, the model is prompted to "Generate the image's embedding, focusing on age, gender, clothing..." This forces the MLLM to project diverse modalities—RGB, infrared, sketch, and text—into a single, shared embedding space. The model is fine-tuned using a hierarchical Low-Rank Adaptation (LoRA-SFT) strategy under a holistic cross-modal alignment objective.
Edge-Based Student: Adaptive SVD Distillation
To bridge the gap between the massive cloud teacher and a lightweight edge student (CLIP ViT-L/14), we introduce Adaptive SVD Distillation. This strategy is motivated by the discovery of a "low-rank phenomenon" in the MLLM's feature space, where a small number of principal components capture the majority of discriminative information.
The distillation process utilizes two primary losses:
- Principal Component Mapping (PCM) Loss: This loss explicitly guides the student to learn the top $k$ principal components of the teacher's features, ensuring the most vital information is prioritized.
- Feature Relation (FR) Loss: This loss preserves the structural relationships between features, allowing the student to capture the "long-tail" of the feature space without the computational burden of full-vector mimicry.
Experimental Results
The framework was evaluated on the newly constructed QrCM-ReID dataset, which includes RGB, IR, sketch, and text modalities, as well as standard benchmarks like LLCM and SYSU-MM01.
Performance Benchmarks
- Cloud Model: Achieved state-of-the-art (SOTA) results, notably reaching 92.87% Rank-1 accuracy and 90.28% mAP on the CUHK-PEDES IR → R task.
- Edge Model: The distilled student significantly outperformed existing specialized models. On the same IR → R task, it achieved 89.14% Rank-1 and 85.23% mAP. In text-to-RGB tasks, it showed marked improvements in mAP (e.g., 58.90% on ICFG-PEDES).
Edge Deployment
The student model was successfully deployed on a TWOWIN TW-T208 edge device. It demonstrated a real-time processing latency of approximately 194ms, translating to roughly 5.14 samples per second, proving its feasibility for real-world smart camera applications.
Conclusion
MLLMEmbed-ReID successfully unifies the fragmented landscape of Cross-Modal ReID. By leveraging the power of MLLMs and a novel SVD-based distillation strategy, we enable the deployment of unified, high-performance ReID models on edge devices. This framework not only simplifies the deployment pipeline but also sets new performance standards for cross-modal retrieval.
Summary updated