LViT-Net: a domain generalization person re-identification model combining local semantics and multi-feature cross fusion

Xintong Hu, Peishun Liu, Xuefang Wang, Peiyao Wu, Ruichun TangVis. Comput. Ind. Biomed. Art 2025 · OpenReview

TL;DR

This paper introduces LViT-Net, a novel dual-branch network for domain generalization person re-identification (ReID). By effectively fusing local spatial details with global semantic context through a multi-feature cross-fusion mechanism, LViT-Net significantly improves the ability of ReID models to generalize to unseen environments and conditions, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Person re-identification (ReID) is a crucial task for visual surveillance, aiming to identify individuals across different cameras. While supervised and semi-supervised methods achieve high accuracy within specific domains, they often fail when deployed in new, unseen domains. This "domain generalization" challenge arises from significant variations in illumination, viewpoints, camera angles, and image quality, leading to high intra-class variability and low inter-class similarity in pedestrian images. Existing methods, often relying on single feature extraction architectures (like CNNs or Transformers), struggle to simultaneously capture both fine-grained local details and overarching global semantic information, thus limiting their robustness and generalization capabilities.

Methods

LViT-Net tackles this problem with a dual-branch parallel network architecture.

  • Local Branch: This branch focuses on extracting detailed local features. It employs a series of local feature blocks that utilize depthwise convolutions for efficiency and a novel Local Multi-Scale Feature Fusion (LMSF) module. LMSF combines channel and spatial attention mechanisms to effectively fuse low-level and high-level local features, enhancing robustness and suppressing irrelevant details.
  • Global Branch: This branch leverages a Vision Transformer (ViT) backbone to capture global semantic representations. The ViT processes images in stages, and features from these stages are fed into the fusion module.
  • Dual Feature Cross Fusion (DFCF): This is the core innovation. DFCF is a module that operates between the local and global branches at each stage. It uses an attention mechanism to selectively fuse local features with global semantic information. Specifically, it projects the global cls_token as a query and local features as keys and values. This allows the model to identify and leverage the most relevant local features that correspond to the global context, creating a dynamic balance between detailed and holistic information. The DFCF module is applied multiple times (optimized to four times) to iteratively refine the fused features.

The fused features from both branches are then passed through a classifier for person identification.

Datasets

The paper evaluates LViT-Net on four large-scale person ReID datasets:

  • Market1501 (M)
  • DukeMTMC-reID (D)
  • MSMT17 (MS)
  • CUHK03-NP (C)

Experiments are conducted using both single-source (training on one dataset, testing on another) and multi-source (training on a combination of datasets, testing on the remaining one) protocols to rigorously assess domain generalization capabilities.

Key Insights and Ideas

  • The Synergy of Local and Global: The paper strongly emphasizes that robust person ReID, especially in a domain-generalized setting, requires a sophisticated interplay between fine-grained local details (like clothing patterns, posture) and high-level global semantics (like overall body shape, context). Neither alone is sufficient.
  • Multi-Scale Fusion is Crucial for Local Features: The LMSF module highlights that simply extracting local features isn't enough; fusing them across different scales is key to capturing the full richness of pedestrian appearance and making them robust to variations.
  • Attention-Based Cross-Fusion for Domain Invariance: The DFCF module's ability to use global context to guide the selection and fusion of relevant local features is a critical step towards learning domain-invariant representations. It's not just about having both types of features, but about how they inform each other.
  • Iterative Fusion Benefits Performance: The ablation study on the number of DFCF applications shows that multiple rounds of cross-fusion significantly improve performance, suggesting a progressive refinement of features.

Key Results

LViT-Net demonstrates superior performance across various single-source and multi-source ReID benchmarks.

  • Single-Source: On the Market1501→CUHK03-NP setting, LViT-Net achieved 26.3% Rank-1 and 26.2% mAP, outperforming the ViT-based model PAT by 0.9% and 0.2%. Similar improvements were observed across other single-source settings.
  • Multi-Source: In the M+D+C→MS setting, LViT-Net achieved 48.7% Rank-1 and 23.3% mAP, outperforming PAT by 3.1% and 1.7%.
  • Robustness: The model shows strong robustness to noise (Gaussian and salt-and-pepper) and environmental variations (lighting, random erasing), with performance degradation being less severe than for baseline methods.
  • Visualization: Grad-CAM visualizations confirm that LViT-Net focuses more accurately on pedestrian features, even across different viewpoints and conditions.

Limitations and Future Work

  • Computational Cost: The authors acknowledge that the reliance on ViT in the global branch leads to a relatively high computational cost, which might impact real-time applications.
  • Occlusion Handling: The current model might not perform optimally on datasets with significant occlusions, as standard ReID datasets were primarily used.
  • Future Directions:
    • Further optimizing generalization performance on datasets like CUHK03.
  • Investigating performance on datasets with occlusions.
  • Exploring text-to-image cross-modal person ReID.
  • Reducing computational cost for real-time deployment.

Summary updated

Abstract

In the task of domain generalization person re-identification (ReID), pedestrian image features exhibit significant intra-class variability and inter-class similarity. Existing methods rely on a single feature extraction architecture and struggle to capture both global context and local spatial information, resulting in weaker generalization to unseen domains. To address this issue, an innovative domain generalization person ReID method–LViT-Net, which combines local semantics and multi-feature cross fusion, is proposed. LViT-Net adopts a dual-branch encoder with a parallel hierarchical structure to extract both local and global discriminative features. In the local branch, the local multi-scale feature fusion module is designed to fuse local feature units at different scales to ensure that the fine-grained local features at various levels are accurately captured, thereby enhancing the robustness of the features. In the global branch, the dual feature cross fusion module fuses local features and global semantic information, focusing on critical semantic information and enabling the mutual refinement and matching of local and global features. This allows the model to achieve a dynamic balance between detailed and holistic information, forming robust feature representations of pedestrians. Extensive experiments demonstrate the effectiveness of LViT-Net. In both single-source and multi-source comparison experiments, the proposed method outperforms existing state-of-the-art methods.
BibTeX
@inproceedings{hu2025lvit,
  author = {Xintong Hu and Peishun Liu and Xuefang Wang and Peiyao Wu and Ruichun Tang},
  title = {{LViT-Net: a domain generalization person re-identification model combining local semantics and multi-feature cross fusion}},
  booktitle = {Vis. Comput. Ind. Biomed. Art},
  year = {2025},
  url = {https://openreview.net/forum?id=VjFNDPBLL5},
}
Figure