HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding

Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, Changsheng Xu

Abstract

Visual grounding aims to locate specific image regions based on natural language queries, a task fundamentally dependent on precise cross-modal alignment. While large-scale multimodal pre-trained models like CLIP offer robust foundational knowledge, significant task gaps—specifically data bias and objective differences—hinder their direct application to fine-grained grounding. We present HiVG, a hierarchical multimodal fine-grained modulation framework designed to bridge these gaps efficiently. HiVG introduces a Multi-layer Adaptive Cross-modal Bridge (MACB) to align multi-level features and a Hierarchical Multimodal Low-Rank Adaptation (HiLoRA) paradigm to prevent perceptual error accumulation. Experimental results across five benchmark datasets demonstrate that HiVG achieves state-of-the-art performance while maintaining superior inference speed and energy efficiency.

1. Introduction

Visual grounding is a critical task in computer vision and natural language processing, requiring a system to interpret a linguistic expression and pinpoint the corresponding object or region within an image. Recent advancements have shifted from training models from scratch to leveraging pre-trained uni-modal backbones (e.g., ResNet, BERT). However, these approaches often fail to fully exploit the pre-aligned multimodal knowledge inherent in models like CLIP (Contrastive Language-Image Pre-training).

Despite CLIP’s success in global image-text alignment, applying it to visual grounding reveals two primary challenges:

  1. Data Bias: CLIP is trained on noisy, large-scale web data, often focusing on salient objects rather than the specific, sometimes subtle, entities described in grounding queries.
  2. Objective Difference: CLIP optimizes for global semantic alignment at the final layer, whereas visual grounding requires fine-grained, spatial information often found in intermediate layers.

2. The HiVG Framework

To address these challenges, HiVG employs a hierarchical modulation strategy that adapts CLIP’s features for the precision required in grounding tasks.

2.1 Multi-layer Adaptive Cross-modal Bridge (MACB)

The MACB serves as a communication channel between the linguistic and visual encoders. Unlike standard methods that only utilize the final output of the encoders, MACB extracts features from multiple layers. It employs a sample-agnostic weighting mechanism to determine the relevance of linguistic concepts to specific visual layers. By using cross-attention to inject text-informed guidance into the visual backbone at various depths, MACB ensures the model maintains spatial awareness while processing semantic information.

2.2 Hierarchical Multimodal LoRA (HiLoRA)

Standard Low-Rank Adaptation (LoRA) updates all layers simultaneously, which can lead to the accumulation of perceptual errors from shallow to deep layers. HiLoRA mitigates this by organizing the network into hierarchical groups and adapting them in stages. This progressive adaptation allows the model to refine its visual perception incrementally, ensuring that the fine-grained details necessary for accurate bounding box prediction are preserved and enhanced throughout the network.

2.3 Training Constraints

HiVG is optimized using two key constraints:

  • Contrastive Learning Constraint (CLC): Maintains the global alignment between the image and the query text.
  • Region-Text Contrastive Constraint (RTCC): A token-wise constraint that utilizes the ground-truth bounding box as a mask. This forces the model to specifically correlate query tokens with the visual features located within the target region, enhancing localization accuracy.

3. Experimental Evaluation

The performance of HiVG was evaluated on five prominent benchmarks: RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities.

3.1 Performance and Efficiency

HiVG consistently outperformed both CLIP-based and detector-based state-of-the-art (SOTA) models. Notably, on the RefCOCOg dataset—known for its long and complex queries—HiVG achieved a significant accuracy improvement of over 4% compared to previous methods.

Beyond accuracy, HiVG demonstrates remarkable efficiency:

  • Inference Speed: HiVG is approximately 8.2x faster than TransVG++.
  • Computational Cost: The model operates with only 13% of the FLOPs required by its leading competitors. This efficiency is largely attributed to the use of a lower input resolution (224x224) and the parameter-efficient nature of the HiLoRA architecture.

3.2 Robustness

Analysis shows that HiVG is particularly robust to query complexity. While other models often see a performance drop as sentence length increases, HiVG’s hierarchical modulation allows it to maintain stable and accurate grounding even with intricate linguistic descriptions.

4. Conclusion

HiVG represents a significant step forward in adapting large-scale multimodal models for fine-grained tasks. By introducing a hierarchical approach to both cross-modal bridging and model adaptation, HiVG bridges the gap between global pre-training and local grounding. Future research will explore adaptive selection mechanisms to further optimize the hierarchical stages and extend this framework to broader multimodal applications such as video understanding and image-text retrieval.

Summary updated

Abstract

Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained models to transfer visual/linguistic knowledge separately while ignoring the multimodal corresponding information. Motivated by recent advancements in contrastive language-image pre-training and low-rank adaptation (LoRA) methods, we aim to solve the grounding task based on multimodal pre-training. However, there exists significant task gaps between pre-training and grounding. Therefore, to address these gaps, we propose a concise and efficient hierarchical multimodal fine-grained modulation framework, namely HiVG. Specifically, HiVG consists of a multi-layer adaptive cross-modal bridge and a hierarchical multimodal low-rank adaptation (Hi LoRA) paradigm. The cross-modal bridge can address the inconsistency between visual features and those required for grounding, and establish a connection between multi-level visual and text features. Hi LoRA prevents the accumulation of perceptual errors by adapting the cross-modal features from shallow to deep layers in a hierarchical manner. Experimental results on five datasets demonstrate the effectiveness of our approach and showcase the significant grounding capabilities as well as promising energy efficiency advantages. The project page: https://github.com/linhuixiao/HiVG.
BibTeX
@inproceedings{xiao2024hivg,
  author = {Linhui Xiao and Xiaoshan Yang and Fang Peng and Yaowei Wang and Changsheng Xu},
  title = {{HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681071},
}
Figure