Abstract
Visual grounding aims to locate specific image regions based on natural language queries, a task fundamentally dependent on precise cross-modal alignment. While large-scale multimodal pre-trained models like CLIP offer robust foundational knowledge, significant task gaps—specifically data bias and objective differences—hinder their direct application to fine-grained grounding. We present HiVG, a hierarchical multimodal fine-grained modulation framework designed to bridge these gaps efficiently. HiVG introduces a Multi-layer Adaptive Cross-modal Bridge (MACB) to align multi-level features and a Hierarchical Multimodal Low-Rank Adaptation (HiLoRA) paradigm to prevent perceptual error accumulation. Experimental results across five benchmark datasets demonstrate that HiVG achieves state-of-the-art performance while maintaining superior inference speed and energy efficiency.
1. Introduction
Visual grounding is a critical task in computer vision and natural language processing, requiring a system to interpret a linguistic expression and pinpoint the corresponding object or region within an image. Recent advancements have shifted from training models from scratch to leveraging pre-trained uni-modal backbones (e.g., ResNet, BERT). However, these approaches often fail to fully exploit the pre-aligned multimodal knowledge inherent in models like CLIP (Contrastive Language-Image Pre-training).
Despite CLIP’s success in global image-text alignment, applying it to visual grounding reveals two primary challenges:
- Data Bias: CLIP is trained on noisy, large-scale web data, often focusing on salient objects rather than the specific, sometimes subtle, entities described in grounding queries.
- Objective Difference: CLIP optimizes for global semantic alignment at the final layer, whereas visual grounding requires fine-grained, spatial information often found in intermediate layers.
2. The HiVG Framework
To address these challenges, HiVG employs a hierarchical modulation strategy that adapts CLIP’s features for the precision required in grounding tasks.
2.1 Multi-layer Adaptive Cross-modal Bridge (MACB)
The MACB serves as a communication channel between the linguistic and visual encoders. Unlike standard methods that only utilize the final output of the encoders, MACB extracts features from multiple layers. It employs a sample-agnostic weighting mechanism to determine the relevance of linguistic concepts to specific visual layers. By using cross-attention to inject text-informed guidance into the visual backbone at various depths, MACB ensures the model maintains spatial awareness while processing semantic information.
2.2 Hierarchical Multimodal LoRA (HiLoRA)
Standard Low-Rank Adaptation (LoRA) updates all layers simultaneously, which can lead to the accumulation of perceptual errors from shallow to deep layers. HiLoRA mitigates this by organizing the network into hierarchical groups and adapting them in stages. This progressive adaptation allows the model to refine its visual perception incrementally, ensuring that the fine-grained details necessary for accurate bounding box prediction are preserved and enhanced throughout the network.
2.3 Training Constraints
HiVG is optimized using two key constraints:
- Contrastive Learning Constraint (CLC): Maintains the global alignment between the image and the query text.
- Region-Text Contrastive Constraint (RTCC): A token-wise constraint that utilizes the ground-truth bounding box as a mask. This forces the model to specifically correlate query tokens with the visual features located within the target region, enhancing localization accuracy.
3. Experimental Evaluation
The performance of HiVG was evaluated on five prominent benchmarks: RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities.
3.1 Performance and Efficiency
HiVG consistently outperformed both CLIP-based and detector-based state-of-the-art (SOTA) models. Notably, on the RefCOCOg dataset—known for its long and complex queries—HiVG achieved a significant accuracy improvement of over 4% compared to previous methods.
Beyond accuracy, HiVG demonstrates remarkable efficiency:
- Inference Speed: HiVG is approximately 8.2x faster than TransVG++.
- Computational Cost: The model operates with only 13% of the FLOPs required by its leading competitors. This efficiency is largely attributed to the use of a lower input resolution (224x224) and the parameter-efficient nature of the HiLoRA architecture.
3.2 Robustness
Analysis shows that HiVG is particularly robust to query complexity. While other models often see a performance drop as sentence length increases, HiVG’s hierarchical modulation allows it to maintain stable and accurate grounding even with intricate linguistic descriptions.
4. Conclusion
HiVG represents a significant step forward in adapting large-scale multimodal models for fine-grained tasks. By introducing a hierarchical approach to both cross-modal bridging and model adaptation, HiVG bridges the gap between global pre-training and local grounding. Future research will explore adaptive selection mechanisms to further optimize the hierarchical stages and extend this framework to broader multimodal applications such as video understanding and image-text retrieval.
Summary updated