Abstract
Visual grounding (VG) is the task of localizing objects in an image based on textual descriptions. While large-scale models have significantly improved VG performance, they impose a heavy computational burden during fine-tuning. This paper introduces DARA, a novel Parameter-Efficient Transfer Learning (PETL) method designed specifically for VG. DARA utilizes Domain-aware Adapters (DA Adapters) to refine intra-modality representations and Relation-aware Adapters (RA Adapters) with shared weights to bridge the gap between vision and language modalities. Empirical results show that DARA achieves state-of-the-art accuracy while updating only 2.13% of the backbone parameters, outperforming full fine-tuning by 0.81% on average across major benchmarks.
Introduction
Visual Grounding (VG) requires a model to accurately map linguistic phrases to specific spatial regions in an image. As the backbone architectures for these tasks grow in size, traditional full fine-tuning becomes increasingly impractical due to high computational costs and storage requirements.
While Parameter-Efficient Transfer Learning (PETL) has seen success in natural language processing, applying it to VG is challenging. Standard PETL methods often fail to capture the complex spatial reasoning and cross-modal alignment necessary for grounding. DARA addresses this by introducing specialized adapters that focus on domain-specific refinement and cross-modal relational learning.
Methodology: Domain and Relation Awareness
DARA integrates two types of bottleneck adapters into the pre-trained vision and language backbones of a VG model (e.g., TransVG):
- Domain-aware Adapters (DA Adapters): Positioned after the Multi-Head Attention layers, these adapters specialize general pre-trained features (from models like ResNet or BERT) for the specific requirements of the visual grounding domain.
- Relation-aware Adapters (RA Adapters): These are inserted in parallel with Feed-Forward Networks. Crucially, RA Adapters share weights between the vision and language branches. This weight-sharing mechanism acts as a bridge, forcing both modalities to learn a shared representation of spatial and relational concepts (e.g., "next to," "inside") early in the processing pipeline.
The overall strategy involves freezing the massive pre-trained backbones and only training these lightweight adapter modules alongside the final modality fusion transformer. This "refine then relate" approach ensures that intra-modality features are optimized before cross-modal interaction occurs.
Experimental Results
The effectiveness of DARA was evaluated on three primary benchmarks: RefCOCO, RefCOCO+ (which lacks absolute location words), and RefCOCOg (which features complex, long-form descriptions).
Efficiency and Performance
DARA demonstrates a remarkable balance between efficiency and accuracy:
- Parameter Reduction: DARA reduces the number of updated backbone parameters by 97.87%, moving from 151M in a standard TransVG model to just 3.21M.
- Accuracy Gains: Despite the reduced parameter count, DARA improved average accuracy by 0.81% over full fine-tuning. On the challenging RefCOCO+ (testA) set, DARA achieved 69.83% accuracy, significantly higher than other PETL methods like LoRA (66.53%).
Comparison with Other PETL Methods
Compared to general-purpose PETL methods like LoRA or AdaptFormer, DARA consistently performs better in VG tasks. This superiority is attributed to the RA Adapters' weight-sharing, which specifically targets the cross-modal relational reasoning that standard adapters overlook.
Discussion and Key Insights
- The Power of Weight Sharing: Sharing weights in RA Adapters serves as a powerful regularizer. It ensures the vision and language branches "align" their understanding of spatial relations, which is critical for grounding tasks where the text describes relative positions.
- Avoiding Catastrophic Forgetting: By keeping the majority of the backbone frozen, DARA preserves the broad knowledge gained during large-scale pre-training while surgically adapting the model to the VG task. This often leads to better generalization than full fine-tuning, which can suffer from overfitting on smaller datasets.
- Future Directions: While DARA effectively optimizes the backbones, the modality fusion module remains fully fine-tuned. Future research could explore applying PETL to the fusion stage itself. Additionally, extending the DARA architecture to non-transformer backbones remains a promising area for investigation.
Conclusion
DARA proves that parameter efficiency does not have to come at the cost of performance. By introducing domain-specific refinement and relation-aware weight sharing, DARA provides a scalable and effective solution for fine-tuning large-scale visual grounding models on limited computational budgets.
Summary updated