DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Grounding

Ting Liu, Xuyang Liu, Siteng Huang, Honggang Chen, Quanjun Yin, Long Qin, Donglin Wang, Yue Hu

Abstract

Visual grounding (VG) is the task of localizing objects in an image based on textual descriptions. While large-scale models have significantly improved VG performance, they impose a heavy computational burden during fine-tuning. This paper introduces DARA, a novel Parameter-Efficient Transfer Learning (PETL) method designed specifically for VG. DARA utilizes Domain-aware Adapters (DA Adapters) to refine intra-modality representations and Relation-aware Adapters (RA Adapters) with shared weights to bridge the gap between vision and language modalities. Empirical results show that DARA achieves state-of-the-art accuracy while updating only 2.13% of the backbone parameters, outperforming full fine-tuning by 0.81% on average across major benchmarks.

Introduction

Visual Grounding (VG) requires a model to accurately map linguistic phrases to specific spatial regions in an image. As the backbone architectures for these tasks grow in size, traditional full fine-tuning becomes increasingly impractical due to high computational costs and storage requirements.

Comparison between standard full fine-tuning and the DARA paradigm, highlighting the significant reduction in tunable parameters.

While Parameter-Efficient Transfer Learning (PETL) has seen success in natural language processing, applying it to VG is challenging. Standard PETL methods often fail to capture the complex spatial reasoning and cross-modal alignment necessary for grounding. DARA addresses this by introducing specialized adapters that focus on domain-specific refinement and cross-modal relational learning.

Methodology: Domain and Relation Awareness

DARA integrates two types of bottleneck adapters into the pre-trained vision and language backbones of a VG model (e.g., TransVG):

  1. Domain-aware Adapters (DA Adapters): Positioned after the Multi-Head Attention layers, these adapters specialize general pre-trained features (from models like ResNet or BERT) for the specific requirements of the visual grounding domain.
  2. Relation-aware Adapters (RA Adapters): These are inserted in parallel with Feed-Forward Networks. Crucially, RA Adapters share weights between the vision and language branches. This weight-sharing mechanism acts as a bridge, forcing both modalities to learn a shared representation of spatial and relational concepts (e.g., "next to," "inside") early in the processing pipeline.
The DARA framework overview and the detailed architecture of Domain-aware and Relation-aware Adapters.

The overall strategy involves freezing the massive pre-trained backbones and only training these lightweight adapter modules alongside the final modality fusion transformer. This "refine then relate" approach ensures that intra-modality features are optimized before cross-modal interaction occurs.

Experimental Results

The effectiveness of DARA was evaluated on three primary benchmarks: RefCOCO, RefCOCO+ (which lacks absolute location words), and RefCOCOg (which features complex, long-form descriptions).

Efficiency and Performance

DARA demonstrates a remarkable balance between efficiency and accuracy:

  • Parameter Reduction: DARA reduces the number of updated backbone parameters by 97.87%, moving from 151M in a standard TransVG model to just 3.21M.
  • Accuracy Gains: Despite the reduced parameter count, DARA improved average accuracy by 0.81% over full fine-tuning. On the challenging RefCOCO+ (testA) set, DARA achieved 69.83% accuracy, significantly higher than other PETL methods like LoRA (66.53%).

Comparison with Other PETL Methods

Compared to general-purpose PETL methods like LoRA or AdaptFormer, DARA consistently performs better in VG tasks. This superiority is attributed to the RA Adapters' weight-sharing, which specifically targets the cross-modal relational reasoning that standard adapters overlook.

Discussion and Key Insights

  • The Power of Weight Sharing: Sharing weights in RA Adapters serves as a powerful regularizer. It ensures the vision and language branches "align" their understanding of spatial relations, which is critical for grounding tasks where the text describes relative positions.
  • Avoiding Catastrophic Forgetting: By keeping the majority of the backbone frozen, DARA preserves the broad knowledge gained during large-scale pre-training while surgically adapting the model to the VG task. This often leads to better generalization than full fine-tuning, which can suffer from overfitting on smaller datasets.
  • Future Directions: While DARA effectively optimizes the backbones, the modality fusion module remains fully fine-tuned. Future research could explore applying PETL to the fusion stage itself. Additionally, extending the DARA architecture to non-transformer backbones remains a promising area for investigation.

Conclusion

DARA proves that parameter efficiency does not have to come at the cost of performance. By introducing domain-specific refinement and relation-aware weight sharing, DARA provides a scalable and effective solution for fine-tuning large-scale visual grounding models on limited computational budgets.

Summary updated

Abstract

Visual grounding (VG) is a challenging task to localize an object in an image based on a textual description. Recent surge in the scale of VG models has substantially improved performance, but also introduced a significant burden on computational costs during fine-tuning. In this paper, we explore applying parameter-efficient transfer learning (PETL) to efficiently transfer the pre-trained vision-language knowledge to VG. Specifically, we propose \textbf{DARA}, a novel PETL method comprising \underline{\textbf{D}}omain-aware \underline{\textbf{A}}dapters (DA Adapters) and \underline{\textbf{R}}elation-aware \underline{\textbf{A}}dapters (RA Adapters) for VG. DA Adapters first transfer intra-modality representations to be more fine-grained for the VG domain. Then RA Adapters share weights to bridge the relation between two modalities, improving spatial reasoning. Empirical results on widely-used benchmarks demonstrate that DARA achieves the best accuracy while saving numerous updated parameters compared to the full fine-tuning and other PETL methods. Notably, with only \textbf{2.13\%} tunable backbone parameters, DARA improves average accuracy by \textbf{0.81\%} across the three benchmarks compared to the baseline model. Our code is available at \url{https://github.com/liuting20/DARA}.
BibTeX
@inproceedings{liu2024dara,
  author = {Ting Liu and Xuyang Liu and Siteng Huang and Honggang Chen and Quanjun Yin and Long Qin and Donglin Wang and Yue Hu},
  title = {{DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Grounding}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10688132},
}
Figure