DARA: Domain- and Relation-Aware Adapters Make Parameter-Efficient Tuning for Visual Grounding

Ting Liu, Xuyang Liu, Siteng Huang, Honggang Chen, Quanjun Yin, Long Qin, Donglin Wang, Yue Hu

TL;DR

DARA (Domain- and Relation-aware Adapters) is a new parameter-efficient tuning method for visual grounding that updates only 2.13% of a model's backbone parameters while actually outperforming full fine-tuning. By using specialized adapters that share weights across vision and language modalities, DARA enables complex spatial reasoning without the massive computational cost of traditional training.

What Problem Does This Paper Address?

Visual Grounding (VG) is the "Where’s Waldo?" of AI: given an image and a text description (e.g., "the second woman on the left"), the model must draw a bounding box around the correct object.

As models like TransVG grow larger, fine-tuning them becomes a nightmare of computational expense. While Parameter-Efficient Transfer Learning (PETL) methods like LoRA or standard Adapters work wonders in NLP, they often struggle with VG. Why? Because VG isn't just about understanding text or images in isolation; it requires deep spatial reasoning and cross-modal alignment. Standard PETL methods often fail to facilitate enough interaction between the visual and linguistic branches, leading to a "performance gap" compared to full fine-tuning.

Methods

The authors introduce DARA, which inserts two specific types of "bottleneck" adapters into the pre-trained vision and language backbones:

  1. Domain-aware Adapters (DA Adapters): These are inserted after the Multi-Head Attention layers. Their job is to refine the general features learned during large-scale pre-training (like BERT or ResNet features) and "specialize" them for the specific nuances of the visual grounding domain.
  2. Relation-aware Adapters (RA Adapters): These are the secret sauce. Inserted in parallel with Feed-Forward Networks, these adapters share weights between the vision and language branches. This weight-sharing mechanism bridges the gap between modalities early in the pipeline, allowing the model to learn shared relational concepts (like "left of" or "inside") before the final fusion stage.

By freezing the massive pre-trained backbones and only training these tiny adapter modules (along with the final fusion transformer), the researchers drastically reduce the "tunable" footprint of the model.

Datasets

The authors evaluated DARA on three gold-standard visual grounding benchmarks:

  • RefCOCO: A standard dataset for referring expressions.
  • RefCOCO+: A more challenging version that forbids the use of absolute location words (like "left" or "top"), forcing the model to rely on appearance and relative relationships.
  • RefCOCOg: Contains longer, more complex, and "flowery" descriptions, testing the model's ability to handle sophisticated language.

Key Insights and Ideas

  • Weight Sharing as a Bridge: The most significant "aha moment" is that sharing weights in the RA Adapters acts as a powerful regularizer and alignment tool. It forces the vision and language branches to "speak the same language" regarding spatial relations.
  • The "Refine then Relate" Order: The authors found that it is crucial to first refine intra-modality features (via DA Adapters) before attempting cross-modal interaction (via RA Adapters). Doing it the other way around actually hurts performance.
  • Less is More: DARA doesn't just match full fine-tuning; it often beats it. This suggests that fully fine-tuning a massive model on a smaller VG dataset might lead to "catastrophic forgetting" or overfitting, whereas DARA’s surgical updates preserve the rich, general knowledge of the pre-trained backbones while adding just enough task-specific "tuning."

Key Results

  • Efficiency: DARA reduces the number of updated backbone parameters by 97.87% (from 151M down to just 3.21M).
  • Performance: Despite the massive reduction in parameters, DARA improved average accuracy by 0.81% across the three benchmarks compared to the full fine-tuning baseline.
  • Superiority over PETL: DARA significantly outperformed other popular PETL methods like LoRA and AdaptFormer, particularly on the complex RefCOCO+ and RefCOCOg datasets, where spatial reasoning is paramount. For example, on RefCOCO+ (testA), DARA achieved 69.83% accuracy, while the next best PETL method (LoRA) trailed at 66.53%.

Limitations and Future Work

While DARA is highly efficient, the paper notes a few areas for further exploration:

  • Fusion Module: While the backbones are frozen, the "Modality Fusion" transformer is still fully trained. Future work could explore applying PETL techniques to this fusion module as well to save even more parameters.
  • Architecture Specificity: The current implementation is tailored for transformer-based VG models like TransVG. Adapting this "relation-aware" logic to other architectures (like Mamba or pure CNN-based grounders) remains an open question.
  • Scaling: The authors suggest that as foundation models continue to scale toward billions of parameters, DARA-style adapters will become even more essential for making these models usable in real-world, resource-constrained environments.

Summary updated

Abstract

Visual grounding (VG) is a challenging task to localize an object in an image based on a textual description. Recent surge in the scale of VG models has substantially improved performance, but also introduced a significant burden on computational costs during fine-tuning. In this paper, we explore applying parameter-efficient transfer learning (PETL) to efficiently transfer the pre-trained vision-language knowledge to VG. Specifically, we propose \textbf{DARA}, a novel PETL method comprising \underline{\textbf{D}}omain-aware \underline{\textbf{A}}dapters (DA Adapters) and \underline{\textbf{R}}elation-aware \underline{\textbf{A}}dapters (RA Adapters) for VG. DA Adapters first transfer intra-modality representations to be more fine-grained for the VG domain. Then RA Adapters share weights to bridge the relation between two modalities, improving spatial reasoning. Empirical results on widely-used benchmarks demonstrate that DARA achieves the best accuracy while saving numerous updated parameters compared to the full fine-tuning and other PETL methods. Notably, with only \textbf{2.13\%} tunable backbone parameters, DARA improves average accuracy by \textbf{0.81\%} across the three benchmarks compared to the baseline model. Our code is available at \url{https://github.com/liuting20/DARA}.
BibTeX
@inproceedings{liu2024dara,
  author = {Ting Liu and Xuyang Liu and Siteng Huang and Honggang Chen and Quanjun Yin and Long Qin and Donglin Wang and Yue Hu},
  title = {{DARA: Domain- and Relation-Aware Adapters Make Parameter-Efficient Tuning for Visual Grounding}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10688132},
}

Loading related content...

Figure