Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding

Fengyuan Shi, Ruopeng Gao, Weilin Huang, Limin Wang

TL;DR

Dynamic MDETR introduces a sparse, dynamic decoder for visual grounding that replaces computationally expensive dense self-attention with 2D adaptive sampling. By focusing only on the most informative image patches—guided directly by the text query—the model reduces computational overhead (GFLOPs) by approximately 44% while simultaneously achieving state-of-the-art accuracy across major benchmarks.

What Problem Does This Paper Address?

Visual grounding—the task of localizing an object in an image based on a natural language query—has recently been dominated by multimodal Transformers like TransVG. However, these models typically use an "encoder-only" architecture where every visual token attends to every other visual and textual token.

This leads to two major issues:

  1. Quadratic Complexity: The $O(n^2)$ cost of self-attention makes processing high-resolution images extremely slow and memory-intensive.
  2. Spatial Redundancy: As shown in Figure 1, most pixels in an image (background, sky, or unrelated objects) are irrelevant to a specific text query. Standard Transformers waste significant computation on these "zero-attention" regions.
Fig. 1. Examples illustrating spatial redundancy in visual grounding task. The top pictures are the padded image as model inputs (640 × 640). The red box is the ground truth and the blue box is the predicted box by TransVG [22]. The bottom pictures show the attention map of [REG] token on visual tokens in the last multimodal encoder layer of TransVG. Only a small number of visual tokens have high attention value, and attentions on most of visual tokens in the image is almost zero, showing the severe spatial redundancy in the image for visual grounding task.

Methods

The authors propose Dynamic MDETR, which decouples the grounding process into two distinct phases: a shallow encoder for initial feature fusion and a deep, dynamic decoder for precise localization.

The Dynamic Multimodal Transformer Decoder

The core innovation lies in the decoder, which consists of two alternating sub-modules stacked over $N$ layers:

  • 2D Adaptive Sampling: Instead of processing the entire feature map, the model predicts a set of 2D offsets relative to a reference point (initially the image center). It then uses bilinear interpolation to sample only a small number of informative visual features (e.g., just 36 points or 9% of the feature map).
  • Text Guided Decoding: Unlike standard DETR, which uses learnable "object queries," Dynamic MDETR uses the language features themselves as queries. This ensures that the sampling process is semantically driven by what the user actually asked for.

As the data passes through the decoder layers, the reference point and sampled locations are iteratively refined, gradually homing in on the target object.

Fig. 2. The framework of Dynamic MDETR. Dynamic MDETR consists of four main components, including feature encoder, multimodal transformer encoder, dynamic multimodal transformer decoder and prediction head. Specifically, the input image and referring expression are fed into the visual encoder and language encoder respectively. After getting visual features and language features, we concatenate them into a sequence. Adding a learnable positional embedding, we input the sequence into the multimodal transformer encoder for multimodal fusion. In the dynamic multimodal transformer decoder, we sample a small number of visual features and decode visual information under the guidance of language. Finally, a prediction head directly regresses one bounding box for the given referring expression.

Datasets

The model was evaluated on five standard visual grounding benchmarks:

  • RefCOCO, RefCOCO+, and RefCOCOg: Large-scale datasets derived from MSCOCO.
  • ReferItGame: A dataset of natural scene photographs.
  • Flickr30K Entities: Focused on phrase-to-region correspondences.
  • VQA 2.0: Used to demonstrate the framework's generalization to broader vision-language tasks.

Key Insights and Ideas

  • Sparsity as a Prior: By acknowledging that only a fraction of the image matters for a specific query, the authors turn a computational burden into a localization strength. The sampling acts as a hard-attention mechanism that filters out noise.
  • Language-First Sampling: Fusing the sampling query with language features (as seen in Figure 4) allows the model to "look" for specific attributes (like "yellow" or "holding") in the 2D space, rather than just looking for generic objects.
  • Decoupling Fusion and Localization: The paper argues that encoders are best for broad cross-modal alignment, while decoders are better suited for the "needle-in-a-haystack" task of spatial regression.
2D Adaptive Sampling Workflow

Key Results

  • Efficiency vs. Accuracy: Dynamic MDETR (ResNet-50) outperforms the encoder-only TransVG (ResNet-101) while using significantly fewer GFLOPs. Specifically, it achieves a 44% reduction in GFLOPs in the multimodal transformer component.
  • State-of-the-Art: By integrating a CLIP backbone, the authors built the first one-stage CLIP-empowered grounding framework, setting new performance records. For instance, it achieved 85.97% accuracy on RefCOCO val, a significant jump over previous non-pre-trained methods.
  • Robustness: Qualitative results show that the CLIP-based version is much more robust to diverse language descriptions and complex spatial relationships compared to standard ResNet backbones.

Limitations and Future Work

While the decoder is dynamic, the multimodal encoder remains static and dense. The authors suggest that making the encoder dynamic could yield even greater efficiency gains.

Additionally, the current implementation uses a fixed number of sampled points per layer. Future iterations might explore adaptive point counts, where the model automatically decides to sample fewer points as it becomes more confident in the object's location. Finally, the authors plan to extend this framework to temporal grounding in video, where spatial and temporal redundancy are even more pronounced.

Summary updated

Abstract

Multimodal transformer exhibits high capacity and flexibility to align image and text for visual grounding. However, the existing encoder-only grounding framework (e.g., TransVG) suffers from heavy computation due to the self-attention operation with quadratic time complexity. To address this issue, we present a new multimodal transformer architecture, coined as Dynamic Mutilmodal DETR (Dynamic MDETR), by decoupling the whole grounding process into encoding and decoding phases. The key observation is that there exists high spatial redundancy in images. Thus, we devise a new dynamic multimodal transformer decoder by exploiting this sparsity prior to speed up the visual grounding process. Specifically, our dynamic decoder is composed of a 2D adaptive sampling module and a text guided decoding module. The sampling module aims to select these informative patches by predicting the offsets with respect to a reference point, while the decoding module works for extracting the grounded object information by performing cross attention between image features and text features. These two modules are stacked alternatively to gradually bridge the modality gap and iteratively refine the reference point of grounded object, eventually realizing the objective of visual grounding. Extensive experiments on five benchmarks demonstrate that our proposed Dynamic MDETR achieves competitive trade-offs between computation and accuracy. Notably, using only 9% feature points in the decoder, we can reduce ~44% GFLOPs of the multimodal transformer, but still get higher accuracy than the encoder-only counterpart. In addition, to verify its generalization ability and scale up our Dynamic MDETR, we build the first one-stage CLIP empowered visual grounding framework, and achieve the state-of-the-art performance on these benchmarks.
BibTeX
@article{shi2024dynamic,
  author = {Fengyuan Shi and Ruopeng Gao and Weilin Huang and Limin Wang},
  title = {{Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding}},
  journal = {IEEE TPAMI},
  year = {2024},
  doi = {10.1109/tpami.2023.3328185},
}
Figure