Abstract
Most vision-and-language models rely on fixed, pre-extracted visual features from object detectors like Faster R-CNN, preventing the visual representation from adapting to downstream tasks. This paper introduces TxT, a fully crossmodal pipeline that replaces Faster R-CNN with Detection Transformers (DETR/Deformable-DETR), enabling true end-to-end training. By injecting global scene context and optimizing the detector's loss function, TxT outperforms standard frozen-feature baselines on Visual Question Answering (VQA).
Introduction
Multimodal tasks like Visual Question Answering (VQA) require precise alignment between textual semantics and visual perception. Current state-of-the-art models typically employ a BERT-like transformer to reason over word embeddings and image region features. However, these image features are almost exclusively pre-extracted using a frozen Faster R-CNN detector.
The reliance on frozen detectors stems from the non-differentiable operations in Faster R-CNN, such as non-maximum suppression (NMS) and region-of-interest (RoI) pooling, which make end-to-end training with large language models computationally prohibitive. While transformer-based detectors like DETR eliminate NMS and are naturally end-to-end trainable, their naive integration into multimodal pipelines often leads to performance degradation. This work identifies the root causes of this failure and proposes the TxT framework to bridge the gap.
The TxT Architecture
The TxT (Transformer-based crossmodal pipeline) connects a transformer-based object detector directly to a transformer language model. The architecture facilitates the flow of gradients from the VQA head all the way back to the initial CNN backbone, allowing the visual features to be fine-tuned specifically for the reasoning task.
To optimize this integration, three key modifications were introduced to the visual backbone:
- Global Context Injection: Unlike Faster R-CNN, transformer detectors can isolate object features to the point of losing scene-level context. TxT addresses this by applying global average pooling to the detector's encoder output and concatenating this global vector to each object feature.
- Scalable Deformable-DETR: To balance performance and compute, the authors developed a variant that queries the multi-scale attention module using a single feature map scale (stride 16) while maintaining access to all scales. This achieves higher efficiency than standard DETR.
- Loss Function Optimization: Standard Deformable-DETR uses focal loss to prioritize rare objects. TxT reverts to cross-entropy loss to maintain an object distribution more suitable for general VQA tasks.
Experimental Results and Analysis
The model was pre-trained on Visual Genome and COCO, then fine-tuned on the VQAv2 benchmark. A critical finding was that superior object detection performance does not automatically translate to better VQA results. While Deformable-DETR is a stronger detector than DETR, its default focal loss shifts the distribution of detected objects toward smaller, less relevant items for general questioning.
As shown in the distributions above, switching from focal loss to cross-entropy loss in Deformable-DETR stabilizes the object count per image. This adjustment, combined with global context injection, allowed the DETR-based features to finally surpass Faster R-CNN baselines. Furthermore, end-to-end training allows the model to select more objects per image (saturating around 80) and attend to broader, more contextually relevant regions.
Key Performance Metrics
- VQA Accuracy: The end-to-end TxT model achieved 69.93% on VQAv2 test-dev, significantly higher than the 68.84% achieved by the Faster R-CNN baseline.
- End-to-End Gain: Unfreezing the visual pipeline provided a +2.3% accuracy boost compared to using frozen DETR features (69.93% vs 67.60%).
- Efficiency: The "Scalable Deformable-DETR" achieved a 0.9% higher Average Precision (AP) than standard DETR while requiring only 10% more compute and training in 50 epochs instead of 500.
Conclusion
TxT demonstrates that end-to-end training of the entire visual-linguistic pipeline is not only possible but highly beneficial for reasoning tasks. By addressing the specific architectural needs of transformer detectors—namely global context and appropriate object distributions—TxT overcomes the limitations of traditional frozen-feature pipelines. While computationally intensive, the performance gains suggest that end-to-end crossmodal learning is a vital direction for future multimodal research.
Summary updated