Abstract
The WSDM2023 Toloka VQA Challenge presents a unique difficulty in computer vision: locating objects implicitly mentioned in interrogative questions. This paper details the winning solution developed by researchers from Nanjing University and Shanghai AI Laboratory. By integrating a unified multi-modal backbone (Uni-Perceiver) with a task-specific ViT-Adapter and a DINO detection head, the team successfully bridged the gap between visual question answering and high-precision localization. Their approach, which emphasizes semantic consistency and global attention, secured first place on the competition leaderboard with a private test score of 76.347 IoU.
Introduction
Standard computer vision tasks typically separate linguistic reasoning from spatial localization. Visual Question Answering (VQA) focuses on generating textual responses to image-based queries, while Visual Grounding (VG) involves locating objects based on explicit descriptive phrases (e.g., "the brown bear"). The WSDM2023 Toloka VQA Challenge introduces a hybrid task that requires both: inferring an object from an interrogative question and then precisely locating it within the image.
The Toloka VQA Challenge
The core difficulty of the Toloka challenge lies in the implicit nature of the target object. Unlike traditional grounding, where the target is named, the model must first solve a reasoning problem. For instance, given the question "What does it use to breathe?", the model must infer the answer is "nose" before attempting to find the bounding box for that specific feature on the animal.
This "middle-ground" task requires a deeper level of cross-modal alignment than standard benchmarks, as the model must map abstract functional descriptions to concrete visual pixels.
Proposed Methodology
The winning architecture moves away from traditional "two-tower" models, which process images and text separately, in favor of a more integrated, unified approach.
1. Unified Backbone: Uni-Perceiver
The team utilized Uni-Perceiver, a multi-modal transformer designed to handle different data types using a single shared set of parameters. This allows for early-stage interaction between the question and the image, facilitating better contextual understanding of the query before localization begins.
2. Spatial Enhancement: ViT-Adapter
While Vision Transformers (ViT) are powerful, they often lack the spatial inductive biases inherent in convolutional neural networks. To address this, the researchers added a ViT-Adapter. This module introduces spatial priors to the plain ViT backbone, significantly improving its ability to perform high-precision localization without requiring a full re-training of the massive backbone.
3. Detection and Refinement
Rather than using a standard VQA head, the task was framed as a single-class object detection problem. The team employed DINO, a state-of-the-art DETR-like detector, to predict the final bounding boxes. To further refine the model, they implemented:
- Paraphrase Augmentation: Using a T5 language model to generate synonymous questions, making the model robust to linguistic variations.
- Auxiliary Loss: A Semantic FPN head was used to provide pixel-level supervision, helping the model distinguish the target object from the background.
Results and Discussion
The proposed solution achieved dominant results, ranking 1st place overall. On the public test set, the model reached 77.5 IoU, and it maintained a high performance of 76.347 IoU on the private test set.
Key Insights
- Semantic Consistency: A critical finding was that pre-training on RefCOCO (declarative text) actually decreased performance by 0.5 IoU. Conversely, pre-training on the GQA dataset—which uses interrogative questions similar to the challenge—provided a 0.9 IoU boost. This highlights the importance of matching the linguistic style of pre-training data to the target task.
- Global Context: Replacing windowed attention with global attention across all layers yielded a +1.0 IoU improvement. This suggests that resolving implicit references requires the model to consider the entire image context simultaneously.
- Scaling and TTA: Moving from a base to a large backbone (Uni-Perceiver-L) provided the single largest gain (+2.9 IoU), while Test-Time Augmentation (TTA) added a final +0.9 IoU to secure the win.
Conclusion
The success of the NJU IMAGINE LAB solution demonstrates that unified multi-modal architectures, when augmented with spatial adapters and task-specific detection heads, are highly effective for complex reasoning-based localization. Future work may focus on narrowing the semantic gap between different linguistic styles and optimizing these heavy models for real-time applications.
Summary updated