Champion Solution for the WSDM2023 Toloka VQA Challenge

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, Tong LuarXiv 2023 · DOI · OpenReview

Abstract

The WSDM2023 Toloka VQA Challenge presents a unique difficulty in computer vision: locating objects implicitly mentioned in interrogative questions. This paper details the winning solution developed by researchers from Nanjing University and Shanghai AI Laboratory. By integrating a unified multi-modal backbone (Uni-Perceiver) with a task-specific ViT-Adapter and a DINO detection head, the team successfully bridged the gap between visual question answering and high-precision localization. Their approach, which emphasizes semantic consistency and global attention, secured first place on the competition leaderboard with a private test score of 76.347 IoU.

Introduction

Standard computer vision tasks typically separate linguistic reasoning from spatial localization. Visual Question Answering (VQA) focuses on generating textual responses to image-based queries, while Visual Grounding (VG) involves locating objects based on explicit descriptive phrases (e.g., "the brown bear"). The WSDM2023 Toloka VQA Challenge introduces a hybrid task that requires both: inferring an object from an interrogative question and then precisely locating it within the image.

The Toloka VQA Challenge

The core difficulty of the Toloka challenge lies in the implicit nature of the target object. Unlike traditional grounding, where the target is named, the model must first solve a reasoning problem. For instance, given the question "What does it use to breathe?", the model must infer the answer is "nose" before attempting to find the bounding box for that specific feature on the animal.

Comparison of VQA, Visual Grounding, and the Toloka Challenge tasks.

This "middle-ground" task requires a deeper level of cross-modal alignment than standard benchmarks, as the model must map abstract functional descriptions to concrete visual pixels.

Proposed Methodology

The winning architecture moves away from traditional "two-tower" models, which process images and text separately, in favor of a more integrated, unified approach.

1. Unified Backbone: Uni-Perceiver

The team utilized Uni-Perceiver, a multi-modal transformer designed to handle different data types using a single shared set of parameters. This allows for early-stage interaction between the question and the image, facilitating better contextual understanding of the query before localization begins.

2. Spatial Enhancement: ViT-Adapter

While Vision Transformers (ViT) are powerful, they often lack the spatial inductive biases inherent in convolutional neural networks. To address this, the researchers added a ViT-Adapter. This module introduces spatial priors to the plain ViT backbone, significantly improving its ability to perform high-precision localization without requiring a full re-training of the massive backbone.

The integrated architecture featuring Uni-Perceiver, ViT-Adapter, and DINO.

3. Detection and Refinement

Rather than using a standard VQA head, the task was framed as a single-class object detection problem. The team employed DINO, a state-of-the-art DETR-like detector, to predict the final bounding boxes. To further refine the model, they implemented:

  • Paraphrase Augmentation: Using a T5 language model to generate synonymous questions, making the model robust to linguistic variations.
  • Auxiliary Loss: A Semantic FPN head was used to provide pixel-level supervision, helping the model distinguish the target object from the background.

Results and Discussion

The proposed solution achieved dominant results, ranking 1st place overall. On the public test set, the model reached 77.5 IoU, and it maintained a high performance of 76.347 IoU on the private test set.

Key Insights

  • Semantic Consistency: A critical finding was that pre-training on RefCOCO (declarative text) actually decreased performance by 0.5 IoU. Conversely, pre-training on the GQA dataset—which uses interrogative questions similar to the challenge—provided a 0.9 IoU boost. This highlights the importance of matching the linguistic style of pre-training data to the target task.
  • Global Context: Replacing windowed attention with global attention across all layers yielded a +1.0 IoU improvement. This suggests that resolving implicit references requires the model to consider the entire image context simultaneously.
  • Scaling and TTA: Moving from a base to a large backbone (Uni-Perceiver-L) provided the single largest gain (+2.9 IoU), while Test-Time Augmentation (TTA) added a final +0.9 IoU to secure the win.

Conclusion

The success of the NJU IMAGINE LAB solution demonstrates that unified multi-modal architectures, when augmented with spatial adapters and task-specific detection heads, are highly effective for complex reasoning-based localization. Future work may focus on narrowing the semantic gap between different linguistic styles and optimizing these heavy models for real-time applications.

Summary updated

Abstract

In this report, we present our champion solution to the WSDM2023 Toloka Visual Question Answering (VQA) Challenge. Different from the common VQA and visual grounding (VG) tasks, this challenge involves a more complex scenario, i.e. inferring and locating the object implicitly specified by the given interrogative question. For this task, we leverage ViT-Adapter, a pre-training-free adapter network, to adapt multi-modal pre-trained Uni-Perceiver for better cross-modal localization. Our method ranks first on the leaderboard, achieving 77.5 and 76.347 IoU on public and private test sets, respectively. It shows that ViT-Adapter is also an effective paradigm for adapting the unified perception model to vision-language downstream tasks. Code and models will be released at https://github.com/czczup/ViT-Adapter/tree/main/wsdm2023.
BibTeX
@misc{gao2023champion,
  author = {Shengyi Gao and Zhe Chen and Guo Chen and Wenhai Wang and Tong Lu},
  title = {{Champion Solution for the WSDM2023 Toloka VQA Challenge}},
  year = {2023},
  eprint = {2301.09045},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.09045},
}
Figure