Leveraging Task-Specific Pre-Training to Reason across Images and Videos

Arka Sadhu, Ram NevatiaWACV 2024 · OpenReview

Abstract

We explore the task of Reasoning Across Images and Video (RAIV), which requires models to reason on a pair of visual inputs comprising various combinations of images and/or videos. Previous work in this area has been limited to image pairs focusing primarily on the existence and/or cardinality of objects. To address this, we leverage existing datasets with rich annotations to generate semantically meaningful queries about actions, objects, and their relationships. We introduce new datasets that encompass visually similar inputs, reasoning over images, across images and videos, or across videos. Recognizing the distinct nature of RAIV compared to existing pre-training objectives which work on single image-text pairs, we explore task-specific pre-training, wherein a pre-trained model is trained on an objective similar to downstream tasks without utilizing fine-tuning datasets. Experiments with several state-of-the-art pre-trained image-language models reveal that task-specific pre-training significantly enhances performance on downstream datasets, even in the absence of additional pre-training data. We provide further ablative studies to guide future work.

Introduction

Modern Vision-Language Pre-training (VLP) models have achieved remarkable success in tasks involving single image-text pairs. However, a significant gap remains: these models often struggle when required to compare and reason across multiple visual sources. This paper, presented at WACV 2024, introduces the Reasoning Across Images and Video (RAIV) framework to address this limitation, moving beyond simple object detection toward complex semantic reasoning.

The RAIV Challenge

The RAIV task requires a model to process two distinct visual inputs—which can be any combination of images and videos—alongside a natural language query. Unlike existing benchmarks like NLVR2, which focus on object existence or counting in image pairs, RAIV demands an understanding of actions, semantic roles, and temporal relationships.

Overview of RAIV task types, including image-image and video-based reasoning.

Methodology: Bridging the Reasoning Gap

The authors propose a two-part solution involving architectural adaptation and a novel pre-training strategy.

Late-Fusion Architecture

The model utilizes a late-fusion approach based on the METER framework. It processes the two visual inputs ($I_1$ and $I_2$) and the text query by encoding ($I_1$, Text) and ($I_2$, Text) separately through a shared transformer. The resulting [CLS] tokens are concatenated to form a joint representation for final classification.

Task-Specific Pre-training (TSP)

The core innovation is Task-Specific Pre-training. Standard VLP models are typically "blind" to the relationship between two visual streams. TSP introduces a "warm-up" phase where the model learns the logic of comparison using synthetic tasks derived from datasets like COCO. By employing Semantic Role Labeling (SRL), the researchers identify objects and actions to create meaningful comparison queries (e.g., "Do both images contain a person performing the same action?").

New Benchmarks for Multi-Visual Reasoning

To evaluate RAIV, the researchers introduced several new datasets derived from ImSitu (images) and VidSitu (videos):

  • Im-Im, Im-Vid, and Vid-Vid: Benchmarks covering all combinations of image and video pairs using both template-based and LLM-generated queries.
  • IP2P (G): A dataset of visually similar image pairs generated via InstructPix2Pix, designed to test fine-grained discrimination.
An example of a visual input used in the RAIV benchmarks, illustrating complex actions like dolphins jumping.

Key Results and Insights

The experiments yielded several critical findings:

  1. Consistent Performance Gains: The TSP phase improved performance across all RAIV tasks by 1% to 3% over state-of-the-art baselines like METER and ALBEF.
  2. Data Efficiency: Models utilizing TSP achieved competitive results using only 10% of the downstream fine-tuning data compared to models trained on the full dataset without TSP.
  3. The Importance of Hard Negatives: During pre-training, "conditional sampling" (selecting pairs that share at least one object) was more effective than random sampling, as it forced the model to focus on subtle semantic differences rather than obvious mismatches.

Conclusion

This work demonstrates that the "format gap" between single-image pre-training and multi-visual reasoning is a significant hurdle for current models. By introducing Task-Specific Pre-training and a suite of challenging benchmarks, the authors provide a roadmap for developing models capable of sophisticated, cross-source visual reasoning. Future work will look toward increasing temporal resolution in videos and moving from discriminative tasks to generative explanations.

Summary updated

Abstract

We explore the task of Reasoning Across Images and Video (RAIV), which requires models to reason on a pair of visual inputs comprising various combinations of images and/or videos. Previous work in this area has been limited to image pairs focusing primarily on the existence and/or cardinality of objects. To address this, we leverage existing datasets with rich annotations to generate semantically meaningful queries about actions, objects, and their relationships. We introduce new datasets that encompass visually similar inputs, reasoning over images, across images and videos, or across videos. Recognizing the distinct nature of RAIV compared to existing pre-training objectives which work on single image-text pairs, we explore task-specific pre-training, wherein a pre-trained model is trained on an objective similar to downstream tasks without utilizing fine-tuning datasets. Experiments with several state-of-the-art pre-trained image-language models reveal that task-specific pre-training significantly enhances performance on downstream datasets, even in the absence of additional pre-training data. We provide further ablative studies to guide future work.
BibTeX
@inproceedings{sadhu2024leveraging,
  author = {Arka Sadhu and Ram Nevatia},
  title = {{Leveraging Task-Specific Pre-Training to Reason across Images and Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=b5Bn5vrYiD},
}
Figure