VideoSetDiff: Identifying and Reasoning Similarities and Differences in Similar Videos

Yue Qiu, Yanjun Sun, Takuma Yagi, Shusaku Egami, Natsuki Miyata, Ken Fukuda, Kensho Hara, Ryusuke SagawaICCV 2025

TL;DR

This paper introduces VideoSetDiff, a new benchmark for evaluating how well AI models can understand and reason about similarities and differences across sets of similar videos. They found that current state-of-the-art multimodal LLMs struggle with these fine-grained, multi-video tasks, highlighting a significant gap in AI's ability to perform human-like visual reasoning.

What Problem Does This Paper Address?

Humans are remarkably adept at observing subtle nuances between similar activities, a skill crucial for learning, skill acquisition, and anomaly detection. Think about learning a new dance move by watching multiple instructors, or a doctor diagnosing a patient by comparing symptoms across several cases. This ability to discern fine-grained similarities and differences across multiple instances of an activity is fundamental to human intelligence. However, most existing AI models, particularly large language models (LLMs) adapted for vision, are trained and evaluated on single videos or images. They lack the capacity for robust multi-video reasoning, making it difficult to tackle real-world applications that inherently involve comparing multiple visual streams. This paper aims to bridge this gap by creating a challenging benchmark and a baseline model to push the boundaries of multi-video understanding.

Methods

The core of this work is the VideoSetDiff dataset and a proposed baseline model called VidSetReasoner.

  • VideoSetDiff Dataset: This benchmark is designed to specifically test multi-video similarity and difference recognition. It consists of two parts:
  • Evaluation Set: This set is built using human annotations, covering nine key aspects (like action attributes, poses, and emotional states) across 27 detailed subcategories. It includes 783 sets of four similar video clips, with 4,859 human-validated question-answer pairs.
  • Training Set: This larger set (3,783 sets) is automatically generated using LLMs to create Q&A pairs and similarity/difference captions, aiming to facilitate fine-tuning for multi-video reasoning.
  • VidSetReasoner Model: This is a simple yet effective baseline designed to incorporate multi-video reasoning.
  • It uses a pre-trained video encoder (specifically from InternVideo2.5) to extract features from each video clip.
  • These features are then projected into the LLM's input space and treated as "prefix tokens." This allows a large language model (LLaMA in this case) to jointly reason over the video features and the textual question.
  • The model is fine-tuned using LoRA on the VideoSetDiff training set, with the video encoder frozen, to efficiently adapt the LLM for multi-video understanding.

Datasets

  • VideoSetDiff (New Benchmark): Introduced in this paper, it comprises 3,783 video sets (783 for evaluation, 3,000 for training) covering instructional and general human activities. It features 76,859 Q&A pairs designed to test fine-grained recognition and cross-video reasoning.
  • HowTo100M [14]: Used for collecting instructional video clips.
  • Panda70M [15]: Used for collecting general human activity video clips.
  • HT-Step [55]: A subset of HowTo100M used for generating the training set.

Key Insights and Ideas

  • Current LLMs are not "video-native": Even advanced multimodal LLMs, when adapted for video, often struggle with tasks requiring nuanced cross-video comparisons. Their strength lies more in single-video understanding or text-based reasoning.
  • Fine-grained details are the bottleneck: Models perform reasonably well on high-level categories (like "action" or "human presence") but falter on subtle details such as sub-action ordering, facial expressions, emotional states, and interpersonal relationships. This suggests a fundamental gap in their pre-training data and architectures.
  • Multi-video reasoning is a distinct challenge: Simply concatenating video features or relying on LLMs to stitch together single-video captions isn't enough. Explicitly modeling cross-video interactions is crucial for effective reasoning.
  • Targeted training matters: The proposed VidSetReasoner, by explicitly incorporating multi-video reasoning and fine-tuning on the VideoSetDiff dataset, shows significant improvements over strong baselines, demonstrating the value of specialized training.

Key Results

  • The VidSetReasoner achieved the highest overall accuracy on the VideoSetDiff benchmark, reaching 68.6%.
  • This model consistently outperformed strong multimodal LLMs like InternVideo2.5 (66.8%) and GPT4o (58.6%) on the VideoSetDiff evaluation set.
  • Models performed better on binary (yes/no) questions than on multiple-choice questions, indicating the increased difficulty of comparative reasoning.
  • Analysis revealed that models struggle most with questions related to human/animal attributes, motivation, time, and viewpoint, highlighting areas where current video understanding is weakest.
  • Increasing the number of sampled frames did not always lead to better performance, suggesting that the quality and representation of frames, rather than just quantity, are critical.

Limitations and Future Work

  • Limitations:
    • Despite improvements, even the best models (including VidSetReasoner) are "far from satisfactory," indicating that robust multi-video reasoning is still a significant challenge.
  • Models still exhibit gaps in handling subtle elements like facial expressions, emotions, and interpersonal relationships.
  • The "caption + GPT-4" pipelines, while showing some reasoning ability, lack the rich visual information needed for accurate recognition.
  • Future Work:
    • The authors suggest that future work should explore broader reasoning capacities, including set-based detail recognition, commonsense inference, causal chains, and integrated multi-modal data across spatiotemporal contexts.
  • They propose that future multimodal LLMs should be designed with architectures and pre-training objectives that explicitly support multi-video understanding.
  • Further fine-tuning datasets and pre-training strategies that emphasize multi-video reasoning are needed.

Summary updated

BibTeX
@inproceedings{qiu2025videosetdiff,
  author = {Yue Qiu and Yanjun Sun and Takuma Yagi and Shusaku Egami and Natsuki Miyata and Ken Fukuda and Kensho Hara and Ryusuke Sagawa},
  title = {{VideoSetDiff: Identifying and Reasoning Similarities and Differences in Similar Videos}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure