VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition

Chenglin Li, Qianglong Chen, Zhi Li, Feng Tao, Yin ZhangarXiv 2024 · OpenReview

Abstract

Recent advancements in Large Video-Language Models (LVLMs) have led to promising results in multimodal video understanding. However, it remains unclear whether these models possess the cognitive capabilities required for high-level tasks, particularly those involving symbolic and abstract perception. Existing benchmarks typically rely on real-world, annotated videos, which lack control over video content and inherent difficulty, limiting their diagnostic power. To bridge this gap, we propose VideoCogQA, a scalable and fully controllable benchmark inspired by game-world environments, designed to evaluate the cognitive abilities of LVLMs. By generating synthetic videos via a programmatic engine, VideoCogQA allows fine-grained control over visual elements, temporal dynamics, and task difficulty. This approach enables a focused evaluation of video cognitive abilities, independent of prior knowledge from visual scene semantics. The dataset includes 800 videos and 3,280 question-answer pairs, featuring tasks related to abstract concepts, symbolic elements, and multimodal integration, with varying levels of difficulty. Experimental results show that even state-of-the-art (SOTA) models, such as GPT-4o, achieve an average performance of 48.8% on tasks involving abstract concepts. Additionally, performance drops by 15% as task complexity increases, highlighting the challenges LVLMs face in maintaining consistent performance. Through this work, we hope to show the limitations of current LVLMs and offer insights into how they can more effectively emulate human cognitive processes in the future.

Introduction

While current Large Video-Language Models (LVLMs) demonstrate impressive performance on real-world datasets, there is growing concern that they rely on "shortcut learning." For instance, a model might correctly identify a "cooking" action simply by recognizing a kitchen environment, rather than perceiving the actual temporal dynamics of the video. Existing benchmarks often lack the fine-grained control necessary to diagnose whether a model's failure stems from poor reasoning or inadequate perception.

This paper introduces VideoCogQA, a "clean room" benchmark where visual elements are symbolic and abstract. By removing real-world semantic shortcuts, the benchmark forces models to rely on pure cognitive abilities such as counting, spatial reasoning, and temporal sequencing.

Methods and Dataset Design

The researchers developed a programmatic video synthesis pipeline using Python and the Pygame package. This approach provides full controllability over variables like object count, movement speed, and interaction complexity. The resulting VideoCogQA dataset consists of 800 synthetic videos and 3,280 question-answer pairs categorized into three difficulty levels: Easy, Medium, and Difficult.

Overview of the ten scenes in VideoCogQA and their associated cognitive tasks.

The benchmark features 10 distinct scenes inspired by classic games, targeting six core cognitive dimensions: Object Perception (OP), Action Perception (AP), Temporal Reasoning (TR), Spatial Reasoning (SR), Game-environment perception (GP), and Full Modal Perception (FP). For example, "Chameleon Grid" tests object perception in a dynamic grid, while "Maze Runner" evaluates spatial navigation and pathfinding.

Key Results and Insights

The Human-AI Performance Gap

Experimental results reveal a massive performance gap between humans and current AI. While the human baseline for these tasks is 96.5%, the best-performing open-source model, Qwen2-VL-72B, achieved only 53.7% average accuracy. Even proprietary state-of-the-art models like GPT-4o struggled, averaging 48.8% on abstract tasks.

Complexity Collapse

A significant finding is the lack of robustness in current LVLM architectures. As task difficulty increases—for instance, by adding more enemies or increasing movement speed—model performance drops sharply. On average, accuracy decreased by approximately 15% when moving from Easy to Difficult tasks.

Model accuracy across Easy, Medium, and Difficult levels, showing a consistent performance drop as complexity increases.

The Visual Bottleneck

The study suggests that the primary weakness lies in the visual encoders (such as CLIP) rather than the reasoning components. When models were provided with text-based logs of the scenes instead of raw video, their performance improved significantly. This indicates that current encoders are not yet optimized to "see" simple, abstract shapes as effectively as they recognize natural, real-world objects.

Conclusion

VideoCogQA highlights the limitations of current LVLMs in handling symbolic and abstract cognition. The benchmark serves as a diagnostic tool, revealing that even the most advanced models struggle with basic spatial and object perception when stripped of real-world context. Future research should focus on improving the generalization of visual encoders to symbolic data and developing training strategies that emphasize pure cognitive reasoning.

Summary updated

Abstract

Recent advancements in Large Video-Language Models (LVLMs) have led to promising results in multimodal video understanding. However, it remains unclear whether these models possess the cognitive capabilities required for high-level tasks, particularly those involving symbolic and abstract perception. Existing benchmarks typically rely on real-world, annotated videos, which lack control over video content and inherent difficulty, limiting their diagnostic power. To bridge this gap, we propose VideoCogQA, a scalable and fully controllable benchmark inspired by game-world environments, designed to evaluate the cognitive abilities of LVLMs. By generating synthetic videos via a programmatic engine, VideoCogQA allows fine-grained control over visual elements, temporal dynamics, and task difficulty. This approach enables a focused evaluation of video cognitive abilities, independent of prior knowledge from visual scene semantics. The dataset includes 800 videos and 3,280 question-answer pairs, featuring tasks related to abstract concepts, symbolic elements, and multimodal integration, with varying levels of difficulty. Experimental results show that even state-of-the-art (SOTA) models, such as GPT-4o, achieve an average performance of 48.8% on tasks involving abstract concepts. Additionally, performance drops by 15% as task complexity increases, highlighting the challenges LVLMs face in maintaining consistent performance. Through this work, we hope to show the limitations of current LVLMs and offer insights into how they can more effectively emulate human cognitive processes in the future.
BibTeX
@misc{li2024vcbench,
  author = {Chenglin Li and Qianglong Chen and Zhi Li and Feng Tao and Yin Zhang},
  title = {{VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition}},
  year = {2024},
  eprint = {2411.09105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.09105},
}
Figure