Abstract
Multimodal Large Language Models (MLLMs) have shown remarkable progress in processing images and text. However, their ability to perceive and reason across synchronized audio and visual streams—true "omnimodal" understanding—remains largely untested. This paper introduces WorldSense, a comprehensive benchmark comprising 1,662 videos and 3,172 questions designed to evaluate this capability. Our findings reveal a significant performance gap: even top-tier proprietary models like Gemini 1.5 Pro score below 50%, while many open-source audio-visual models fail to outperform random guessing.
Introduction: The Gap in Multimodal Perception
Human interaction with the world is inherently multisensory. We don't just see a car; we hear its engine and the screech of its tires. While current MLLMs excel at static image analysis or silent video captioning, they often lack the ability to synthesize audio-visual cues. Existing benchmarks frequently rely on static images or do not strictly require the integration of audio to answer questions. WorldSense addresses this by providing a rigorous testbed where both modalities are essential for success.
The WorldSense Benchmark
WorldSense is engineered to ensure that neither video nor audio alone provides sufficient information to solve its tasks. By requiring the synthesis of synchronized streams, it tests the limits of current multimodal architectures.
Data Curation and Annotation
The researchers curated high-quality clips from YouTube, specifically targeting content with strong audio-visual correlations. A team of 80 expert annotators developed multiple-choice question-answer (QA) pairs. To maintain high standards, a dual-verification process was employed: human experts verified multimodal necessity, and existing MLLMs were used to filter out trivial questions.
Dataset Composition
The benchmark includes:
- 1,662 synchronized clips with an average duration of 141 seconds.
- 3,172 QA pairs covering 8 primary domains and 26 cognitive tasks, including causal reasoning and spatial awareness.
- Videos sourced from FineVideo and Music-AVQA to ensure diversity in content and musical complexity.
Key Findings and Model Performance
The evaluation of various models—ranging from video-only to specialized audio-visual architectures—yielded several critical insights into the state of the art.
The Integration Paradox
Surprisingly, many open-source models designed for both audio and video performed poorly, often hovering around 25% accuracy (random chance). This suggests that current architectures struggle to effectively fuse these modalities. In contrast, proprietary models led the field, though they still face substantial challenges.
- Gemini 1.5 Pro: 48.0% accuracy
- GPT-4o: 42.6% accuracy
- LLaVA-Video (Video-only): 40.2% accuracy
- Claude 3.5 Sonnet: 34.8% accuracy
- Unified-IO-2 / OneLLM: ~23-25% accuracy
Ablation Study
The necessity of multimodal integration was confirmed through ablation studies. For instance, Gemini 1.5 Pro's performance dropped from 48.0% to approximately 34% when restricted to either audio-only or video-only inputs, confirming that the benchmark effectively tests the synthesis of both streams.
Audio vs. Subtitles
The study found that providing models with raw audio significantly outperformed providing text-based subtitles. Subtitles fail to capture paralinguistic features such as tone, rhythm, and environmental sounds, which are often crucial for deep understanding.
Weaknesses in Reasoning
Models across the board struggled with tasks involving spatial reasoning, counting, and emotional intelligence. These tasks require a fine-grained alignment of visual expressions with vocal nuances, a level of integration that current models have yet to master.
Conclusion and Future Directions
WorldSense highlights a massive gap in current AI perception. While proprietary models show promise, the lack of effective audio-visual integration in open-source models is a major hurdle. Future research should focus on early fusion architectures, interleaved training on naturally coupled multimodal data, and incorporating more reasoning-focused training data.
By pushing the boundaries of omnimodal understanding, WorldSense provides a roadmap for developing AI that truly senses the world as humans do.
Summary updated