WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs

Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Weidi Xie

WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable progress in processing images and text. However, their ability to perceive and reason across synchronized audio and visual streams—true "omnimodal" understanding—remains largely untested. This paper introduces WorldSense, a comprehensive benchmark comprising 1,662 videos and 3,172 questions designed to evaluate this capability. Our findings reveal a significant performance gap: even top-tier proprietary models like Gemini 1.5 Pro score below 50%, while many open-source audio-visual models fail to outperform random guessing.

Introduction: The Gap in Multimodal Perception

Human interaction with the world is inherently multisensory. We don't just see a car; we hear its engine and the screech of its tires. While current MLLMs excel at static image analysis or silent video captioning, they often lack the ability to synthesize audio-visual cues. Existing benchmarks frequently rely on static images or do not strictly require the integration of audio to answer questions. WorldSense addresses this by providing a rigorous testbed where both modalities are essential for success.

The WorldSense Benchmark

WorldSense is engineered to ensure that neither video nor audio alone provides sufficient information to solve its tasks. By requiring the synthesis of synchronized streams, it tests the limits of current multimodal architectures.

Examples of WorldSense tasks requiring the integration of audio and visual information.

Data Curation and Annotation

The researchers curated high-quality clips from YouTube, specifically targeting content with strong audio-visual correlations. A team of 80 expert annotators developed multiple-choice question-answer (QA) pairs. To maintain high standards, a dual-verification process was employed: human experts verified multimodal necessity, and existing MLLMs were used to filter out trivial questions.

Dataset Composition

The benchmark includes:

  • 1,662 synchronized clips with an average duration of 141 seconds.
  • 3,172 QA pairs covering 8 primary domains and 26 cognitive tasks, including causal reasoning and spatial awareness.
  • Videos sourced from FineVideo and Music-AVQA to ensure diversity in content and musical complexity.

Key Findings and Model Performance

The evaluation of various models—ranging from video-only to specialized audio-visual architectures—yielded several critical insights into the state of the art.

The Integration Paradox

Surprisingly, many open-source models designed for both audio and video performed poorly, often hovering around 25% accuracy (random chance). This suggests that current architectures struggle to effectively fuse these modalities. In contrast, proprietary models led the field, though they still face substantial challenges.

  • Gemini 1.5 Pro: 48.0% accuracy
  • GPT-4o: 42.6% accuracy
  • LLaVA-Video (Video-only): 40.2% accuracy
  • Claude 3.5 Sonnet: 34.8% accuracy
  • Unified-IO-2 / OneLLM: ~23-25% accuracy

Ablation Study

The necessity of multimodal integration was confirmed through ablation studies. For instance, Gemini 1.5 Pro's performance dropped from 48.0% to approximately 34% when restricted to either audio-only or video-only inputs, confirming that the benchmark effectively tests the synthesis of both streams.

Audio vs. Subtitles

The study found that providing models with raw audio significantly outperformed providing text-based subtitles. Subtitles fail to capture paralinguistic features such as tone, rhythm, and environmental sounds, which are often crucial for deep understanding.

Weaknesses in Reasoning

Models across the board struggled with tasks involving spatial reasoning, counting, and emotional intelligence. These tasks require a fine-grained alignment of visual expressions with vocal nuances, a level of integration that current models have yet to master.

Conclusion and Future Directions

WorldSense highlights a massive gap in current AI perception. While proprietary models show promise, the lack of effective audio-visual integration in open-source models is a major hurdle. Future research should focus on early fusion architectures, interleaved training on naturally coupled multimodal data, and incorporating more reasoning-focused training data.

By pushing the boundaries of omnimodal understanding, WorldSense provides a roadmap for developing AI that truly senses the world as humans do.

Summary updated

BibTeX
@inproceedings{hong2026worldsense,
  author = {Jack Hong and Shilin Yan and Jiayin Cai and Xiaolong Jiang and Yao Hu and Weidi Xie},
  title = {{WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=YxsfxAvJv4},
}
Figure