OmniGAIA: Towards Native Omni-Modal AI Agents

Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Shijian Wang, Guanting Dong, Jiajie Jin, Hao Wang, Yinuo Wang, Ji-Rong Wen, Yuan Lu, Zhicheng Dou

TL;DR

OmniGAIA is a new benchmark and training framework designed to move AI agents beyond simple "look and describe" tasks toward true omni-modal intelligence. By introducing a challenging multi-hop reasoning benchmark and the OmniAtlas agent—which can selectively "look" and "listen" to media while using external tools—the researchers have significantly narrowed the performance gap between open-source models and proprietary giants like Gemini-3-Pro.

What Problem Does This Paper Address?

While we’ve seen an explosion in Multimodal Large Language Models (MLLMs), most are still "bi-modal" (e.g., they can see an image or hear an audio clip, but rarely both at once with deep reasoning). Furthermore, existing benchmarks often focus on simple perception—asking "what is in this video?"—rather than agentic behavior.

Real-world tasks are messy: they require "omni-modal" perception (integrating video, audio, and images), long-horizon reasoning, and the ability to use tools (like web search or code execution) to verify facts. This paper addresses the lack of both a rigorous evaluation for these "omni-modal agents" and a recipe for training open-source models to handle such complexity.

Methods

The authors propose a two-pronged solution: a new way to build benchmarks and a new way to train agents.

1. Omni-modal Event Graph Construction

To create truly difficult tasks, the authors developed a pipeline that:

  • Mines signals: Uses models like Gemini-3-Flash to extract fine-grained, time-stamped descriptions of video, audio, and images.
  • Builds a graph: Connects these cross-modal entities and events into a complex "event graph."
  • Expands with tools: Uses a reasoning agent (DeepSeek-V3.2) to find "missing links" by searching the web or calling other tools, making the final question impossible to answer from the media alone.
  • Fuzzifies: Masks key information to force the model to traverse the entire logical path to find the answer.

2. The OmniAtlas Agent

OmniAtlas is a foundation agent featuring two core innovations:

  • Active Omni-Modal Perception: Instead of "shoving" an entire 10-minute video into the model (which is token-expensive and lossy), the agent can autonomously call tools like read_video(t_start, t_end) or read_audio() to selectively inspect segments it finds relevant.
  • OmniDPO (Fine-Grained Error Correction): Standard training often penalizes a whole trajectory if it fails. OmniDPO identifies the exact first point of failure (e.g., a bad tool call or a perception error) and generates a corrected prefix, creating high-quality preference pairs for optimization.

Datasets

  • OmniGAIA (New): The primary contribution for evaluation. It consists of 360 tasks across 9 real-world domains (Geography, Tech, Finance, etc.). It uses data from sources like FineVideo, LongVideoBench, and COCO 2017.
  • Training Data: The authors synthesized 2,156 high-quality trajectories using a "hindsight-guided tree exploration" strategy to fine-tune their OmniAtlas models.

Key Insights and Ideas

  • Tool Use is the Decisive Bottleneck: The researchers found that simply increasing model size (scaling parameters) provides diminishing returns. For example, a massive 560B parameter model was outperformed by a 30B model that had better tool-use logic.
  • The "Cascading Failure" Problem: On hard tasks, open-source models show near-saturated tool misuse (90%+). If the agent fails to acquire the right evidence early via a tool, the entire reasoning chain collapses.
  • Native Perception vs. Tool-Based Perception: For strong models (like Gemini), "native" perception (processing media directly) is superior. However, for weaker open-source models, giving them a tool to "describe this audio" can act as a helpful crutch for easy and medium tasks, though it fails on complex, multi-hop reasoning.
  • Active Perception is Efficient: By allowing the agent to "look where needed," OmniAtlas avoids the information loss associated with the aggressive downsampling typically required for long-context video.

Key Results

The benchmark proved to be exceptionally difficult:

  • Proprietary Lead: Gemini-3-Pro is the state-of-the-art, achieving 62.5% Pass@1.
  • Open-Source Gap: The baseline Qwen-3-Omni (30B) only managed 13.3%.
  • OmniAtlas Gains: The OmniAtlas training recipe significantly boosted performance. The OmniAtlas-Qwen-3 model reached 20.8% (a +7.5 absolute improvement).
  • Small Model Success: On smaller backbones (7B), OmniAtlas improved performance by nearly 3.7x (from 3.6% to 13.3%).

Limitations and Future Work

Despite the improvements, perception remains a persistent bottleneck. Even with OmniAtlas, visual and audio perception error rates remain high (30%–50%). The "Hard" subset of the benchmark remains a formidable challenge for all models.

The authors suggest three future directions:

  1. Omni-modal Agentic RL: Directly optimizing long-horizon policies using reinforcement learning.
  2. Scalable Tool Services: Integrating more diverse tools (like the Model Context Protocol) for broader tasks.
  3. Embodied Agents: Moving these omni-modal capabilities into the physical world for robotics and real-world assistance.

Summary updated

Abstract

Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.
BibTeX
@misc{li2026omnigaia,
  author = {Xiaoxi Li and Wenxiang Jiao and Jiarui Jin and Shijian Wang and Guanting Dong and Jiajie Jin and Hao Wang and Yinuo Wang and Ji-Rong Wen and Yuan Lu and Zhicheng Dou},
  title = {{OmniGAIA: Towards Native Omni-Modal AI Agents}},
  year = {2026},
  eprint = {2602.22897},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.22897},
}
Figure