TL;DR
OmniGAIA is a new benchmark and training framework designed to move AI agents beyond simple "look and describe" tasks toward true omni-modal intelligence. By introducing a challenging multi-hop reasoning benchmark and the OmniAtlas agent—which can selectively "look" and "listen" to media while using external tools—the researchers have significantly narrowed the performance gap between open-source models and proprietary giants like Gemini-3-Pro.
What Problem Does This Paper Address?
While we’ve seen an explosion in Multimodal Large Language Models (MLLMs), most are still "bi-modal" (e.g., they can see an image or hear an audio clip, but rarely both at once with deep reasoning). Furthermore, existing benchmarks often focus on simple perception—asking "what is in this video?"—rather than agentic behavior.
Real-world tasks are messy: they require "omni-modal" perception (integrating video, audio, and images), long-horizon reasoning, and the ability to use tools (like web search or code execution) to verify facts. This paper addresses the lack of both a rigorous evaluation for these "omni-modal agents" and a recipe for training open-source models to handle such complexity.
Methods
The authors propose a two-pronged solution: a new way to build benchmarks and a new way to train agents.
1. Omni-modal Event Graph Construction
To create truly difficult tasks, the authors developed a pipeline that:
- Mines signals: Uses models like Gemini-3-Flash to extract fine-grained, time-stamped descriptions of video, audio, and images.
- Builds a graph: Connects these cross-modal entities and events into a complex "event graph."
- Expands with tools: Uses a reasoning agent (DeepSeek-V3.2) to find "missing links" by searching the web or calling other tools, making the final question impossible to answer from the media alone.
- Fuzzifies: Masks key information to force the model to traverse the entire logical path to find the answer.
2. The OmniAtlas Agent
OmniAtlas is a foundation agent featuring two core innovations:
- Active Omni-Modal Perception: Instead of "shoving" an entire 10-minute video into the model (which is token-expensive and lossy), the agent can autonomously call tools like
read_video(t_start, t_end)orread_audio()to selectively inspect segments it finds relevant. - OmniDPO (Fine-Grained Error Correction): Standard training often penalizes a whole trajectory if it fails. OmniDPO identifies the exact first point of failure (e.g., a bad tool call or a perception error) and generates a corrected prefix, creating high-quality preference pairs for optimization.
Datasets
- OmniGAIA (New): The primary contribution for evaluation. It consists of 360 tasks across 9 real-world domains (Geography, Tech, Finance, etc.). It uses data from sources like FineVideo, LongVideoBench, and COCO 2017.
- Training Data: The authors synthesized 2,156 high-quality trajectories using a "hindsight-guided tree exploration" strategy to fine-tune their OmniAtlas models.
Key Insights and Ideas
- Tool Use is the Decisive Bottleneck: The researchers found that simply increasing model size (scaling parameters) provides diminishing returns. For example, a massive 560B parameter model was outperformed by a 30B model that had better tool-use logic.
- The "Cascading Failure" Problem: On hard tasks, open-source models show near-saturated tool misuse (90%+). If the agent fails to acquire the right evidence early via a tool, the entire reasoning chain collapses.
- Native Perception vs. Tool-Based Perception: For strong models (like Gemini), "native" perception (processing media directly) is superior. However, for weaker open-source models, giving them a tool to "describe this audio" can act as a helpful crutch for easy and medium tasks, though it fails on complex, multi-hop reasoning.
- Active Perception is Efficient: By allowing the agent to "look where needed," OmniAtlas avoids the information loss associated with the aggressive downsampling typically required for long-context video.
Key Results
The benchmark proved to be exceptionally difficult:
- Proprietary Lead: Gemini-3-Pro is the state-of-the-art, achieving 62.5% Pass@1.
- Open-Source Gap: The baseline Qwen-3-Omni (30B) only managed 13.3%.
- OmniAtlas Gains: The OmniAtlas training recipe significantly boosted performance. The OmniAtlas-Qwen-3 model reached 20.8% (a +7.5 absolute improvement).
- Small Model Success: On smaller backbones (7B), OmniAtlas improved performance by nearly 3.7x (from 3.6% to 13.3%).
Limitations and Future Work
Despite the improvements, perception remains a persistent bottleneck. Even with OmniAtlas, visual and audio perception error rates remain high (30%–50%). The "Hard" subset of the benchmark remains a formidable challenge for all models.
The authors suggest three future directions:
- Omni-modal Agentic RL: Directly optimizing long-horizon policies using reinforcement learning.
- Scalable Tool Services: Integrating more diverse tools (like the Model Context Protocol) for broader tasks.
- Embodied Agents: Moving these omni-modal capabilities into the physical world for robotics and real-world assistance.
Summary updated