TL;DR
This paper introduces MEXA, a novel framework that dynamically aggregates specialized AI "expert" models to achieve general multimodal reasoning. By intelligently selecting and combining the outputs of these experts, MEXA demonstrates superior performance across diverse tasks like video understanding, audio QA, 3D scene analysis, and medical imaging, without requiring task-specific training.
What Problem Does This Paper Address?
The field of AI has seen remarkable progress in multimodal learning, enabling systems to understand and reason across various data types like images, audio, and text. However, building a unified AI system that can flexibly handle the increasing diversity of modalities and the complexity of real-world tasks remains a significant challenge. Existing approaches often rely on monolithic, end-to-end trained models that require extensive fine-tuning for each new task or modality. This approach is not only computationally expensive but also lacks transparency and adaptability. Furthermore, early fusion of multimodal inputs can obscure the reasoning process. The authors aim to address this by creating a modular, training-free framework that can dynamically leverage specialized AI "experts" to tackle a wide range of multimodal reasoning problems.
Methods
MEXA employs a dynamic, multi-expert aggregation strategy. The core idea is to assemble a diverse pool of pre-trained "expert" models, each specialized in a particular modality-task combination.
- Expert Pool: Experts are designed to extract information relevant to their specific skill and modality, converting it into a unified textual representation. These experts cover a wide range of capabilities, from perceptual tasks (image, audio, 3D) to text extraction (OCR) and structured data analysis (charts, tables).
- Expert Selection Module (Router): A multimodal Large Language Model (MLLM) acts as a router. Given a task context and a question, it intelligently selects the most relevant experts from the pool based on the input modalities and the required reasoning skills. This selection process is designed to be dynamic and task-aware.
- Aggregator Module: The outputs from the selected experts (which are textual descriptions) are then fed into a Large Reasoning Model (LRM), acting as an aggregator. This aggregator systematically integrates the complementary information from the experts and performs deep reasoning to generate the final answer. This approach allows for interpretable reasoning and avoids simple heuristic merging.
Crucially, MEXA is training-free. It leverages existing pre-trained experts and an MLLM router, making it highly adaptable and extensible to new modalities or tasks without requiring additional training overhead.
Datasets
The authors evaluate MEXA on several challenging multimodal benchmarks:
- Video-MMMU: For video reasoning, covering diverse domains like medicine, mathematics, and arts.
- MMAU: For audio question answering, spanning speech, music, and general audio.
- SQA3D: For 3D situated reasoning, focusing on understanding spatial relationships in 3D environments.
- M3D-VQA: For medical question answering, specifically using 3D CT scans.
The paper does not introduce new benchmarks but utilizes existing ones to demonstrate MEXA's broad applicability.
Key Insights and Ideas
- Modularity and Specialization are Key: The success of MEXA hinges on the idea that breaking down complex multimodal reasoning into specialized expert tasks and then intelligently combining their outputs is more effective than a monolithic approach.
- Dynamic Expert Selection: The use of an MLLM as a router is crucial for adapting to diverse tasks. It allows the system to "think" about which experts are needed for a given problem, rather than relying on a fixed pipeline.
- Interpretable Reasoning: By generating textual outputs from each expert and then aggregating them, MEXA offers a more transparent reasoning process compared to models that fuse information implicitly.
- Training-Free Adaptability: The ability to achieve strong performance without task-specific fine-tuning is a significant advantage, making MEXA highly scalable and practical for real-world applications.
- Addressing Redundancy and Conflicts: The framework is designed to handle situations where experts might provide similar or conflicting information, using the aggregator to reconcile these outputs.
Key Results
MEXA consistently outperforms strong multimodal baselines across all evaluated benchmarks:
- Video-MMMU: Achieved a 23.6% accuracy gain over the leading open-source LLM and performed comparably to human experts.
- MMAU: Showed substantial improvements over audio LLMs, with gains of 4.1%, 5.9%, and 26.3% in different categories.
- SQA3D: Demonstrated a 2% accuracy gain over state-of-the-art 3D-based LLMs.
- M3D-VQA: Significantly outperformed general-purpose MLLMs, achieving average accuracy improvements of 36.1%, 8.7%, and 1.6% across different medical QA types.
The paper also highlights that GPT-4o performs better as a router than Qwen2.5-VL, and DeepSeek excels as an aggregator due to its strong long-context reasoning capabilities.
Limitations and Future Work
- Frozen Pre-trained Experts: The reasoning fidelity of MEXA is constrained by the inherent capabilities of the frozen pre-trained experts. If an expert makes an error or hallucinates, it can impact the final prediction.
- Expert Quality: The overall prediction quality is affected by the precision and expressiveness of individual experts.
Future work could involve integrating more advanced pre-trained experts to improve accuracy and robustness without altering the core framework. The authors believe their modular approach provides a strong foundation for future research in generalized multimodal reasoning.
Summary updated