MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation

Shoubin Yu, Yue Zhang, Ziyang Wang, Jaehong Yoon, Mohit Bansal

Abstract

Combining pre-trained expert models offers substantial potential for scalable multimodal reasoning, but building a unified framework remains challenging due to the increasing diversity of input modalities and task complexity. We introduce MEXA, a training-free framework that performs modality- and task-aware aggregation of multiple expert models to enable effective multimodal reasoning across diverse and distinct domains. MEXA dynamically selects expert models based on the input modality and the task-specific reasoning demands. Each expert model generates interpretable textual reasoning outputs, which are then aggregated by a Large Reasoning Model (LRM) to produce the final answer. This modular design allows flexible and transparent multimodal reasoning across diverse domains without additional training overhead.

Introduction

Current multimodal AI systems typically rely on monolithic architectures trained on massive datasets. While effective for common modalities like images and text, these systems face significant hurdles in scalability, transparency, and reasoning depth. Integrating new modalities often necessitates architectural redesigns, and "early fusion" approaches create black-box models where the reasoning process is opaque. MEXA addresses these challenges by introducing a modular, training-free "Swiss Army Knife" approach to multimodal reasoning.

Methodology

The MEXA framework operates in three distinct, modular stages:

  1. Dynamic Selection (The Router): Upon receiving a query, MEXA utilizes a Multimodal Large Language Model (e.g., GPT-4o) as a router. It analyzes the question and task context to select the most relevant specialized experts from a diverse pool.
  2. Expert Execution (The Perception): Selected experts process raw data (video, audio, 3D, etc.) and generate interpretable textual descriptions. This converts diverse sensory inputs into a universal linguistic interface.
  3. Expert Aggregation (The Reasoning): A Large Reasoning Model (LRM), such as DeepSeek, aggregates these textual descriptions. Using Chain-of-Thought reasoning, the LRM synthesizes the expert outputs to derive the final answer.
Overview of the MEXA framework architecture, illustrating the flow from input to final answer through dynamic selection, expert execution, and aggregation.

This modularity allows for the seamless integration of new or improved expert models and reasoning engines without requiring retraining of the entire system.

Experimental Evaluation

MEXA was evaluated across four diverse benchmarks to test its general reasoning capabilities: Video-MMMU (multidisciplinary video reasoning), MMAU (multi-task audio understanding), SQA3D (situated 3D scene understanding), and M3D-VQA (3D medical CT scan reasoning).

Key Results

MEXA consistently outperformed strong proprietary and open-source multimodal baselines. On the Video-MMMU benchmark, MEXA achieved an overall accuracy of 71.5%, representing a 5.7% improvement over the best baseline and nearing human-expert performance (74.4%).

Performance comparison of MEXA against state-of-the-art open-source and proprietary models on the Video-MMMU benchmark.

In other domains, MEXA showed significant gains:

  • Audio (MMAU): A +12.2% improvement over existing audio-language models.
  • 3D Understanding (SQA3D): Outperformed specialized 3D models by +1.7%.
  • Medical (M3D): Improved average accuracy by +1.6% over domain-specific models.

Discussion and Insights

A core innovation of MEXA is the use of text as a universal interface. By forcing experts to describe their perceptions in natural language, the system leverages the advanced reasoning capabilities of LLMs while maintaining transparency. Furthermore, ablation studies revealed that different models excel in different roles: GPT-4o proved to be an effective router, while DeepSeek excelled as an aggregator.

Limitations

As a training-free framework, MEXA's performance is inherently tied to the quality of its pre-trained experts. Hallucinations in expert descriptions can lead to incorrect reasoning by the aggregator. Additionally, the multi-stage pipeline introduces higher inference latency compared to single-pass monolithic models.

Conclusion

MEXA demonstrates that a dynamic aggregation of specialized experts can outperform massive, general-purpose models in complex multimodal reasoning tasks. Its modular, training-free design provides a scalable and transparent path forward for the next generation of AI systems.

Summary updated

Abstract

Combining pre-trained expert models offers substantial potential for scalable multimodal reasoning, but building a unified framework remains challenging due to the increasing diversity of input modalities and task complexity. For instance, medical diagnosis requires precise reasoning over structured clinical tables, while financial forecasting depends on interpreting plot-based data to make informed predictions. To tackle this challenge, we introduce MEXA, a training-free framework that performs modality- and task-aware aggregation of multiple expert models to enable effective multimodal reasoning across diverse and distinct domains. MEXA dynamically selects expert models based on the input modality and the task-specific reasoning demands (i.e., skills). Each expert model, specialized in a modality task pair, generates interpretable textual reasoning outputs. MEXA then aggregates and reasons over these outputs using a Large Reasoning Model (LRM) to produce the final answer. This modular design allows flexible and transparent multimodal reasoning across diverse domains without additional training overhead. We extensively evaluate our approach on diverse multimodal benchmarks, including Video Reasoning, Audio Reasoning, 3D Understanding, and Medical QA. MEXA consistently delivers performance improvements over strong multimodal baselines, highlighting the effectiveness and broad applicability of our expert-driven selection and aggregation in diverse multimodal reasoning tasks.
BibTeX
@inproceedings{yu2025mexa,
  author = {Shoubin Yu and Yue Zhang and Ziyang Wang and Jaehong Yoon and Mohit Bansal},
  title = {{MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure