Abstract
Multimodal Large Language Models (MLLMs) traditionally rely on linear or query-based projectors to bridge the gap between visual encoders and linguistic decoders. However, these methods often suffer from information loss and a persistent semantic gap. This paper introduces MAGE (Multimodal Alignment and Generation Enhancement), a framework featuring an Intelligent Alignment Network (IAN) and a dual-loss training strategy. By optimizing for both dimensional consistency and semantic integrity, MAGE achieves state-of-the-art performance on multimodal benchmarks while utilizing significantly fewer visual tokens than existing models.
1. Introduction
In the evolution of Multimodal Large Language Models (MLLMs), the "projector" serves as the critical bridge translating visual features from encoders like CLIP into the embedding space of a Large Language Model (LLM). Despite recent progress, two primary challenges persist:
- Information Loss: Simple linear projectors often fail to capture complex semantic relationships, while complex query-based mechanisms like Q-Formers may overlook global context.
- The Semantic Gap: Most models are trained primarily on cross-entropy (generation) loss. This does not explicitly force visual features to align geometrically with text embeddings, resulting in "vector gaps" where the model recognizes an image but struggles to map it precisely to linguistic concepts.
MAGE addresses these spatial and semantic losses to enhance both multimodal understanding and "Any-to-Any" generation capabilities.
2. Methodology
The MAGE architecture is built upon three core innovations designed to tighten the coupling between visual and textual modalities.
2.1 Intelligent Alignment Network (IAN)
The IAN replaces standard projectors with a two-stage refinement process:
- Vector Alignment Block (VAB): Utilizes MLPs and nonlinear transformations to map the high-dimensional output of the visual encoder to the LLM's embedding space.
- Semantic Enhancement Block (SEB): Employs a learnable convolutional layer to capture local spatial details and a cross-modal self-attention mechanism to inject high-level semantic context into the visual features.
2.2 Dual-Loss Training Strategy
To bridge the semantic gap, MAGE employs a dual-loss approach during training:
- Image-Text Generation (ITG) Loss: A standard cross-entropy loss used for natural language generation.
- Image-Text Distance Minimization (ITDM) Loss: A Mean Squared Error (MSE) loss that explicitly minimizes the distance between the processed visual vector and the corresponding text embedding. This acts as a "semantic anchor," forcing visual representations to reside in the same geometric space as their textual counterparts.
2.3 Hierarchical Agent Framework
For "Any-to-Any" generation, MAGE functions as an orchestrator. It generates structured JSON plans to invoke external tools (e.g., Stable Diffusion for image generation, TTS for audio) to fulfill complex, multi-step user instructions.
3. Datasets and Training
MAGE was trained using a multi-stage pipeline:
- Pre-training: Initial semantic alignment using the CC-558K dataset.
- Instruction Tuning: A mixture of 980K samples, including the ALLaVA-Instruct-VFLAN-4V dataset.
- HMDSet: The authors introduced the Human-Machine Dialogue Dataset, containing 31K samples of complex multimodal tasks. This dataset is specifically designed to train the model's tool-calling and task-planning abilities across text, image, audio, and video modalities.
4. Experimental Results
MAGE demonstrates significant improvements across several key benchmarks, emphasizing efficiency and robustness.
4.1 Performance and Token Efficiency
One of the most striking findings is MAGE's token efficiency. MAGE-7B achieves a score of 71.2 on MMBench, outperforming LLaVA-1.5 (64.3) and LLaVA-NeXT (67.4). Notably, MAGE achieves these results using only 144 visual tokens, compared to the 576 tokens used by many competitors. This suggests that the quality of alignment is more critical than the sheer volume of visual data provided to the LLM.
4.2 Robustness and Hallucination
On the SEED-Bench, MAGE reached a score of 65.3, surpassing specialized architectures like C-Abstractor. Furthermore, on the POPE benchmark for hallucination detection, MAGE-13B achieved 89.9, indicating a high degree of reliability in identifying objects actually present in a scene.
4.3 Ablation Studies
Ablation experiments confirmed the necessity of the proposed modules. Removing either the IAN or the dual-loss strategy resulted in a noticeable performance degradation (e.g., a ~4 point drop in SEED scores), validating the synergy between the architecture and the optimization objective.
5. Conclusion
MAGE represents a significant step forward in bridging the visual-semantic gap in MLLMs. By combining the Intelligent Alignment Network with a distance-minimizing loss function, the model achieves superior understanding and reduced hallucination with high token efficiency. While currently dependent on an agent framework for non-text generation, future iterations aim to scale the architecture and expand the HMDSet to support even more complex reasoning chains and native multimodal generation.
Summary updated