Abstract
SynerGen-VL is a unified multimodal model designed to bridge the gap between image understanding and generation within a single next-token prediction framework. By moving away from complex, multi-component "Frankenstein" architectures and adopting a novel "token folding" strategy alongside specialized "vision experts," SynerGen-VL achieves state-of-the-art performance with only 2.4B activated parameters. This approach allows the model to handle high-resolution images efficiently while preserving the linguistic reasoning capabilities of the base language model, rivaling the performance of models three times its size.
Introduction: The Multimodal Complexity Problem
Current Multimodal Large Language Models (MLLMs) often resemble "Frankenstein" systems. To perceive the world, they typically rely on external encoders like CLIP; to create imagery, they plug into separate diffusion models. This hybrid nature introduces significant challenges:
- Architectural Complexity: Managing disparate training objectives (e.g., diffusion for generation vs. autoregression for text) complicates scaling and optimization.
- The Resolution Trap: Discrete image tokenizers often require low downsampling ratios to maintain visual detail. For high-resolution images, this results in extremely long token sequences that can overwhelm a model's memory and compute budget.
- Knowledge Interference: Fine-tuning a general-purpose LLM on visual data from scratch can lead to "catastrophic forgetting," where the model's original linguistic and reasoning strengths are degraded.
Methodology: Synergistic Design
SynerGen-VL addresses these issues through an encoder-free, unified architecture that treats images and text as sequences of discrete tokens.
Token Folding and Unfolding
To process high-resolution images without exploding sequence length, the authors introduce Token Folding. This mechanism compresses groups of visual tokens (e.g., a $2 \times 8$ patch) into a single "hyper-token" before they enter the LLM. For a $512 \times 512$ image, this reduces the sequence from 4096 tokens to just 256—a 16x reduction. During generation, a shallow Token Unfold head expands these hyper-tokens back into the full sequence required for pixel rendering.
Vision Experts (MMoE)
Rather than updating all parameters for visual tasks, SynerGen-VL utilizes Vision Experts—image-specific Feed-Forward Networks (FFNs). By keeping the core language weights largely frozen and routing visual information through these experts, the model maintains its high "intelligence" floor, preserving complex reasoning capabilities required for benchmarks like MathVista.
Training and Datasets
The model's development followed a two-stage progressive alignment strategy:
- Pretraining: The model was exposed to over 800 million samples from diverse datasets, including LAION-En/Aesthetics, COYO, and SAM, to learn fundamental visual concepts.
- Instruction Tuning: Refinement was performed using the InternVL-1.5 dataset (approx. 5M samples) to hone multimodal dialogue and high-quality image generation.
Key Results and Performance
Despite having only 2.4B activated parameters, SynerGen-VL demonstrates remarkable efficiency and capability:
- Image Understanding: On OCRBench, it scored 721, outperforming many larger encoder-free models in text-in-image recognition.
- Image Generation: The model achieved a 7.65 FID score on MSCOCO-30K, indicating high-fidelity synthesis.
- Reasoning: It maintained strong performance on MathVista and MMMU, proving that the "Vision Expert" approach successfully prevents linguistic degradation.
- GenEval: With an overall score of 0.61, it outperformed Emu3-Gen (0.54) and matched the performance of more complex architectures like Janus.
Conclusion and Future Directions
SynerGen-VL proves that a streamlined, unified approach to multimodality can be both more efficient and more capable than hybrid systems. By leveraging token folding and vision experts, it sets a new standard for parameter-efficient multimodal modeling. Future work will focus on developing custom, higher-efficiency tokenizers and scaling the framework to larger base models (7B to 70B parameters) to further push the boundaries of unified image-text intelligence.
Summary updated