SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiao-Gang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai

Abstract

SynerGen-VL is a unified multimodal model designed to bridge the gap between image understanding and generation within a single next-token prediction framework. By moving away from complex, multi-component "Frankenstein" architectures and adopting a novel "token folding" strategy alongside specialized "vision experts," SynerGen-VL achieves state-of-the-art performance with only 2.4B activated parameters. This approach allows the model to handle high-resolution images efficiently while preserving the linguistic reasoning capabilities of the base language model, rivaling the performance of models three times its size.

Introduction: The Multimodal Complexity Problem

Current Multimodal Large Language Models (MLLMs) often resemble "Frankenstein" systems. To perceive the world, they typically rely on external encoders like CLIP; to create imagery, they plug into separate diffusion models. This hybrid nature introduces significant challenges:

  1. Architectural Complexity: Managing disparate training objectives (e.g., diffusion for generation vs. autoregression for text) complicates scaling and optimization.
  2. The Resolution Trap: Discrete image tokenizers often require low downsampling ratios to maintain visual detail. For high-resolution images, this results in extremely long token sequences that can overwhelm a model's memory and compute budget.
  3. Knowledge Interference: Fine-tuning a general-purpose LLM on visual data from scratch can lead to "catastrophic forgetting," where the model's original linguistic and reasoning strengths are degraded.
Comparison of SynerGen-VL with existing multimodal architectures. SynerGen-VL (e) achieves a unified training objective without externalized models or additional semantic pretraining.

Methodology: Synergistic Design

SynerGen-VL addresses these issues through an encoder-free, unified architecture that treats images and text as sequences of discrete tokens.

Token Folding and Unfolding

To process high-resolution images without exploding sequence length, the authors introduce Token Folding. This mechanism compresses groups of visual tokens (e.g., a $2 \times 8$ patch) into a single "hyper-token" before they enter the LLM. For a $512 \times 512$ image, this reduces the sequence from 4096 tokens to just 256—a 16x reduction. During generation, a shallow Token Unfold head expands these hyper-tokens back into the full sequence required for pixel rendering.

Comparison of standard tokenization (left) vs. SynerGen-VL's Token Folding and Vision Expert approach (right).

Vision Experts (MMoE)

Rather than updating all parameters for visual tasks, SynerGen-VL utilizes Vision Experts—image-specific Feed-Forward Networks (FFNs). By keeping the core language weights largely frozen and routing visual information through these experts, the model maintains its high "intelligence" floor, preserving complex reasoning capabilities required for benchmarks like MathVista.

Training and Datasets

The model's development followed a two-stage progressive alignment strategy:

  1. Pretraining: The model was exposed to over 800 million samples from diverse datasets, including LAION-En/Aesthetics, COYO, and SAM, to learn fundamental visual concepts.
  2. Instruction Tuning: Refinement was performed using the InternVL-1.5 dataset (approx. 5M samples) to hone multimodal dialogue and high-quality image generation.

Key Results and Performance

Despite having only 2.4B activated parameters, SynerGen-VL demonstrates remarkable efficiency and capability:

  • Image Understanding: On OCRBench, it scored 721, outperforming many larger encoder-free models in text-in-image recognition.
  • Image Generation: The model achieved a 7.65 FID score on MSCOCO-30K, indicating high-fidelity synthesis.
  • Reasoning: It maintained strong performance on MathVista and MMMU, proving that the "Vision Expert" approach successfully prevents linguistic degradation.
  • GenEval: With an overall score of 0.61, it outperformed Emu3-Gen (0.54) and matched the performance of more complex architectures like Janus.

Conclusion and Future Directions

SynerGen-VL proves that a streamlined, unified approach to multimodality can be both more efficient and more capable than hybrid systems. By leveraging token folding and vision experts, it sets a new standard for parameter-efficient multimodal modeling. Future work will focus on developing custom, higher-efficiency tokenizers and scaling the framework to larger base models (7B to 70B parameters) to further push the boundaries of unified image-text intelligence.

Summary updated

Abstract

The remarkable success of Large Language Models (LLMs) has extended to the multimodal domain, achieving outstanding performance in image understanding and generation. Recent efforts to develop unified Multimodal Large Language Models (MLLMs) that integrate these capabilities have shown promising results. However, existing approaches often involve complex designs in model architecture or training pipeline, increasing the difficulty of model training and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful encoder-free MLLM capable of both image understanding and generation. To address challenges identified in existing encoder-free unified MLLMs, we introduce the token folding mechanism and the vision-expert-based progressive alignment pretraining strategy, which effectively support high-resolution image understanding while reducing training complexity. After being trained on large-scale mixed image-text data with a unified next-token prediction objective, SynerGen-VL achieves or surpasses the performance of existing encoder-free unified MLLMs with comparable or smaller parameter sizes, and narrows the gap with task-specific state-of-the-art models, highlighting a promising path toward future unified MLLMs. Our code and models shall be released.
BibTeX
@inproceedings{li2025synergen,
  author = {Hao Li and Changyao Tian and Jie Shao and Xizhou Zhu and Zhaokai Wang and Jinguo Zhu and Wenhan Dou and Xiao-Gang Wang and Hongsheng Li and Lewei Lu and Jifeng Dai},
  title = {{SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02771},
}
Figure