Abstract
SynerGen-VL is a unified, encoder-free multimodal model designed to bridge the gap between image understanding and generation. By employing a single next-token prediction objective, it simplifies the traditional multi-component architecture. The model introduces two key innovations: token folding to efficiently process high-resolution images by reducing sequence length, and vision experts to prevent the degradation of pretrained linguistic knowledge. With only 2.4B activated parameters, SynerGen-VL achieves state-of-the-art performance across various benchmarks, demonstrating that compact, unified models can rival or exceed the capabilities of much larger, specialized systems.
The Challenge of Architectural Bloat
Current Multimodal Large Language Models (MLLMs) often suffer from "architectural bloat," relying on external diffusion models and separate vision encoders (like CLIP) to handle both understanding (captioning, VQA) and generation (text-to-image). This complexity leads to two major hurdles:
- The Resolution-Sequence Trade-off: High-resolution images produce prohibitively long token sequences that exceed the LLM's context window.
- Knowledge Interference: Training an LLM to understand images often degrades its original linguistic knowledge, leading to worse performance on pure text tasks.
Methodology: Folding and Expertise
SynerGen-VL simplifies the architecture by treating text and images as discrete tokens within a standard autoregressive Transformer. Its novelty lies in two specific mechanisms:
- Token Folding & Unfolding: To support high-resolution inputs without exploding the sequence length, the model concatenates a grid of image tokens (e.g., a $2 \times 8$ patch) into a single embedding. This reduces the sequence length by 16x, allowing the model to process high-res inputs using only 256 tokens in the LLM core. For generation, a shallow "unfolding" head expands these back into the original VQ tokens.
- Vision Experts (MMoE): Inspired by Mixture-of-Experts, the model adds image-specific Feed-Forward Networks (FFNs) to each layer. Visual tokens pass through the vision expert, while text tokens use the original LLM weights. This prevents multimodal training from "overwriting" the LLM’s language capabilities.
Training and Datasets
The model follows a two-stage progressive alignment strategy:
- Stage 1 & 2: Training on roughly 837 million samples (including LAION-Aesthetics, Coyo-700M, and SAM) to bridge visual and textual concepts.
- Instruction Tuning: Fine-tuning on 5 million bilingual instructions from the InternVL-1.5 dataset to refine perception and dialogue.
Key Results and Insights
Despite having only 2.4B activated parameters, SynerGen-VL punches well above its weight class:
- Image Understanding: It surpasses the 8B-parameter Emu3-Chat and Chameleon on benchmarks like OCRBench and TextVQA.
- Image Generation: It achieves a zero-shot FID of 7.65 on MSCOCO, significantly better than Emu3-Gen (19.3) and comparable to the much larger Janus (8.53).
- Task Disentanglement: Internal analysis shows that representations for "understanding" and "generation" are nearly identical in shallow layers but become highly specialized in deeper layers, suggesting the model builds a shared foundational visual concept before branching into task-specific logic.
Limitations and Future Work
While efficient, SynerGen-VL faces the inherent speed limitations of autoregressive generation compared to modern diffusion models. Its performance is also capped by the quality of the underlying VQ-VAE tokenizer. Future directions include exploring higher resolutions and extending the "expert" framework to video and audio for a truly universal "Any-to-Any" model.
Summary updated