Introduction
Current Multimodal Large Language Models (MLLMs) often resemble "Frankenstein" architectures, stitching together disparate components like CLIP for vision understanding and Diffusion models for image generation. This modularity introduces complexity and prevents true cross-modal synergy. Liquid introduces a unified auto-regressive paradigm that enables standard LLMs to both "see" and "draw" by treating visual data as discrete tokens within an expanded vocabulary.
A Unified Multi-modal Paradigm
Unlike previous approaches that rely on externalized modules or separate heads for different tasks, Liquid achieves full unification. By continuing the training of pre-trained LLMs (such as Gemma-2 or Qwen2.5) with an extended vocabulary of 8,192 visual tokens, the researchers transformed these models into native multimodal generators.
The technical implementation relies on a streamlined pipeline:
- Image Tokenization: Images are converted into 1,024 discrete codes using a VQGAN.
- Unified Token Space: Both text (BPE-tokenized) and image tokens reside in the same space.
- Next-Token Prediction: The model is trained with a single objective—predicting the next token, whether it is linguistic or visual.
Scaling Laws and Cross-Modal Synergy
A key discovery of the Liquid project is the "Vanishing Multimodal Tax." In smaller models, adding visual tasks often degrades language performance. However, as model capacity scales (e.g., toward 32B parameters), this interference disappears. Furthermore, the researchers found that visual understanding and generation tasks are mutually beneficial; improving one task leads to gains in the other, suggesting a shared semantic foundation.
Key Results
Liquid demonstrates state-of-the-art performance across both generation and understanding benchmarks:
- Image Generation: Liquid-7B achieved a Fréchet Inception Distance (FID) of 5.47 on the MJHQ-30K benchmark, surpassing specialized models like Stable Diffusion XL.
- Multimodal Understanding: The model outperformed Chameleon on understanding tasks and matched the performance of established models like LLaVA.
- Efficiency: By leveraging "continue training" on existing LLMs rather than training from scratch, Liquid is approximately 100x more cost-effective than previous unified attempts.
Conclusion
Liquid proves that pre-trained LLMs are inherently capable of becoming high-quality multimodal generators. By unifying the architecture and training objective, Liquid paves the way for more efficient, scalable, and synergistic multimodal AI systems. Future work will focus on refining tokenization to better align visual and linguistic semantics and scaling to higher resolutions.
Summary updated