Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Xi Victoria Lin

Introduction

Scaling multi-modal foundation models that process text, images, and speech in a single sequence—often referred to as "early-fusion" models—presents significant computational challenges. While Large Language Models (LLMs) have demonstrated remarkable scalability, the integration of high-resolution visual and auditory data exponentially increases training costs. Mixture-of-Transformers (MoT) addresses this by introducing a sparse architecture that decouples modality-specific parameters, significantly reducing the computational budget required to achieve state-of-the-art performance.

The MoT Architecture

The core innovation of MoT is the strategic decoupling of model parameters based on data modality. In a standard dense Transformer, every parameter is tasked with processing all input types, leading to optimization conflicts. MoT instead assigns separate Feed-Forward Networks (FFNs), attention projection matrices ($W_Q, W_K, W_V, W_O$), and Layer Normalization parameters to each modality.

The MoT architecture decouples parameters into modality-specific towers while maintaining a global attention mechanism.

Unlike traditional Mixture-of-Experts (MoE) models that rely on learned, often unstable routing mechanisms, MoT utilizes deterministic, rule-based routing. A token's modality (text, image, or speech) automatically dictates which set of parameters it engages. Crucially, the self-attention mechanism remains global; while projections are specialized, every token can attend to every other token across the sequence, preserving the model's cross-modal reasoning capabilities.

Latent Space Motivation

The design of MoT is motivated by an empirical observation of how dense models process multi-modal data. Principal Component Analysis (PCA) of the latent feature space reveals that tokens from different modalities naturally cluster into distinct regions, even when forced through shared weights.

PCA visualizations of latent feature spaces across layers 1, 5, 17, and 32 demonstrate the inherent separation of modalities.

As shown in the visualizations, this separation becomes more pronounced in deeper layers. MoT "leans into" this natural behavior by providing specialized hardware for these distinct signals, avoiding the optimization "tug-of-war" that occurs when disparate modalities compete for the same parameter capacity.

Experimental Results and Efficiency

The researchers evaluated MoT across large-scale datasets, including the Llama 2 corpus, 380 million licensed images, and 60,000 hours of speech data. The results demonstrate substantial "iso-performance" speedups:

  • Compute Savings: For the Chameleon 7B (Text + Image) setting, MoT reached baseline performance using only 55.8% of the FLOPs. With the addition of speech, this dropped further to 37.2%.
  • Wall-Clock Acceleration: On NVIDIA A100 GPUs, MoT achieved dense-baseline image quality in 47.2% of the time and text quality in 75.6% of the time.
  • Superior Scaling: In a Transfusion setting, a 760M MoT model outperformed a 1.4B dense baseline on key image metrics (CLIP and FID scores) while utilizing half the compute.

Conclusion

Mixture-of-Transformers provides a scalable path forward for multi-modal foundation models. By decoupling parameters and utilizing deterministic routing, MoT resolves the optimization friction inherent in dense early-fusion models. This approach not only slashes training costs but also allows for the integration of diverse modalities like speech and high-resolution images without compromising the efficiency of the underlying text model.

Summary updated

BibTeX
@article{liang2024mixture,
  author = {Weixin Liang and Lili Yu and Liang Luo and Srinivasan Iyer and Ning Dong and Chunting Zhou and Gargi Ghosh and Mike Lewis and Wen-tau Yih and Luke Zettlemoyer and Xi Victoria Lin},
  title = {{Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=Nu6N69i8SB},
}
Figure