OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces

Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, Zhou Zhao

Introduction

The emergence of "omni" multimodal models, such as GPT-4o and Gemini, has revolutionized how we interact with AI across different data types. However, creating open-source models that truly understand a diverse range of modalities—including 3D, audio, images, and text—remains a formidable task. The primary bottlenecks are twofold: the extreme scarcity of multi-way paired data (such as 3D-audio or 3D-image pairs) and the massive computational resources required to train large-scale models from scratch.

Methodology: The OmniBind Framework

OmniBind introduces a novel approach to multimodal scaling by "binding" together existing specialist models into a single, shared representation space. Rather than training a massive backbone from the ground up, OmniBind integrates 14 pre-trained specialist models, remapping their latent spaces into a unified "binding space" anchored by the powerful EVA-CLIP-18B foundation model.

OmniBind technical architecture featuring dynamic routing and decoupling loss.

The framework is built upon three key technical pillars:

  1. Space Binding via Pseudo-retrieval: To overcome the lack of paired data, the authors use existing models to retrieve "pseudo-pairs" from large unimodal datasets. Lightweight MLP projectors are then trained to align these specialist embeddings with the shared EVA-CLIP space.
  2. Dynamic Weight Routing: Drawing inspiration from Mixture-of-Experts (MoE) architectures, OmniBind employs learnable routers for each modality. Instead of simple averaging, the router analyzes the input and dynamically assigns weights to various specialist models, allowing the system to prioritize the most relevant "expert" for any given task.
  3. Language Representation Decoupling ($L_{dec}$): Recognizing that text descriptions vary significantly across modalities (e.g., structural focus in 3D vs. action focus in audio), the authors implemented a decoupling loss. This ensures the language router can distinguish between specialized text styles, preventing representation collapse and maintaining high-fidelity alignment.

Experimental Setup and Datasets

The model was trained using a vast collection of unpaired data, including 800k point clouds from Objaverse, audio samples from AudioSet, and images from ImageNet. Text data was aggregated from specialized datasets like AudioCaps, Clotho, and COCO. Evaluation was conducted across 13 diverse benchmarks, including ESC-50 (audio classification), ModelNet40 (3D classification), and Flickr30K (image-text retrieval).

Key Results and Emergent Capabilities

OmniBind-Full (30B) achieved state-of-the-art performance across nearly all 13 benchmarks, outperforming existing open-source omni-models such as ImageBind and LanguageBind.

Omni-alignment radar chart comparing OmniBind against existing benchmarks across 13 tasks.

Beyond raw metrics, OmniBind demonstrates several remarkable properties:

  • Efficiency: The 30B parameter model was trained in just three days on a single node of 8x NVIDIA 4090 GPUs, representing an order of magnitude improvement in training efficiency compared to traditional methods.
  • Emergent Cross-Modal Alignment: Despite never being trained on 3D-audio pairs, the model successfully performs zero-shot 3D-audio retrieval, such as matching the sound of a ticking clock to its corresponding 3D model.
  • Composable Understanding: The shared space supports "multimodal arithmetic." For example, subtracting a 3D cat embedding from an image of a cat and dog on a sofa results in a vector that accurately represents "dogs on a sofa."

Conclusion

OmniBind proves that scaling multimodal models is achievable without supercomputing clusters by strategically "binding" the knowledge of existing specialists. While currently focused on four modalities, the framework's modular design paves the way for future integration of video, tactile, or other sensory data. As specialist models continue to evolve, OmniBind provides a scalable blueprint for the next generation of omni-modal AI.

Summary updated

Abstract

Recently, human-computer interaction with various modalities has shown promising applications, like GPT-4o and Gemini. Given the foundational role of multimodal joint representation in understanding and generation pipelines, high-quality omni joint representations would be a step toward co-processing more diverse multimodal information. In this work, we present OmniBind, large-scale multimodal joint representation models ranging in scale from 7 billion to 30 billion parameters, which support 3D, audio, image, and language inputs. Due to the scarcity of data pairs across all modalities, instead of training large models from scratch, we propose remapping and binding the spaces of various pre-trained specialist models together. This approach enables "scaling up" by indirectly increasing the model parameters and the amount of seen data. To effectively integrate various spaces, we dynamically assign weights to different spaces by learning routers with two objectives: cross-modal overall alignment and language representation decoupling. Notably, since binding and routing spaces both only require lightweight networks, OmniBind is extremely training-efficient. Learning the largest 30B model requires merely unpaired unimodal data and approximately 3 days on a single 8-4090 node. Extensive experiments demonstrate the versatility and superiority of OmniBind as an omni representation model, highlighting its great potential for diverse applications, such as any-query and composable multimodal understanding.
BibTeX
@inproceedings{wang2025omnibind,
  author = {Zehan Wang and Ziang Zhang and Hang Zhang and Luping Liu and Rongjie Huang and Xize Cheng and Hengshuang Zhao and Zhou Zhao},
  title = {{OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=l2izo0z7gu},
}
Figure