OmniVec2 - A Novel Transformer Based Network for Large Scale Multimodal and Multitask Learning

Siddharth Srivastava, Gaurav SharmaCVPR 2024 · OpenReview

Abstract

We present a novel multimodal multitask network and associated training algorithm. The method is capable of ingesting data from approximately 12 different modalities namely image, video, audio, text, depth, point cloud, time series, tabular, graph, X-ray, infrared, IMU, and hyper-spectral. The proposed approach utilizes modality specialized tokenizers, a shared transformer architecture, and cross-attention mechanisms to project the data from different modalities into a unified embedding space. It addresses multimodal and multitask scenarios by incorporating modality-specific task heads for different tasks in respective modalities. We propose a novel pretraining strategy with iterative modality switching to initialize the network, and a training algorithm which trades off fully joint training over all modalities, with training on pairs of modalities at a time. We provide comprehensive evaluation across 25 datasets from 12 modalities and show state of the art performances, demonstrating the effectiveness of the proposed architecture, pretraining strategy and adapted multitask training.

Introduction

Most machine learning models are specialists: a model trained for computer vision rarely understands audio, and a text-based LLM doesn't inherently "know" how to process a 3D point cloud. While recent research has moved toward "foundation models," most still rely on modality-specific encoders or are limited to a few related data types (e.g., image-text-video).

The challenge lies in creating a truly "omni-modal" network that can ingest heterogeneous data in a holistic way. Specifically, how do you train a single shared backbone to learn meaningful representations from data as different as a tabular spreadsheet, a 1D time series, and a 3D graph, without the model becoming a "jack of all trades, master of none"? OmniVec2 addresses this by employing a unified transformer-based architecture designed to handle 12 diverse data modalities.

Methods

Architecture

The OmniVec2 architecture consists of four primary components designed to bridge the gap between heterogeneous data types:

  1. Modality-Specific Tokenizers: Instead of a one-size-fits-all approach, the model uses specialized tokenizers for each of the 12 modalities to capture their unique structures.
  2. Shared Transformer Backbone: A BERT-based transformer acts as a central "bottleneck" where knowledge from all modalities is processed and regularized.
  3. Cross-Attention Fusion: When training on multiple modalities, cross-attention modules are used to bridge the gap between different data streams, allowing the features of one modality to inform the other.
  4. Task-Specific Heads: While the backbone is shared, the final predictions are made by lightweight, task-specific heads tailored to the required output format (e.g., classification vs. segmentation).
The OmniVec2 architecture featuring modality-specific tokenizers, a shared transformer backbone, and cross-attention fusion modules.

Training Strategy: Iterative Modality Switching

The core innovation in training is the Iterative Modality Switching approach, which proceeds in three stages:

  • Stage 1 (Unimodal Masked Pretraining): Standard unsupervised learning where the model predicts masked parts of the input, one modality at a time.
  • Stage 2 (Multimodal Masked Pretraining): The model is trained on pairs of modalities simultaneously. It must use unmasked tokens from Modality A to help predict masked tokens in Modality B, forcing the network to find commonalities in the latent space.
  • Stage 3 (Supervised Multitask Training): The model is fine-tuned on specific tasks using pairs of datasets, employing "task balancing" to ensure that simple tasks don't overshadow complex ones during gradient updates.

Evaluation

Datasets

The model was evaluated on a massive suite of 25 benchmark datasets covering 12 modalities, including visual (ImageNet1K, MSCOCO), temporal (Kinetics-400), audio (AudioSet), text (GLUE), 3D spatial (S3DIS, ModelNet40), and specialized domains like graphs (PCQM4M-LSC), time-series, tabular data, infrared, hyperspectral, and X-ray.

Key Results

OmniVec2 achieved state-of-the-art (SOTA) or near-SOTA results across almost every category:

  • Image Recognition: 94.6% top-1 accuracy on iNaturalist-2018, outperforming the original OmniVec (93.8%) and MAE (86.8%).
  • Audio: 99.1% accuracy on ESC50, beating the specialized Audio Spectrogram Transformer (85.7%).
  • 3D Segmentation: 77.1 mIoU on S3DIS, surpassing Swin3D and PointTransformer.
  • Cross-Domain Generalization: On the MSR-VTT video-text retrieval task, the model achieved a Recall@10 of 85.8%, significantly higher than previous multimodal efforts.
  • Unseen Modalities: The model showed a remarkable ability to adapt to modalities it wasn't primarily pretrained on, such as achieving 98.1% accuracy on X-ray classification.
Performance comparison on the S3DIS 3D segmentation dataset, where OmniVec2 achieves state-of-the-art results.

Discussion and Conclusion

The success of OmniVec2 highlights several key insights. First, training on pairs of modalities acts as a bridge, allowing knowledge to propagate across the entire network more effectively than trying to force all 12 modalities at once. Second, modality-specific tokenizers significantly outperform unified tokenizers, suggesting that while the "thinking" (backbone) can be shared, the "sensing" (tokenization) needs to be tailored to the data's physics. Finally, the shared backbone acts as a regularizer; training on diverse tasks simultaneously prevents overfitting to any single modality.

While the model still requires modality-specific task heads, OmniVec2 demonstrates that the path to "Omni-AI" lies in smarter, pairwise training of shared architectures across the entire spectrum of human data. Future work will explore scaling to larger transformer architectures and developing truly universal output heads.

Summary updated

Abstract

We present a novel multimodal multitask network and associated training algorithm. The method is capable of ingesting data from approximately 12 different modalities namely image, video, audio, text, depth, point cloud, time series, tabular, graph, X-ray, infrared, IMU, and hyper-spectral. The proposed approach utilizes modality specialized tokenizers, a shared transformer architecture, and cross-attention mechanisms to project the data from different modalities into a unified embedding space. It addresses multimodal and multitask scenarios by incorpo-rating modality-specific task heads for different tasks in respective modalities. We propose a novel pretraining strategy with iterative modality switching to initialize the network, and a training algorithm which trades off fully joint training over all modalities, with training on pairs of modalities at a time. We provide comprehensive evaluation across 25 datasets from 12 modalities and show state of the art performances, demonstrating the effectiveness of the proposed architecture, pretraining strategy and adapted multitask training.
BibTeX
@inproceedings{srivastava2024omnivec,
  author = {Siddharth Srivastava and Gaurav Sharma},
  title = {{OmniVec2 - A Novel Transformer Based Network for Large Scale Multimodal and Multitask Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=7DUh7bAPSW},
}
Figure