UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks

Yanan Sun, Zihan Zhong, Qi Fan, Chi-Keung Tang, Yu-Wing TaiarXiv 2023 · DOI · arXiv

Abstract

UniBoost challenges the dominance of image-text pair pre-training (e.g., CLIP) by demonstrating that unsupervised, unimodal models trained on images or text alone provide a richer foundation for vision-language tasks. By utilizing a multitask intermediate fine-tuning framework to align these powerful unimodal encoders, UniBoost achieves significant performance gains in zero-shot semantic segmentation and visual question answering, proving that the "union" of unimodal data is more potent than the "intersection" of noisy image-text pairs.

The Bottlenecks of Image-Text Pair Pre-training

Recent advancements in computer vision have been driven by models like CLIP, which learn from massive web-scraped image-text pairs. However, this approach faces two primary limitations: the Intersection Problem and the Noise Problem.

The intersection problem arises because these models are restricted to data where both modalities coexist, ignoring the vast "union" of independent image and text data available on the internet. Furthermore, web-scraped pairs are often noisy, with misaligned captions that can hinder model accuracy. UniBoost argues that unimodal models—such as Masked Autoencoders (MAE) for images and T5 for text—are superior feature extractors because they are trained on larger, more diverse, and cleaner datasets without the constraint of matching pairs.

Comparison of pre-training paradigms: Image-Text Pair (Intersection) vs. Unsupervised Unimodal (Union).

The UniBoost Framework

To bridge the gap between independent image and text encoders, the researchers introduced UniBoost, a multitask intermediate fine-tuning framework. The architecture consists of three core components:

  1. Unimodal Encoders: The system is initialized with high-capacity, unsupervised weights (e.g., MAE-L for vision and T5-B for language) rather than training from scratch.
  2. The UniBoost-Neck: A multiway multimodal transformer module designed to handle diverse interactions, including unimodal processing, language-guided embedding, and deep fusion for complex reasoning.
  3. Multitask Intermediate Fine-tuning: Before downstream application, the model undergoes a "warm-up" phase on 12 diverse in-domain tasks (e.g., classification, captioning, depth estimation) across 6.5 million images. This phase forces the separate encoders to align their feature spaces and learn to "talk" to each other.
Overview of the UniBoost architecture, featuring unimodal encoders and the multiway UniBoost-Neck.

Key Results and Insights

UniBoost consistently outperforms state-of-the-art CLIP-based models across several benchmarks:

  • Zero-shot Semantic Segmentation: On PASCAL-5i, UniBoost improved mIoU by 6.5% (from 52.3% to 58.8%). On COCO-20i, it saw a 6.2% boost.
  • Visual Question Answering (VQA): On the novel split of VQA v2.0, UniBoost achieved a 3.2% higher average accuracy (45.3% vs. 42.1%) compared to CLIP-initialized baselines.
  • Object Detection: In transductive settings on PASCAL VOC 2007, UniBoost reached 77.4 mAP, outperforming specialized models like DenseCLIP.

The success of UniBoost highlights that unimodal pre-training offers broader coverage of real-world data distributions. By learning the "purity" of each modality first, the model captures more complex patterns than models forced to match noisy pairs from the start. Additionally, the multiway nature of the UniBoost-Neck prevents the "one-size-fits-all" performance degradation often seen in simpler multimodal architectures.

Conclusion and Future Work

UniBoost demonstrates the power of combining high-capacity unimodal models for multimodal tasks. While currently focused on vision and language, future work could expand this approach to include additional modalities like audio or sensor data. Furthermore, scaling the intermediate fine-tuning phase beyond 6.5 million images could unlock even greater performance gains, potentially setting a new standard for zero-shot vision-language understanding.

Summary updated

Abstract

Large-scale joint training of multimodal models, e.g., CLIP, have demonstrated great performance in many vision-language tasks. However, image-text pairs for pre-training are restricted to the intersection of images and texts, limiting their ability to cover a large distribution of real-world data, where noise can also be introduced as misaligned pairs during pre-processing. Conversely, unimodal models trained on text or image data alone through unsupervised techniques can achieve broader coverage of diverse real-world data and are not constrained by the requirement of simultaneous presence of image and text. In this paper, we demonstrate that using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our thorough studies validate that models pre-trained as such can learn rich representations of both modalities, improving their ability to understand how images and text relate to each other. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models by 6.5% (52.3% $\rightarrow$ 58.8%) on PASCAL-5$^i$ and 6.2% (27.2% $\rightarrow$ 33.4%) on COCO-20$^i$ semantic segmentation under zero-shot setting respectively. By learning representations of both modalities, unimodal pre-training offers broader coverage, reduced misalignment errors, and the ability to capture more complex features and patterns in the real-world data resulting in better performance especially for zero-shot vision-language tasks.
BibTeX
@misc{sun2023uniboost,
  author = {Yanan Sun and Zihan Zhong and Qi Fan and Chi-Keung Tang and Yu-Wing Tai},
  title = {{UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks}},
  year = {2023},
  eprint = {2306.04715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.04715},
}
Figure