Abstract
UniBoost challenges the dominance of image-text pair pre-training (e.g., CLIP) by demonstrating that unsupervised, unimodal models trained on images or text alone provide a richer foundation for vision-language tasks. By utilizing a multitask intermediate fine-tuning framework to align these powerful unimodal encoders, UniBoost achieves significant performance gains in zero-shot semantic segmentation and visual question answering, proving that the "union" of unimodal data is more potent than the "intersection" of noisy image-text pairs.
The Bottlenecks of Image-Text Pair Pre-training
Recent advancements in computer vision have been driven by models like CLIP, which learn from massive web-scraped image-text pairs. However, this approach faces two primary limitations: the Intersection Problem and the Noise Problem.
The intersection problem arises because these models are restricted to data where both modalities coexist, ignoring the vast "union" of independent image and text data available on the internet. Furthermore, web-scraped pairs are often noisy, with misaligned captions that can hinder model accuracy. UniBoost argues that unimodal models—such as Masked Autoencoders (MAE) for images and T5 for text—are superior feature extractors because they are trained on larger, more diverse, and cleaner datasets without the constraint of matching pairs.
The UniBoost Framework
To bridge the gap between independent image and text encoders, the researchers introduced UniBoost, a multitask intermediate fine-tuning framework. The architecture consists of three core components:
- Unimodal Encoders: The system is initialized with high-capacity, unsupervised weights (e.g., MAE-L for vision and T5-B for language) rather than training from scratch.
- The UniBoost-Neck: A multiway multimodal transformer module designed to handle diverse interactions, including unimodal processing, language-guided embedding, and deep fusion for complex reasoning.
- Multitask Intermediate Fine-tuning: Before downstream application, the model undergoes a "warm-up" phase on 12 diverse in-domain tasks (e.g., classification, captioning, depth estimation) across 6.5 million images. This phase forces the separate encoders to align their feature spaces and learn to "talk" to each other.
Key Results and Insights
UniBoost consistently outperforms state-of-the-art CLIP-based models across several benchmarks:
- Zero-shot Semantic Segmentation: On PASCAL-5i, UniBoost improved mIoU by 6.5% (from 52.3% to 58.8%). On COCO-20i, it saw a 6.2% boost.
- Visual Question Answering (VQA): On the novel split of VQA v2.0, UniBoost achieved a 3.2% higher average accuracy (45.3% vs. 42.1%) compared to CLIP-initialized baselines.
- Object Detection: In transductive settings on PASCAL VOC 2007, UniBoost reached 77.4 mAP, outperforming specialized models like DenseCLIP.
The success of UniBoost highlights that unimodal pre-training offers broader coverage of real-world data distributions. By learning the "purity" of each modality first, the model captures more complex patterns than models forced to match noisy pairs from the start. Additionally, the multiway nature of the UniBoost-Neck prevents the "one-size-fits-all" performance degradation often seen in simpler multimodal architectures.
Conclusion and Future Work
UniBoost demonstrates the power of combining high-capacity unimodal models for multimodal tasks. While currently focused on vision and language, future work could expand this approach to include additional modalities like audio or sensor data. Furthermore, scaling the intermediate fine-tuning phase beyond 6.5 million images could unlock even greater performance gains, potentially setting a new standard for zero-shot vision-language understanding.
Summary updated