Abstract
Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether this joint learning paradigm can help understand each individual modality. In this work, we conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing a broad range of tasks, aiming to assess the quality of the learned representations in a nuanced manner. Interestingly, our empirical observations suggest that vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
Introduction
The rise of Vision-and-Language Pretraining (VLP) has revolutionized AI, producing models like CLIP that can "understand" images through natural language. However, the impact of multimodal training on underlying visual representations remains poorly understood. Does learning from text descriptions enhance a model's visual capabilities across all tasks, or does it introduce specific biases? This research investigates the tension between high-level semantic understanding (labeling objects) and low-level localization (finding exact boundaries).
Methodology
The authors employ a probing methodology to evaluate frozen pretrained visual backbones. By training only a simple prediction head (typically a single linear layer) on extracted features, they assess the inherent quality of the representation. If a basic layer can solve a task, the necessary information must be explicitly present in the features.
The study compares three representative Vision-and-Language (VL) models—CLIP, OFA, and FLAVA—against two advanced vision-only (V) models—MAE and MOCOv3. These models are evaluated across five tasks using benchmarks such as Visual Genome for object classification, VAW for attribute prediction, MSCOCO for detection and segmentation, and PartImageNet for part-level semantic segmentation.
Key Findings
The Semantic-Localization Trade-off
A fundamental divide exists between VL and vision-only models. VL models capture richer semantic information, excelling at predicting fine-grained labels. Conversely, vision-only models, particularly generative ones like MAE, retain significantly more spatial detail, making them superior for "dense" prediction tasks requiring precise localization.
Language as a "Blurring" Filter
Language supervision appears to encourage models to group pixels into abstract concepts. While beneficial for classification, this process seems to "wash out" the precise localized information required for identifying exact object boundaries. This is evident in attribute prediction: VL models dominate in abstract categories like "action," but the gap narrows for "visually grounded" attributes like "texture."
The Impact of Fine-tuning
Interestingly, fine-tuning a model on a specific task (e.g., ImageNet classification) can actually degrade its performance on the broader probing suite. The model learns to discard information irrelevant to the specific task, losing the general-purpose richness of the original pretrained representation.
Results
The quantitative results highlight the performance split. VL models consistently lead in label-based tasks, while vision-only models excel in localization-heavy tasks.
As shown in the table, OFA (VL) achieves the highest accuracy for object prediction on Visual Genome (57.13%), whereas MAE (V) leads in MSCOCO instance segmentation (22.05 mAP) and Part Segmentation (42.30 mIOU).
Conclusion
This study reveals that while language supervision provides a powerful semantic prior, it often comes at the cost of spatial precision. Future research should focus on developing pretraining objectives that combine the semantic richness of multimodal learning with the localization accuracy of unimodal vision training. Bridging this gap is essential for creating truly universal visual backbones capable of excelling at both "what" and "where" tasks.
Summary updated