Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models

Zhuowan Li, Cihang Xie, Benjamin Van Durme, Alan Yuille

Abstract

Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether this joint learning paradigm can help understand each individual modality. In this work, we conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing a broad range of tasks, aiming to assess the quality of the learned representations in a nuanced manner. Interestingly, our empirical observations suggest that vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.

Introduction

The rise of Vision-and-Language Pretraining (VLP) has revolutionized AI, producing models like CLIP that can "understand" images through natural language. However, the impact of multimodal training on underlying visual representations remains poorly understood. Does learning from text descriptions enhance a model's visual capabilities across all tasks, or does it introduce specific biases? This research investigates the tension between high-level semantic understanding (labeling objects) and low-level localization (finding exact boundaries).

Overview of the probing tasks across the semantic-localization spectrum.

Methodology

The authors employ a probing methodology to evaluate frozen pretrained visual backbones. By training only a simple prediction head (typically a single linear layer) on extracted features, they assess the inherent quality of the representation. If a basic layer can solve a task, the necessary information must be explicitly present in the features.

The study compares three representative Vision-and-Language (VL) models—CLIP, OFA, and FLAVA—against two advanced vision-only (V) models—MAE and MOCOv3. These models are evaluated across five tasks using benchmarks such as Visual Genome for object classification, VAW for attribute prediction, MSCOCO for detection and segmentation, and PartImageNet for part-level semantic segmentation.

Key Findings

The Semantic-Localization Trade-off

A fundamental divide exists between VL and vision-only models. VL models capture richer semantic information, excelling at predicting fine-grained labels. Conversely, vision-only models, particularly generative ones like MAE, retain significantly more spatial detail, making them superior for "dense" prediction tasks requiring precise localization.

Language as a "Blurring" Filter

Language supervision appears to encourage models to group pixels into abstract concepts. While beneficial for classification, this process seems to "wash out" the precise localized information required for identifying exact object boundaries. This is evident in attribute prediction: VL models dominate in abstract categories like "action," but the gap narrows for "visually grounded" attributes like "texture."

The Impact of Fine-tuning

Interestingly, fine-tuning a model on a specific task (e.g., ImageNet classification) can actually degrade its performance on the broader probing suite. The model learns to discard information irrelevant to the specific task, losing the general-purpose richness of the original pretrained representation.

Results

The quantitative results highlight the performance split. VL models consistently lead in label-based tasks, while vision-only models excel in localization-heavy tasks.

Performance comparison of VL and vision-only models across various probing tasks.

As shown in the table, OFA (VL) achieves the highest accuracy for object prediction on Visual Genome (57.13%), whereas MAE (V) leads in MSCOCO instance segmentation (22.05 mAP) and Part Segmentation (42.30 mIOU).

Conclusion

This study reveals that while language supervision provides a powerful semantic prior, it often comes at the cost of spatial precision. Future research should focus on developing pretraining objectives that combine the semantic richness of multimodal learning with the localization accuracy of unimodal vision training. Bridging this gap is essential for creating truly universal visual backbones capable of excelling at both "what" and "where" tasks.

Summary updated

Abstract

Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether this joint learning paradigm can help understand each individual modality. In this work, we conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing a broad range of tasks, aiming to assess the quality of the learned representations in a nuanced manner. Interestingly, our empirical observations suggest that vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models. Code will be released at https://github.com/Lizw14/visual_probing
BibTeX
@misc{li2022localization,
  author = {Zhuowan Li and Cihang Xie and Benjamin Van Durme and Alan Yuille},
  title = {{Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models}},
  year = {2022},
  eprint = {2212.00281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.00281},
}
Figure