TL;DR
UPrompt is a new U-Net-inspired framework for adapting Vision-Language Models (VLMs) that solves the "granularity dilemma"—the trade-off between broad contextual understanding and fine-grained detail. By establishing a bidirectional information flow between different scales of prompts, UPrompt achieves state-of-the-art performance across 17 benchmarks while offering flexible efficiency trade-offs.
What Problem Does This Paper Address?
Adapting massive Vision-Language Models (like CLIP) to specific tasks often relies on prompt learning—optimizing a small set of learnable tokens rather than the whole model. However, current methods usually pick a single "granularity."
This creates a bottleneck:
- Global prompts capture the big picture but fail at fine-grained tasks (e.g., distinguishing between two very similar species of birds).
- Local prompts focus on specific image regions but lose the overall context, leading to poor generalization across different types of tasks.
The authors identify this as a "granularity bottleneck" that limits how well these models can adapt to diverse real-world scenarios.
Methods
Inspired by the U-Net architecture famous in medical imaging, the authors propose UPrompt, which constructs parallel hierarchies for both vision and language.
- Multi-Granularity Pathways:
- Vision: The model uses progressive spatial pooling to create a hierarchy of visual tokens (from a detailed 14x14 grid down to a single global 1x1 token).
- Language: Since text doesn't have a geometric structure like pixels, the authors use an LLM (Llama 3) to generate a semantic hierarchy, ranging from simple class names to highly detailed descriptions.
- Bidirectional Connection: This is the "secret sauce" that links the scales:
- Coarse-to-Fine Cascaded Enhancement (CE): During the forward pass, global contextual information is injected into fine-grained features using cross-attention. This ensures that local details are understood within the context of the whole scene.
- Fine-to-Coarse Hierarchical Supervision (HS): During training, the most detailed (finest) level acts as a "teacher." Through knowledge distillation, it regularizes the coarser levels to ensure they remain semantically consistent and don't "drift" away from the core meaning.
Datasets
The authors conducted an exhaustive evaluation using 17 different benchmarks:
- Cross-modal Retrieval: Flickr30K and MSCOCO.
- Classification & Generalization: 11 diverse datasets including ImageNet, StanfordCars (fine-grained), FGVCAircraft (fine-grained), and EuroSAT (satellite imagery).
- Out-of-Distribution (OOD): ImageNet-A, -R, -Sketch, and -V2.
- Generalized Zero-Shot Learning: CUB-200 and AWA2.
Key Insights and Ideas
- The U-Net Analogy for Prompts: The most striking idea is treating abstract prompt embeddings like a spatial grid. By applying U-Net principles (skip connections and hierarchical fusion) to prompt space, they bridge the gap between high-level semantics and low-level details.
- Solving Semantic Drift: Coarser representations often become "blurry" or ambiguous during training. The Fine-to-Coarse Supervision ensures that even the most compressed prompts benefit from the rich information available at the finest scale.
- Performance-Efficiency Knob: UPrompt allows users to choose their granularity at inference time. You can use the "Fine" head for maximum accuracy or the "Coarse" head for lightning-fast inference with minimal performance loss.
Key Results
UPrompt consistently outperformed existing state-of-the-art (SOTA) methods:
- Retrieval: On the MSCOCO dataset, it outperformed MAMET and VPKE by +4.1 and +7.3 rSum, respectively.
- Generalization: In base-to-novel generalization (learning new classes), it surpassed CoCoA-Mix by +5.09%.
- Efficiency: The "medium-grained" version of UPrompt matched the performance of the SOTA PSRC model but at only 1/3 of the computational cost.
- Robustness: The method proved effective across different VLM backbones (SigLIP, EVA-CLIP) and was not sensitive to which specific LLM was used to generate the text hierarchy.
Limitations and Future Work
The authors acknowledge one primary limitation: the granularity hierarchy currently requires manual design. The pooling sizes for images and the specific prompts for the LLM-generated text hierarchy are pre-defined.
Future Work: The authors suggest exploring adaptive granularity selection, where the model could automatically determine the optimal level of detail needed based on the complexity of the specific task or image at hand.
Summary updated