UPrompt: Bidirectional Multi-granularity Learning for Vision-Language Models

Biao Chen, Xiangxu Zhao, Yunya Zhang, Yuchen Wang, Mengmeng Jing, Lin ZuoOpenReview 2026 · Submission to ICLR 2026 · Rejected · OpenReview

TL;DR

UPrompt is a new U-Net-inspired framework for adapting Vision-Language Models (VLMs) that solves the "granularity dilemma"—the trade-off between broad contextual understanding and fine-grained detail. By establishing a bidirectional information flow between different scales of prompts, UPrompt achieves state-of-the-art performance across 17 benchmarks while offering flexible efficiency trade-offs.

What Problem Does This Paper Address?

Adapting massive Vision-Language Models (like CLIP) to specific tasks often relies on prompt learning—optimizing a small set of learnable tokens rather than the whole model. However, current methods usually pick a single "granularity."

This creates a bottleneck:

  • Global prompts capture the big picture but fail at fine-grained tasks (e.g., distinguishing between two very similar species of birds).
  • Local prompts focus on specific image regions but lose the overall context, leading to poor generalization across different types of tasks.

The authors identify this as a "granularity bottleneck" that limits how well these models can adapt to diverse real-world scenarios.

Methods

Inspired by the U-Net architecture famous in medical imaging, the authors propose UPrompt, which constructs parallel hierarchies for both vision and language.

  1. Multi-Granularity Pathways:
    • Vision: The model uses progressive spatial pooling to create a hierarchy of visual tokens (from a detailed 14x14 grid down to a single global 1x1 token).
  2. Language: Since text doesn't have a geometric structure like pixels, the authors use an LLM (Llama 3) to generate a semantic hierarchy, ranging from simple class names to highly detailed descriptions.
  3. Bidirectional Connection: This is the "secret sauce" that links the scales:
  4. Coarse-to-Fine Cascaded Enhancement (CE): During the forward pass, global contextual information is injected into fine-grained features using cross-attention. This ensures that local details are understood within the context of the whole scene.
  5. Fine-to-Coarse Hierarchical Supervision (HS): During training, the most detailed (finest) level acts as a "teacher." Through knowledge distillation, it regularizes the coarser levels to ensure they remain semantically consistent and don't "drift" away from the core meaning.

Datasets

The authors conducted an exhaustive evaluation using 17 different benchmarks:

  • Cross-modal Retrieval: Flickr30K and MSCOCO.
  • Classification & Generalization: 11 diverse datasets including ImageNet, StanfordCars (fine-grained), FGVCAircraft (fine-grained), and EuroSAT (satellite imagery).
  • Out-of-Distribution (OOD): ImageNet-A, -R, -Sketch, and -V2.
  • Generalized Zero-Shot Learning: CUB-200 and AWA2.

Key Insights and Ideas

  • The U-Net Analogy for Prompts: The most striking idea is treating abstract prompt embeddings like a spatial grid. By applying U-Net principles (skip connections and hierarchical fusion) to prompt space, they bridge the gap between high-level semantics and low-level details.
  • Solving Semantic Drift: Coarser representations often become "blurry" or ambiguous during training. The Fine-to-Coarse Supervision ensures that even the most compressed prompts benefit from the rich information available at the finest scale.
  • Performance-Efficiency Knob: UPrompt allows users to choose their granularity at inference time. You can use the "Fine" head for maximum accuracy or the "Coarse" head for lightning-fast inference with minimal performance loss.

Key Results

UPrompt consistently outperformed existing state-of-the-art (SOTA) methods:

  • Retrieval: On the MSCOCO dataset, it outperformed MAMET and VPKE by +4.1 and +7.3 rSum, respectively.
  • Generalization: In base-to-novel generalization (learning new classes), it surpassed CoCoA-Mix by +5.09%.
  • Efficiency: The "medium-grained" version of UPrompt matched the performance of the SOTA PSRC model but at only 1/3 of the computational cost.
  • Robustness: The method proved effective across different VLM backbones (SigLIP, EVA-CLIP) and was not sensitive to which specific LLM was used to generate the text hierarchy.

Limitations and Future Work

The authors acknowledge one primary limitation: the granularity hierarchy currently requires manual design. The pooling sizes for images and the specific prompts for the LLM-generated text hierarchy are pre-defined.

Future Work: The authors suggest exploring adaptive granularity selection, where the model could automatically determine the optimal level of detail needed based on the complexity of the specific task or image at hand.

Summary updated

Abstract

The prompt learning paradigm for vision-language models is effective yet faces the dilemma of balancing granularity: global prompts lack fine-grained semantic awareness, while local prompts ignore overall contextual associations, leading to limited cross task generalization. This dilemma exists in dense prediction tasks. Inspired by the U-Net framework that unifying multi-level representations across different granularities, we propose UPrompt, a novel bidirectional multi-granularity prompt learning framework for vision-language models. Similar to how U-Net integrates fine and coarse features through symmetric encoder-decoder pathways with cross-level connections, UPrompt constructs parallel multi-granularity representations in both visual and textual modalities, where coarse-to-fine cascaded enhancement propagates global contextual information to refine local details, while fine-to-coarse hierarchical supervision ensures semantic consistency across scales. Extensive experiments on 17 benchmarks validate our effectiveness. Our method outperforms MAMET and VPKE by +4.1 and +7.3 rSum on MSCOCO, surpasses CoCoA-Mix by +5.09\% in base-to-novel generalization, while maintaining competitive performance with minimal overhead (coarse-grained) and matching PSRC with 1/3 cost (medium-grained).
BibTeX
@misc{chen2026uprompt,
  author = {Biao Chen and Xiangxu Zhao and Yunya Zhang and Yuchen Wang and Mengmeng Jing and Lin Zuo},
  title = {{UPrompt: Bidirectional Multi-granularity Learning for Vision-Language Models}},
  year = {2026},
  url = {https://openreview.net/forum?id=MgnQ0dHoUJ},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}
Figure