LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment

Junyi Hu, Qiji Zhou, Lei Zhang, Yue ZhangarXiv 2026 · DOI · arXiv

TL;DR

This paper introduces LAGO, a novel framework for zero-shot visual-text alignment that focuses on identifying the most relevant image regions for matching text descriptions. By adaptively guiding the search for these regions based on confidence, LAGO achieves state-of-the-art performance with significantly fewer computational resources than previous methods.

What Problem Does This Paper Address?

Zero-shot recognition aims to identify objects in images using text descriptions without explicit training data for those objects. While powerful models like CLIP have made strides, they often rely on matching text to the entire image. This approach struggles with "fine-grained" recognition, where crucial details are localized to specific object parts, attributes, or textures. Existing methods that try to address this by examining multiple image regions often rely on generating a vast number of random crops, which is computationally expensive and can lead to redundant or irrelevant information. Furthermore, aggressively using early, potentially inaccurate, text guidance to select regions can create a "prediction loop," where errors in region selection reinforce themselves, leading to worse performance.

Methods

LAGO tackles these challenges with a two-stage, confidence-aware approach:

  1. Class-Agnostic Object-Centric Initialization: Instead of random crops, LAGO first uses an object proposal generator (FastSAM) to identify salient object regions. This provides a stable, visually grounded starting point without relying on uncertain class predictions.
  2. Adaptive Text-Guided Refinement: This stage refines the region selection by incorporating text guidance. Crucially, the strength of this guidance is adaptively controlled by an intermediate confidence score. High-confidence predictions lead to stronger text-guided focusing, while low-confidence predictions maintain more visual diversity to avoid premature commitment to potentially wrong regions. This "confidence-aware" mechanism is designed to break the prediction loop.
  3. Object-Context Dual-Channel Aggregation: LAGO combines evidence from object-focused regions, contextual crops, and the full image to make a final decision.

Datasets

LAGO was evaluated on six standard zero-shot benchmarks:

  • ImageNet
  • CUB (Caltech-UCSD Birds 200)
  • Oxford Pets
  • DTD (Describable Textures Dataset)
  • Food101
  • Places365

It was also tested on four natural distribution-shift benchmarks derived from ImageNet:

  • ImageNet-V2
  • ImageNet-R (Rendition)
  • ImageNet-S (Sketch)
  • ImageNet-A (Adversarial)

Key Insights and Ideas

  • Region Discovery is Key: The paper argues that for fine-grained zero-shot recognition, how you find relevant image regions is as important as what text descriptions you use.
  • The "Prediction Loop" is Real: Aggressively using early, uncertain text guidance to select image regions can lead to a self-reinforcing error cycle, hindering performance.
  • Confidence is Crucial for Adaptation: Dynamically adjusting text guidance based on prediction confidence is vital for robust performance, especially under distribution shifts where initial predictions are less reliable.
  • Efficiency Through Selectivity: LAGO demonstrates that a smaller, more intelligently selected set of regions can outperform a larger, randomly sampled set, leading to significant computational savings.

Key Results

  • State-of-the-Art Performance: LAGO achieves top accuracy on standard zero-shot benchmarks, outperforming strong baselines like WCA, BiFTA, and FG-CLIP.
  • Robustness to Distribution Shifts: The method shows significant gains on challenging distribution-shift datasets, indicating its ability to generalize.
  • Efficiency Gains: LAGO requires substantially fewer candidate regions at inference time compared to methods like WCA, leading to more efficient computation. For example, on ImageNet-V2, LAGO consistently outperforms WCA as the candidate region budget increases, with gains up to +1.12% at a budget of 64 regions.
  • Qualitative Improvements: Visualizations show that LAGO effectively refines region selection, focusing on more discriminative and semantically relevant areas, especially in challenging cases.

Limitations and Future Work

The authors acknowledge that LAGO still relies on the quality of external object proposals and offline text descriptions. Its multi-stage inference pipeline is also more complex than single-pass global alignment. Future work could focus on:

  • Stronger proposal generation.
  • More robust text prototype construction.
  • Simpler, leaner, or more integrated inference schemes for multi-stage localization and refinement.
  • Addressing potential biases inherited from pretrained models and text descriptions.

Summary updated

BibTeX
@misc{hu2026lago,
  author = {Junyi Hu and Qiji Zhou and Lei Zhang and Yue Zhang},
  title = {{LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment}},
  year = {2026},
  eprint = {2605.08156},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.08156},
}
Figure