Abstract
Image-text models excel at image-level tasks but struggle with detailed visual understanding. While these models provide strong visual-language alignment, segmentation models like SAM2 offer precise spatial boundaries for objects. To this end, we propose TextRegion, a simple, effective, and training-free framework that combines the strengths of image-text models and SAM2 to generate powerful text-aligned region tokens. These tokens enable detailed visual understanding while preserving open-vocabulary capabilities. They can be directly applied to various downstream tasks, including open-world semantic segmentation, referring expression comprehension, and grounding. We conduct extensive evaluations and consistently achieve superior or competitive performance compared to state-of-the-art training-free methods. Additionally, our framework is compatible with many image-text models, making it highly practical and easily extensible as stronger models emerge.
Introduction
Contrastive image-text models like CLIP have revolutionized computer vision by providing a "universal" vocabulary for images. However, these models are primarily trained to understand images at a global level—they can identify a "golden retriever" in a photo, but they struggle to precisely localize its features or distinguish it from a "yellow lab" in the same frame. This "localization gap" is typically addressed through expensive training-based methods or noisy training-free patch-based approaches that lack spatial coherence.
The TextRegion Framework
TextRegion is a simple, training-free framework that bridges the gap between global image-text alignment and precise spatial segmentation. By extracting "region tokens" directly from the internal features of frozen foundation models, it enables high-performance open-vocabulary segmentation and visual grounding without any additional training.
The core of TextRegion is a technique called Mask-based Attention Pooling. Instead of relying on the standard global [CLS] token, the framework extracts features from the model's internal layers:
- Mask Generation: SAM2 is used to generate precise, high-resolution masks for all potential objects in an image.
- Feature Extraction: The image is fed through a frozen vision encoder. The researchers found that the Value (V) vectors in the final attention block are semantically rich but spatially localized, making them ideal for region-level tasks.
- Region Token Creation: By pooling these Value vectors according to the SAM2 masks, the framework creates a "Region Token" for every object. This token lives in the same mathematical space as text embeddings, allowing for direct comparison with natural language queries.
To further refine performance, TextRegion utilizes Global Patch Removal to filter out context-heavy "register" patches and Multi-resolution Encoding to improve the detection of small objects.
Experimental Evaluation
The authors evaluated TextRegion across a massive battery of standard benchmarks, including PASCAL VOC, COCO-Stuff, and RefCOCO.
Key Results
- Segmentation: On the COCO-Stuff benchmark, TextRegion achieved 41.1 mIoU using a standard CLIP ViT-B/16, significantly outperforming previous training-free leaders like Trident (38.6) and MaskCLIP (23.3).
- Referring Expressions: In tasks requiring the model to find specific objects based on complex descriptions, TextRegion outperformed specialized baselines across all RefCOCO variants.
- Efficiency: The method is highly efficient, requiring only 0.20 seconds per image on a single A100 GPU.
Key Insights
A critical finding of this work is that the Value projection in the transformer's final block is far superior for localization than the attention weights or the output features. This suggests that the semantic identity of a patch is preserved in the values before they are mixed by global attention. Furthermore, because the framework is architecture-agnostic, it can be easily extended as stronger vision models emerge.
Conclusion and Future Work
While TextRegion provides a powerful training-free solution, it is currently limited by the quality of SAM2's mask proposals and the reasoning capabilities of the CLIP text encoder. Future directions include integrating TextRegion with Large Language Models (LLMs) to solve the reasoning bottleneck, allowing the system to interpret complex queries into simpler terms that the region tokens can easily match.