Abstract
Open-vocabulary semantic segmentation (OVSS) is a critical task in computer vision that requires models to segment object categories not seen during training. Current state-of-the-art approaches typically leverage Vision-Language Pre-training (VLP) models like CLIP, matching image pixels to textual category names. However, this paradigm struggles with low-quality or novel textual inputs, such as ambiguous names, neologisms, or categories that are difficult to describe concisely. To overcome these limitations, we propose AttrSeg, a novel framework inspired by human cognition that utilizes attribute decomposition and aggregation. AttrSeg decouples class names into diverse visual attributes and then hierarchically aggregates these descriptions into a robust global classifier. Our approach demonstrates significant improvements across standard benchmarks and a specialized "Fantastic Beasts" dataset designed to test the limits of name-based recognition.
The Problem: Limitations of Name-Based Segmentation
Traditional OVSS models assume that textual category names are accurate, complete, and present in the pre-trained lexicon. In practice, this assumption is often violated. The authors identify three primary failure modes:
- Ambiguity: Brief names like "crane" can refer to multiple distinct concepts (a bird vs. a machine).
- Neologisms: New or fictional words (e.g., "Niffler") are absent from pre-trained vocabularies.
- Unnameability: Some objects are inherently difficult for users to name precisely, leading to vague or incorrect labels.
When a model encounters these issues, its ability to generate accurate segmentation masks collapses because the textual embedding does not align with the visual features. AttrSeg addresses this by mimicking human cognition: if a name is unknown, the system can still recognize the object through a description of its parts, colors, and textures.
The AttrSeg Framework
AttrSeg addresses the "name barrier" by shifting the focus from labels to descriptive attributes. The framework operates in two distinct stages:
1. Decomposition Stage
In this stage, a category name is expanded into a set of fine-grained visual attributes (e.g., "long neck," "pointy ears," "fluffy texture"). The framework employs two strategies for attribute construction:
- LLM-based Generation: For common categories, Large Language Models (LLMs) like ChatGPT are prompted to provide useful visual features for distinguishing the target object.
- Manual Annotation: For human-invented or highly specialized categories (like those in the "Fantastic Beasts" dataset), attributes are manually collected to ensure accuracy.
2. Aggregation Stage
The core technical innovation of AttrSeg is its hierarchical aggregation architecture. Rather than treating attributes as a simple "bag of words," the model learns to integrate them into a single, discriminative global embedding.
The aggregation process involves:
- Fusion Module: Transformer-based layers facilitate deep interaction between visual tokens and attribute tokens, allowing the model to refine representations based on mutual context.
- Clustering Module: Utilizing Slot Attention, the model progressively groups fine-grained attribute embeddings into fewer, more abstract tokens across multiple stages ($L$). This hierarchy allows the model to capture both local details (e.g., texture) and global structures (e.g., overall shape).
- Mask Calculation: The final aggregated embedding serves as a classifier, which is compared against image features via similarity computation to produce the final segmentation mask.
Experimental Evaluation
The effectiveness of AttrSeg was evaluated on several standard benchmarks, including PASCAL-5$^i$, COCO-20$^i$, and PASCAL Context. To specifically test the model's handling of novel and ambiguous terms, the authors introduced the "Fantastic Beasts" dataset, containing 20 categories of fictional creatures from the film series.
Key Results
- Superior Performance: AttrSeg consistently outperformed existing name-based methods. On PASCAL-5$^i$, it achieved a mean IoU (mIoU) of 56.4%, a significant jump from the previous state-of-the-art of 52.3%.
- Handling Fictional Concepts: On the "Fantastic Beasts" dataset, traditional models like LSeg performed poorly because the creature names were unknown. AttrSeg, by using descriptive attributes, achieved an mIoU of 58.7%, proving that attributes serve as a "universal language" for recognition.
- Robustness: Ablation studies showed that the hierarchical aggregation is more effective than flat aggregation, and the model remains robust even when some provided attributes are not visible in the specific image.
Conclusion
AttrSeg represents a significant step forward in open-vocabulary semantic segmentation by moving beyond the constraints of simple class names. By leveraging the descriptive power of attributes and a sophisticated hierarchical aggregation mechanism, the model mimics human-like reasoning to identify and segment novel objects. While challenges remain regarding computational overhead and the need for manual effort in specialized domains, the attribute-driven approach provides a robust foundation for more flexible and interpretable computer vision systems.
Summary updated