TL;DR
AttrSeg introduces a novel "decomposition-aggregation" framework for open-vocabulary semantic segmentation, moving away from a strict reliance on exact class names. By breaking down categories into descriptive visual attributes and hierarchically recombining them, it successfully segments ambiguous, rare, or entirely new objects that stump traditional vision-language models.
What Problem Does This Paper Address?
Current open-vocabulary semantic segmentation (OVSS) models rely heavily on vision-language pre-training (VLP) frameworks like CLIP to map text labels directly to image pixels. However, this paradigm operates under an unrealistic assumption: that users will always provide perfect, unambiguous category names that already exist within the model's pre-trained vocabulary.
In real-world scenarios, this assumption fails in three major ways:
- Ambiguity: Short or incomplete names (e.g., "crane") can refer to multiple distinct concepts (a bird vs. a construction machine).
- Neologisms: Fictional terms, brand new words, or slang (e.g., "Niffler") do not exist in pre-trained lexicons.
- Unnameability: Rare, highly specialized, or obscure objects are often difficult for everyday users to name accurately.
When faced with these low-quality or unseen textual inputs, standard OVSS models suffer a severe drop in segmentation performance.
Methods
Inspired by cognitive psychology—specifically how humans learn to recognize new concepts by identifying their constituent parts and features—the authors propose AttrSeg. The framework divides the task into two core stages:
1. Decomposition Stage
Instead of passing a raw class name to the model, AttrSeg decouples it into a collection of diverse visual attributes (spanning color, shape, parts, and materials). The authors design two strategies for this:
- LLM Generation: For common but potentially ambiguous categories, they leverage Large Language Models (like ChatGPT) with structured question templates to automatically generate clean, semicolon-separated attribute lists (e.g., "flamingo" $\rightarrow$ wading bird; pink or reddish color; long legs; long neck).
- Manual Collection: For fictional or completely novel categories where LLMs lack knowledge, attributes are collected via human annotation.
2. Aggregation Stage
Feeding a raw list of attributes into a text encoder isn't enough; the attributes must be contextually mapped to the image. AttrSeg introduces a Hierarchical Aggregation Architecture consisting of $L$ stages to progressively fuse and compress attribute tokens into a single global representation.
Each stage alternates between two components:
- Fusion Module: Uses transformer encoder layers to facilitate global, long-range interactions between visual tokens and attribute tokens.
- Clustering Module: Utilizes learnable cluster tokens as centers. By applying cross-attention to both the visual and attribute embeddings, followed by slot attention, it groups multiple detailed attribute tokens into fewer, more abstract tokens.
Ultimately, the attributes are condensed into a single token, and the final segmentation mask is generated by calculating the pixel-level similarity between this token and the visual embeddings.
Datasets
The authors evaluate AttrSeg across several standard benchmarks augmented with LLM-generated attributes, as well as a brand-new dataset designed to simulate real-world vocabulary gaps:
- Standard Benchmarks: PASCAL-5i, COCO-20i, Pascal VOC, and Pascal Context.
- Fantastic Beasts Dataset (New): To rigorously test the framework against neologisms and unnameable categories, the authors curated a dataset of 20 magical creatures from the Fantastic Beasts film series (e.g., Bowtruckle, Demiguise, Mooncalf). It contains images, segmentation masks, and detailed attributes carefully labeled by fan volunteers. Standard CLIP models achieve an average classification accuracy of just 11% on these categories, highlighting their obscurity.
Key Insights and Ideas
- Bypassing Vocabulary Limits via Visual Anchors: By describing what an object looks like rather than what it is called, AttrSeg allows pre-trained models to reason about completely alien words. Fictional terms are successfully anchored to known visual concepts (e.g., recognizing a "Kappa" via "scaly, green skin; sharp claws and beak").
- Perceptual Hierarchy Alignment: Visual features are naturally processed at different granularities (local textures vs. global shapes). AttrSeg’s multi-stage clustering mirrors this; lower stages focus on local regions and low-level features, while higher stages concentrate on semantically abstract, global information.
- Robustness to Attribute Noise: The aggregation module is surprisingly resilient to noisy or inaccurate attributes (e.g., an attribute like "four-legged" provided for a dog that happens to be lying down in a way that hides its legs). Because the clustering centers exchange information with the visual tokens, the model learns to dynamically down-weight or ignore attributes that lack visual evidence in the image.
Key Results
- Surpassing SOTA with Less Direct Input: On PASCAL-5i and COCO-20i, AttrSeg consistently outperforms competitive baselines like LSeg. Remarkably, AttrSeg using only attribute descriptions often beats SOTA models that have access to the exact class names (achieving 56.4% mIoU on PASCAL-5i using a ViT-L backbone).
- Conquering Unseen Categories: On the challenging Fantastic Beasts dataset, traditional models fail completely when given class names (mIoU < 10%). AttrSeg establishes a powerful new baseline, reaching up to 61.9% mIoU by leveraging its attribute decomposition-aggregation pipeline.
Limitations and Future Work
- Dependence on Foundation Model Biases: Since AttrSeg builds upon pre-trained VLPs (CLIP) and LLMs (ChatGPT), it inherently inherits any data biases or blind spots present in those foundation models.
- Inference Overhead: Operating the framework requires a descriptive list of attributes at inference time. While this can be automated via LLMs for common categories, it requires manual user input or specialized knowledge bases when dealing with entirely new, custom, or user-invented categories.
Summary updated