Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation

Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Ya Zhang, Yanfeng Wang

Abstract

Open-vocabulary semantic segmentation (OVSS) is a critical task in computer vision that requires models to segment object categories not seen during training. Current state-of-the-art approaches typically leverage Vision-Language Pre-training (VLP) models like CLIP, matching image pixels to textual category names. However, this paradigm struggles with low-quality or novel textual inputs, such as ambiguous names, neologisms, or categories that are difficult to describe concisely. To overcome these limitations, we propose AttrSeg, a novel framework inspired by human cognition that utilizes attribute decomposition and aggregation. AttrSeg decouples class names into diverse visual attributes and then hierarchically aggregates these descriptions into a robust global classifier. Our approach demonstrates significant improvements across standard benchmarks and a specialized "Fantastic Beasts" dataset designed to test the limits of name-based recognition.

The Problem: Limitations of Name-Based Segmentation

Traditional OVSS models assume that textual category names are accurate, complete, and present in the pre-trained lexicon. In practice, this assumption is often violated. The authors identify three primary failure modes:

  1. Ambiguity: Brief names like "crane" can refer to multiple distinct concepts (a bird vs. a machine).
  2. Neologisms: New or fictional words (e.g., "Niffler") are absent from pre-trained vocabularies.
  3. Unnameability: Some objects are inherently difficult for users to name precisely, leading to vague or incorrect labels.
Motivation and overview of the AttrSeg framework, highlighting the failures of name-only approaches and the proposed decomposition-aggregation solution.

When a model encounters these issues, its ability to generate accurate segmentation masks collapses because the textual embedding does not align with the visual features. AttrSeg addresses this by mimicking human cognition: if a name is unknown, the system can still recognize the object through a description of its parts, colors, and textures.

The AttrSeg Framework

AttrSeg addresses the "name barrier" by shifting the focus from labels to descriptive attributes. The framework operates in two distinct stages:

1. Decomposition Stage

In this stage, a category name is expanded into a set of fine-grained visual attributes (e.g., "long neck," "pointy ears," "fluffy texture"). The framework employs two strategies for attribute construction:

  • LLM-based Generation: For common categories, Large Language Models (LLMs) like ChatGPT are prompted to provide useful visual features for distinguishing the target object.
  • Manual Annotation: For human-invented or highly specialized categories (like those in the "Fantastic Beasts" dataset), attributes are manually collected to ensure accuracy.

2. Aggregation Stage

The core technical innovation of AttrSeg is its hierarchical aggregation architecture. Rather than treating attributes as a simple "bag of words," the model learns to integrate them into a single, discriminative global embedding.

Detailed architecture of the AttrSeg framework, showing the Decomposition Stage (LLM and manual) and the Aggregation Stage featuring Fusion and Clustering modules.

The aggregation process involves:

  • Fusion Module: Transformer-based layers facilitate deep interaction between visual tokens and attribute tokens, allowing the model to refine representations based on mutual context.
  • Clustering Module: Utilizing Slot Attention, the model progressively groups fine-grained attribute embeddings into fewer, more abstract tokens across multiple stages ($L$). This hierarchy allows the model to capture both local details (e.g., texture) and global structures (e.g., overall shape).
  • Mask Calculation: The final aggregated embedding serves as a classifier, which is compared against image features via similarity computation to produce the final segmentation mask.

Experimental Evaluation

The effectiveness of AttrSeg was evaluated on several standard benchmarks, including PASCAL-5$^i$, COCO-20$^i$, and PASCAL Context. To specifically test the model's handling of novel and ambiguous terms, the authors introduced the "Fantastic Beasts" dataset, containing 20 categories of fictional creatures from the film series.

Key Results

  • Superior Performance: AttrSeg consistently outperformed existing name-based methods. On PASCAL-5$^i$, it achieved a mean IoU (mIoU) of 56.4%, a significant jump from the previous state-of-the-art of 52.3%.
  • Handling Fictional Concepts: On the "Fantastic Beasts" dataset, traditional models like LSeg performed poorly because the creature names were unknown. AttrSeg, by using descriptive attributes, achieved an mIoU of 58.7%, proving that attributes serve as a "universal language" for recognition.
  • Robustness: Ablation studies showed that the hierarchical aggregation is more effective than flat aggregation, and the model remains robust even when some provided attributes are not visible in the specific image.

Conclusion

AttrSeg represents a significant step forward in open-vocabulary semantic segmentation by moving beyond the constraints of simple class names. By leveraging the descriptive power of attributes and a sophisticated hierarchical aggregation mechanism, the model mimics human-like reasoning to identify and segment novel objects. While challenges remain regarding computational overhead and the need for manual effort in specialized domains, the attribute-driven approach provides a robust foundation for more flexible and interpretable computer vision systems.

Summary updated

Abstract

Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent works explore vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when meet with ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users. To address these issues, this work proposes a novel decomposition-aggregation framework, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to enrich semantic contexts. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labelling for human-invented categories. In the aggregation stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation is further designed to achieve multi-level alignment and deep fusion between vision and text. The final result is obtained by computing the embedding similarity between aggregated attributes and images. To evaluate the effectiveness, we annotate three datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation.
BibTeX
@inproceedings{ma2023open,
  author = {Chaofan Ma and Yuhuan Yang and Chen Ju and Fei Zhang and Ya Zhang and Yanfeng Wang},
  title = {{Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=ESEM1lNoeS},
}
Figure