AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation

Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Ya Zhang, Yanfeng Wang

TL;DR

AttrSeg introduces a novel "decomposition-aggregation" framework for open-vocabulary semantic segmentation, moving away from a strict reliance on exact class names. By breaking down categories into descriptive visual attributes and hierarchically recombining them, it successfully segments ambiguous, rare, or entirely new objects that stump traditional vision-language models.

What Problem Does This Paper Address?

Current open-vocabulary semantic segmentation (OVSS) models rely heavily on vision-language pre-training (VLP) frameworks like CLIP to map text labels directly to image pixels. However, this paradigm operates under an unrealistic assumption: that users will always provide perfect, unambiguous category names that already exist within the model's pre-trained vocabulary.

In real-world scenarios, this assumption fails in three major ways:

  1. Ambiguity: Short or incomplete names (e.g., "crane") can refer to multiple distinct concepts (a bird vs. a construction machine).
  2. Neologisms: Fictional terms, brand new words, or slang (e.g., "Niffler") do not exist in pre-trained lexicons.
  3. Unnameability: Rare, highly specialized, or obscure objects are often difficult for everyday users to name accurately.

When faced with these low-quality or unseen textual inputs, standard OVSS models suffer a severe drop in segmentation performance.

Figure 1: Left: Open-vocabulary semantic segmentation (OVSS) assumes the given new textual categories are accurate, complete, and exist in pre-trained lexicons. However, in real-life situations, practical uses are limited due to textual ambiguity, neologisms, and unnameability. Middle: We propose a novel attribute decomposition-aggregation framework where vanilla class names are first decomposed into various attribute descriptions (decomposition stage), and then, different attribute representations are aggregated hierarchically into a final class representation for further segmentations (aggregation stage). Right: Our framework successfully addresses the aforementioned issues and facilitates more practical applications of OVSS in real-world scenarios.

Methods

Inspired by cognitive psychology—specifically how humans learn to recognize new concepts by identifying their constituent parts and features—the authors propose AttrSeg. The framework divides the task into two core stages:

1. Decomposition Stage

Instead of passing a raw class name to the model, AttrSeg decouples it into a collection of diverse visual attributes (spanning color, shape, parts, and materials). The authors design two strategies for this:

  • LLM Generation: For common but potentially ambiguous categories, they leverage Large Language Models (like ChatGPT) with structured question templates to automatically generate clean, semicolon-separated attribute lists (e.g., "flamingo" $\rightarrow$ wading bird; pink or reddish color; long legs; long neck).
  • Manual Collection: For fictional or completely novel categories where LLMs lack knowledge, attributes are collected via human annotation.

2. Aggregation Stage

Feeding a raw list of attributes into a text encoder isn't enough; the attributes must be contextually mapped to the image. AttrSeg introduces a Hierarchical Aggregation Architecture consisting of $L$ stages to progressively fuse and compress attribute tokens into a single global representation.

Each stage alternates between two components:

  • Fusion Module: Uses transformer encoder layers to facilitate global, long-range interactions between visual tokens and attribute tokens.
  • Clustering Module: Utilizes learnable cluster tokens as centers. By applying cross-attention to both the visual and attribute embeddings, followed by slot attention, it groups multiple detailed attribute tokens into fewer, more abstract tokens.

Ultimately, the attributes are condensed into a single token, and the final segmentation mask is generated by calculating the pixel-level similarity between this token and the visual embeddings.

Figure 2: Overview of Attribute Decomposition-Aggregation Framework. (a) Decomposition stage aims to decouple vanilla class names into various attribute descriptions. We design two strategies to build attributes, i.e., using LLMs and manual collections. (b) Aggregation stage aims to merge separated attribute representations into an integrated global description. We propose to hierarchically aggregate attribute tokens to one specific token in L stages. Each stage alternates a fusion module and a clustering module. Masks are generated by calculating the similarity.

Datasets

The authors evaluate AttrSeg across several standard benchmarks augmented with LLM-generated attributes, as well as a brand-new dataset designed to simulate real-world vocabulary gaps:

  • Standard Benchmarks: PASCAL-5i, COCO-20i, Pascal VOC, and Pascal Context.
  • Fantastic Beasts Dataset (New): To rigorously test the framework against neologisms and unnameable categories, the authors curated a dataset of 20 magical creatures from the Fantastic Beasts film series (e.g., Bowtruckle, Demiguise, Mooncalf). It contains images, segmentation masks, and detailed attributes carefully labeled by fan volunteers. Standard CLIP models achieve an average classification accuracy of just 11% on these categories, highlighting their obscurity.

Key Insights and Ideas

  • Bypassing Vocabulary Limits via Visual Anchors: By describing what an object looks like rather than what it is called, AttrSeg allows pre-trained models to reason about completely alien words. Fictional terms are successfully anchored to known visual concepts (e.g., recognizing a "Kappa" via "scaly, green skin; sharp claws and beak").
  • Perceptual Hierarchy Alignment: Visual features are naturally processed at different granularities (local textures vs. global shapes). AttrSeg’s multi-stage clustering mirrors this; lower stages focus on local regions and low-level features, while higher stages concentrate on semantically abstract, global information.
  • Robustness to Attribute Noise: The aggregation module is surprisingly resilient to noisy or inaccurate attributes (e.g., an attribute like "four-legged" provided for a dog that happens to be lying down in a way that hides its legs). Because the clustering centers exchange information with the visual tokens, the model learns to dynamically down-weight or ignore attributes that lack visual evidence in the image.

Key Results

  • Surpassing SOTA with Less Direct Input: On PASCAL-5i and COCO-20i, AttrSeg consistently outperforms competitive baselines like LSeg. Remarkably, AttrSeg using only attribute descriptions often beats SOTA models that have access to the exact class names (achieving 56.4% mIoU on PASCAL-5i using a ViT-L backbone).
  • Conquering Unseen Categories: On the challenging Fantastic Beasts dataset, traditional models fail completely when given class names (mIoU < 10%). AttrSeg establishes a powerful new baseline, reaching up to 61.9% mIoU by leveraging its attribute decomposition-aggregation pipeline.

Limitations and Future Work

  • Dependence on Foundation Model Biases: Since AttrSeg builds upon pre-trained VLPs (CLIP) and LLMs (ChatGPT), it inherently inherits any data biases or blind spots present in those foundation models.
  • Inference Overhead: Operating the framework requires a descriptive list of attributes at inference time. While this can be automated via LLMs for common categories, it requires manual user input or specialized knowledge bases when dealing with entirely new, custom, or user-invented categories.

Summary updated

Abstract

Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent studies have explored vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when encountering ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users. To address these issues, this work proposes a novel attribute decomposition-aggregation framework, AttrSeg, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to complement semantic contexts from multiple perspectives. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labeling for human-invented categories. In the aggregation stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation architecture is further proposed to achieve multi-level aggregations, leveraging the meticulously designed clustering module. The final results are obtained by computing the similarity between aggregated attributes and images embeddings. To evaluate the effectiveness, we annotate three types of datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation.
BibTeX
@inproceedings{ma2023attrseg,
  author = {Chaofan Ma and Yuhuan Yang and Chen Ju and Fei Zhang and Ya Zhang and Yanfeng Wang},
  title = {{AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=ESEM1lNoeS},
}
Figure