Focus-Consistent Multi-Level Aggregation for Compositional Zero-Shot Learning

Fengyuan Dai, Siteng Huang, Min Zhang, Biao Gong, Donglin Wang

Compositional Zero-Shot Learning (CZSL) aims to recognize novel concepts by combining known "primitives," such as identifying a "blue apple" after seeing "red apples" and "blue cars." While modern CZSL models typically use a three-branch architecture (attribute, object, and composition), they often suffer from two critical limitations: feature uniformity and spatial disconnect.

The Problem: Uniformity and Disconnect

Most existing methods feed identical high-level features from the final layer of a backbone network to all three classification branches. This is suboptimal because attributes (e.g., "striped") often rely on fine-grained, low-level textures, whereas objects (e.g., "zebra") require high-level semantic shapes. Furthermore, without coordination, the attribute branch might focus on the background while the object branch focuses on the subject, leading to "hallucinated" predictions.

Conceptual comparison between existing methods and the proposed FOMA approach.

Methodology: FOMA

The proposed Focus-Consistent Multi-Level Aggregation (FOMA) framework addresses these issues through two primary innovations: Multi-Level Feature Aggregation (MFA) and a Focus-Consistent Constraint.

Multi-Level Feature Aggregation (MFA)

Instead of using only the final backbone layer, MFA extracts features from multiple depths (low, middle, and high levels). An "Aggregation Predictor" sub-network dynamically calculates weights for these features based on the input image. This allows the attribute branch to prioritize texture-rich low-level features, while the object branch focuses on semantic-rich high-level features.

Focus-Consistent Constraint

To ensure spatial alignment, FOMA introduces a penalty loss ($L_f$) based on attention maps. This constraint encourages the spatial focus of the composition branch to align with the combined focus of the attribute and object branches. By enforcing this consistency, the branches implicitly share information; for instance, if the object branch locates a "dog," the constraint guides the attribute branch to the same region to identify "wet" fur.

The FOMA architecture, highlighting the Multi-level Feature Aggregation and Focus-Consistent Constraint modules.

Attention Pooling

FOMA replaces standard Global Average Pooling (GAP) with a transformer-inspired attention mechanism. This preserves critical spatial information that is typically lost when flattening features into a single vector, further enhancing the model's ability to localize relevant primitives.

Experimental Results

FOMA was evaluated on three major CZSL benchmarks: UT-Zappos, C-GQA, and Clothing16K. It achieved state-of-the-art (SOTA) performance across nearly all metrics:

  • C-GQA & Clothing16K: FOMA reached SOTA results in Seen accuracy, Unseen accuracy, Area Under the Curve (AUC), and Harmonic Mean (HM). On C-GQA, it improved the HM from 14.5 to 15.2.
  • UT-Zappos: The model achieved the highest Unseen accuracy (68.0%) and AUC (33.1%), demonstrating superior generalization to novel pairs.

Qualitative visualizations of attention maps confirmed that FOMA consistently identifies the correct regions of interest, such as focusing on the texture of a "red couch" rather than misidentifying it based on shape alone.

Conclusion and Future Work

FOMA demonstrates that tailoring visual features to specific primitive types while enforcing spatial consistency significantly improves compositional reasoning. While highly effective, future work could explore more modern backbones like Vision Transformers (ViTs) and extend the framework to "open-world" settings where primitives themselves may be unseen during training.

Summary updated

Abstract

To transfer knowledge from seen attribute-object compositions to recognize unseen ones, recent compositional zero-shot learning (CZSL) methods mainly discuss the optimal classification branches to identify the elements, leading to the popularity of employing a three-branch architecture. However, these methods mix up the underlying relationship among the branches, in the aspect of consistency and diversity. Specifically, consistently providing the highest-level features for all three branches increases the difficulty in distinguishing classes that are superficially similar. Furthermore, a single branch may focus on suboptimal regions when spatial messages are not shared between the personalized branches. Recognizing these issues and endeavoring to address them, we propose a novel method called Focus-Consistent Multi-Level Aggregation (FOMA). Our method incorporates a Multi-Level Feature Aggregation (MFA) module to generate personalized features for each branch based on the image content. Additionally, a Focus-Consistent Constraint encourages a consistent focus on the informative regions, thereby implicitly exchanging spatial information between all branches. Extensive experiments on three benchmark datasets (UT-Zappos, C-GQA, and Clothing16K) demonstrate that our FOMA outperforms SOTA.
BibTeX
@misc{dai2024focus,
  author = {Fengyuan Dai and Siteng Huang and Min Zhang and Biao Gong and Donglin Wang},
  title = {{Focus-Consistent Multi-Level Aggregation for Compositional Zero-Shot Learning}},
  year = {2024},
  eprint = {2408.17083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.17083},
}
Figure