Symmetry and Group in Attribute-Object Compositions

Yonglu Li, Yue Xu, Xiaohan Mao, Cewu Lu

Introduction

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen combinations of attributes and objects, such as "purple cauliflower," by learning from seen pairs during training. A fundamental challenge in CZSL is contextuality: the visual appearance of an attribute is often heavily dependent on the object it modifies. For example, a "peeled apple" looks significantly different from a "peeled egg." Traditional approaches often treat attributes as static visual patterns or simple linear operators, which fails to capture the underlying physical transformations and the "rationality" of these changes.

Illustration of compositionality, contextuality, and the proposed symmetry principle.

The SymNet Framework

To address these issues, the authors propose SymNet, a framework grounded in group theory. SymNet models attributes as transformations rather than static features. The architecture consists of two primary components: a Coupling Network (CoN), which applies an attribute to an object representation, and a Decoupling Network (DecoN), which removes it.

The training of SymNet is guided by four fundamental group theory axioms and a novel Symmetry Principle:

  1. Symmetry: Applying an attribute to an object that already possesses it (e.g., "peeling" a peeled apple) should result in no change. Similarly, removing an attribute from an object that lacks it should leave the representation unchanged.
  2. Closure: Transformations must result in valid representations within the defined attribute-object space.
  3. Invertibility: The coupling and decoupling operations should be inverses; adding and then removing the same attribute should return the object to its initial state.
  4. Commutativity: The order of applying multiple attributes should not affect the final representation.
Overview of the SymNet framework, including group axioms and the Relative Moving Distance (RMD) inference method.

Inference via Relative Moving Distance (RMD)

A key innovation of SymNet is its inference mechanism. Instead of a traditional classifier, the model uses Relative Moving Distance (RMD). For a given image, the model calculates how much the representation "moves" in latent space when a candidate attribute is added versus when it is removed. If adding the attribute results in a smaller displacement than removing it, the model infers that the attribute is already present in the image. This approach treats attribute recognition as a measurement of dynamic change, making it more robust to the visual variability across different object categories.

Experimental Results

SymNet was evaluated on two benchmark datasets: MIT-States, which features a wide variety of noisy attribute-object pairs, and UT-Zappos50K, a fine-grained dataset of shoes.

SymNet achieved state-of-the-art (SOTA) performance:

  • MIT-States: Reached a Top-1 accuracy of 19.9%, improving upon the previous SOTA of 17.8%.
  • UT-Zappos: Achieved 52.1% accuracy, compared to the previous best of 49.9%.
  • Attribute Recognition: The RMD method significantly boosted standalone attribute identification, showing an 8.3% improvement on UT-Zappos.

Qualitative results further demonstrated the model's ability to perform complex attribute manipulations. For instance, the model could successfully retrieve images of "murky water" by taking a "heavy water" representation, decoupling "heavy," and coupling "murky."

Conclusion

By integrating group theory axioms and the symmetry principle, SymNet provides a mathematically rigorous foundation for modeling attribute-object compositions. Treating attributes as transformations allows the model to overcome the limitations of contextuality and achieve superior generalization on unseen combinations. Future research may address the "synonym problem"—where visually similar but linguistically distinct attributes (e.g., "ancient" vs. "old") penalize accuracy—and explore modeling continuous degrees of attributes.

Summary updated

Abstract

Attributes and objects can compose diverse compositions. To model the compositional nature of these general concepts, it is a good choice to learn them through transformations, such as coupling and decoupling. However, complex transformations need to satisfy specific principles to guarantee the rationality. In this paper, we first propose a previously ignored principle of attribute-object transformation: Symmetry. For example, coupling peeled-apple with attribute peeled should result in peeled-apple, and decoupling peeled from apple should still output apple. Incorporating the symmetry principle, a transformation framework inspired by group theory is built, i.e. SymNet. SymNet consists of two modules, Coupling Network and Decoupling Network. With the group axioms and symmetry property as objectives, we adopt Deep Neural Networks to implement SymNet and train it in an end-to-end paradigm. Moreover, we propose a Relative Moving Distance (RMD) based recognition method to utilize the attribute change instead of the attribute pattern itself to classify attributes. Our symmetry learning can be utilized for the Compositional Zero-Shot Learning task and outperforms the state-of-the-art on widely-used benchmarks. Code is available at https://github.com/DirtyHarryLYL/SymNet.
BibTeX
@inproceedings{li2020symmetry,
  author = {Yonglu Li and Yue Xu and Xiaohan Mao and Cewu Lu},
  title = {{Symmetry and Group in Attribute-Object Compositions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=bO58H2yFVvK},
}
Figure