Large-Scale Attribute-Object Compositions

Filip Radenovic, Animesh Sinha, Albert Gordo, Tamara Berg, Dhruv MahajanarXiv 2021 · OpenReview

Abstract

We study the problem of learning how to predict attribute-object compositions from images, and its generalization to unseen compositions missing from the training data. To the best of our knowledge, this is a first large-scale study of this problem, involving hundreds of thousands of compositions. We train our framework with images from Instagram using hashtags as noisy weak supervision. We make careful design choices for data collection and modeling, in order to handle noisy annotations and unseen compositions. Finally, extensive evaluations show that learning to compose classifiers outperforms late fusion of individual attribute and object predictions, especially in the case of unseen attribute-object pairs.

Introduction

In standard image classification, models are typically trained to identify discrete object categories, such as a "car" or a "bicycle." However, real-world visual understanding requires a more descriptive approach that accounts for attributes—identifying a red car, a vintage car, or a damaged car.

Examples of attribute-object compositions across different categories.

The primary challenge in this domain is the quadratic explosion of labels. With 1,000 objects and 1,000 attributes, there are 1,000,000 possible combinations, making manual labeling at scale impossible. Furthermore, many combinations are "unseen" during training. An AI might encounter "red dresses" and "blue cars" but must generalize to identify a "red car" without direct supervision. Existing systems often struggle, either treating every pair as a unique category (which fails to scale) or using "late fusion" (multiplying independent probabilities), which ignores how attributes fundamentally alter an object's visual representation.

Methodology: The CompNet Framework

To address these challenges, we propose CompNet, an end-to-end architecture designed to learn compositional representations. CompNet utilizes three distinct output heads:

  1. Object Head: Predicts the noun (e.g., "bicycle").
  2. Attribute Head: Predicts the adjective (e.g., "broken").
  3. Composition Head: This head uses a Multi-Layer Perceptron (MLP) to take the weights of the object and attribute classifiers as input and "composes" them into a new classifier for a specific pair.

By reasoning in the "classifier space," the model learns how a concept like "broken" modifies the visual features of a "bicycle." To handle the massive scale of potential pairs, we employ Hard Negative Mining, selecting challenging incorrect compositions during training to refine accuracy. Additionally, we introduced a Conditional Loss function to handle noisy Instagram hashtag data, preventing the model from being penalized for predicting correct but untagged attributes.

Datasets and Large-Scale Weak Supervision

Our study introduces two massive weakly-supervised datasets derived from Instagram:

  • IG-8k-1k: The largest dataset of its kind, containing 78 million images, 7,694 objects, 1,237 attributes, and over 280,000 unique compositions.
  • IG-504-144: A smaller subset for detailed ablation studies.
  • Marketplace Dataset: A human-annotated dataset used to verify that findings from noisy data generalize to clean, real-world e-commerce scenarios.

A critical part of our pipeline involved "hashtag engineering" to filter out non-visual or non-interpretable tags (e.g., #inspired), ensuring the model focuses on visual, shared attributes.

Key Results and Insights

Our evaluations demonstrate that explicitly learning to combine concepts is significantly more effective than late fusion.

  • Superior Generalization: CompNet significantly outperformed the "Softmax Product" baseline on unseen compositions. On the IG-504-144 dataset, it achieved 11.2 mAP on unseen pairs, compared to the baseline's 7.2 mAP.
  • Scaling Benefits: As training data increased from 2.5M to 78M images, the performance gap between "seen" and "unseen" compositions narrowed, suggesting that large-scale weak supervision is a viable path toward zero-shot visual understanding.
  • Handling Unseen Data: We found it crucial not to treat unseen compositions as "false" during training. Instead, excluding them from the loss calculation prevents the model from learning incorrect negative associations for valid but unobserved pairs.

Conclusion and Future Work

While CompNet shows strong performance, challenges remain. The current model assumes that any combination of object and attribute hashtags in an image constitutes a valid pair, which introduces label noise. Future work will explore weak localization to ensure attributes are correctly associated with specific objects rather than the background. Additionally, we aim to address attribute bias (e.g., the strong association between "striped" and "zebra") to improve the model's general reasoning capabilities across diverse contexts.

Summary updated

Abstract

We study the problem of learning how to predict attribute-object compositions from images, and its generalization to unseen compositions missing from the training data. To the best of our knowledge, this is a first large-scale study of this problem, involving hundreds of thousands of compositions. We train our framework with images from Instagram using hashtags as noisy weak supervision. We make careful design choices for data collection and modeling, in order to handle noisy annotations and unseen compositions. Finally, extensive evaluations show that learning to compose classifiers outperforms late fusion of individual attribute and object predictions, especially in the case of unseen attribute-object pairs.
BibTeX
@misc{radenovic2021large,
  author = {Filip Radenovic and Animesh Sinha and Albert Gordo and Tamara Berg and Dhruv Mahajan},
  title = {{Large-Scale Attribute-Object Compositions}},
  year = {2021},
  eprint = {2105.11373},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.11373},
}
Figure