Multi-label Object Attribute Classification using a Convolutional Neural Network

Soubarna Banik, Mikko Lauri, Simone FrintroparXiv 2018 · arXiv

TL;DR

The paper introduces the Deep Attribute Network (DAN), a convolutional neural network designed to classify low-level object attributes like color, shape, pattern, and texture. By modifying and fine-tuning an object recognition network for multi-label classification, DAN successfully learns generic attributes that can describe unfamiliar objects, outperforming traditional handcrafted feature methods and off-the-shelf deep features.

What Problem Does This Paper Address?

Standard computer vision models excel at naming objects (e.g., "dog," "car"), but they struggle when encountering something completely new. If a robot encounters an unfamiliar object, simply failing to classify it isn't helpful. However, if the robot can recognize that the object is round, smooth, and metallic, it can infer how to interact with or grasp it.

While recent deep learning research has explored high-level attributes (like "wears a hat" or "has a side-mirror"), these are highly specific to certain object classes. This paper focuses on low-level visual attributes (color, shape, pattern, texture) which are universally applicable across different categories. The challenge lies in the fact that standard object recognition networks naturally discard these low-level details in their upper layers to achieve class invariance (e.g., recognizing both white and yellow tigers as "tigers"). Furthermore, low-level attributes are largely independent of one another, meaning models that explicitly force attribute correlations risk overfitting the data.

Methods

The authors propose the Deep Attribute Network (DAN), which adapts the popular VGG-16 architecture (and later validates the approach using ResNet-50) for multi-label attribute classification.

Key technical modifications include:

  • Architecture Shift: The final softmax layer of VGG-16 (designed for mutually exclusive classes) is replaced with a sigmoid layer, allowing the network to predict multiple co-occurring attributes simultaneously (e.g., an object can be blue, rectangular, and metallic all at once).
  • Two-Phase Fine-Tuning: To prevent the network from discarding low-level visual features, the authors freeze the early layers and first train only the newly added fully-connected layers (fcA and fcB). In the second phase, they unfreeze and fine-tune the network down to the conv5_1 layer, forcing the upper layers to retain attribute-specific details.
  • Dynamic Weighted Cross-Entropy Loss: To combat severe class imbalance within the dataset (e.g., far more "black" objects than "pink" ones), the authors implement a loss function that dynamically weights classes based on their frequency within each training batch.

Datasets

The model was trained and evaluated on two primary datasets:

  1. ImageNet Attribute Dataset: Contains 9,600 images annotated with 25 low-level attributes spanning color, shape, pattern, and texture.
  2. a-Pascal Dataset: A subset of the PASCAL VOC 2008 dataset containing 20 object classes. The authors selected 3 shape and 5 material attributes from its 64 annotations to test DAN's generalization capabilities.

Key Insights and Ideas

  • The Invariance Dilemma: Standard classification networks are trained to ignore attribute variations to group different-looking objects into the same class. DAN flips this mechanism by fine-tuning the later convolutional layers, redirecting the network's focus back to the foundational visual building blocks.
  • Implicit Correlation over Explicit Modeling: Unlike previous approaches that use complex graphs or Recurrent Neural Networks to force relationships between attributes (e.g., linking "mustache" to "male"), DAN learns attributes jointly through shared hidden layers. This allows the model to share low-level features like edges and corners without forcing artificial, non-universal label correlations that cause overfitting.

Key Results

  • State-of-the-Art Performance: On the ImageNet Attribute test set, DAN achieved a micro mean Average Precision (mAP) of 0.74 and an ROC-AUC score of 0.90.
  • Outperforming Baselines: DAN outperformed traditional handcrafted feature methods (Russakovsky et al.) and an off-the-shelf feature baseline (VGG-16 + SVM) by 3–18% and 8–21% in ROC-AUC scores, respectively. The "pattern" attribute group (e.g., spotted, striped) saw the most dramatic improvement.
  • Cropped vs. Un-cropped Data: When evaluated on images cropped tightly around the target object, DAN's performance improved by 2–7% in micro mode, demonstrating that background clutter slightly penalizes attribute verification.
  • Attribute Localization: Using an algorithm called Excitation Backprop, the authors generated preliminary attention maps showing that DAN can successfully localize specific attributes (like a "red" bowl) even when the object is partially occluded.

Limitations and Future Work

A notable limitation is the model's performance on highly ambiguous or rare classes. For example, performance dropped significantly on pattern classes like striped and spotted due to massive visual variance and high rates of ambiguous annotations in the dataset. Similarly, rare colors like pink (78 training samples) and violet (34 samples) suffered due to data scarcity.

For future work, the authors plan to extend DAN's architecture explicitly into attribute-level object localization, creating systems capable of searching for and finding objects in cluttered environments based purely on descriptive text queries.

Summary updated

Abstract

Objects of different classes can be described using a limited number of attributes such as color, shape, pattern, and texture. Learning to detect object attributes instead of only detecting objects can be helpful in dealing with a priori unknown objects. With this inspiration, a deep convolutional neural network for low-level object attribute classification, called the Deep Attribute Network (DAN), is proposed. Since object features are implicitly learned by object recognition networks, one such existing network is modified and fine-tuned for developing DAN. The performance of DAN is evaluated on the ImageNet Attribute and a-Pascal datasets. Experiments show that in comparison with state-of-the-art methods, the proposed model achieves better results.
BibTeX
@misc{banik2018multi,
  author = {Soubarna Banik and Mikko Lauri and Simone Frintrop},
  title = {{Multi-label Object Attribute Classification using a Convolutional Neural Network}},
  year = {2018},
  eprint = {1811.04309},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.04309},
}

Loading related content...

Figure