Multi-task CNN Model for Attribute Prediction

Abrar H. Abdulnabi, Gang Wang, Jiwen Lu, Kui Jia

Abstract

This paper proposes a joint multi-task learning algorithm to better predict attributes in images using deep convolutional neural networks (CNN). We consider learning binary semantic attributes through a multi-task CNN model, where each CNN will predict one binary attribute. The multi-task learning allows CNN models to simultaneously share visual knowledge among different attribute categories. Each CNN will generate attribute-specific feature representations, and then we apply multi-task learning on the features to predict their attributes. In our multi-task framework, we propose a method to decompose the overall model's parameters into a latent task matrix and combination matrix. Furthermore, under-sampled classifiers can leverage shared statistics from other classifiers to improve their performance. Natural grouping of attributes is applied such that attributes in the same group are encouraged to share more knowledge. Meanwhile, attributes in different groups will generally compete with each other, and consequently share less knowledge. We show the effectiveness of our method on two popular attribute datasets.

Introduction

Predicting image attributes—such as "striped," "furry," or "wearing a necktie"—presents a more complex challenge than standard object classification. While an object typically has one primary label, it can possess dozens of attributes that are rarely independent. These attributes often co-occur and share underlying visual patterns.

Traditional approaches typically train independent classifiers for each attribute, ignoring these inherent correlations. This is particularly problematic for "under-sampled" attributes—those that appear infrequently in a dataset. Without sufficient training data, standalone models for these attributes often perform poorly. This research addresses the challenge of designing a system that shares "visual common sense" across related attributes while maintaining their distinct characteristics.

Proposed Methodology: MTL-CNN

The core of the proposed framework is the Multi-task CNN (MTL-CNN) architecture. Unlike standard models that learn a single weight matrix for all attributes, this approach introduces a novel parameter decomposition.

Parameter Decomposition ($W = LS$)

The model decomposes the final classification weight matrix $W$ into two distinct components:

  1. Latent Task Matrix ($L$): A shared layer that captures fundamental visual patterns common across many attributes.
  2. Combination Matrix ($S$): A set of vectors that determines how each specific attribute "mixes and matches" the patterns found in $L$.
The MTL-CNN architecture decomposes weight parameters into a shared latent task matrix $L$ and a combination matrix $S$.

Natural Grouping and Regularization

To further refine the sharing of knowledge, attributes are organized into logical "natural groups." For instance, colors like red, blue, and black are grouped together, as are textures like striped and furry.

Example of natural grouping for the Clothing Attributes dataset, categorizing attributes into Colors, Patterns, Cloth-parts, and Appearance.

The model employs specific mathematical constraints ($L_{21}$ and $L_1$ norms) to regulate how information is shared. Attributes within the same group are encouraged to share more information from the $L$ matrix, while attributes in different groups are forced to "compete," ensuring that irrelevant features are not shared across unrelated categories.

Experimental Setup and Datasets

The model was evaluated on two major benchmarks:

  • Clothing Attributes Dataset: 1,856 images with 23 binary attributes (e.g., necktie, scarf, gender).
  • Animals with Attributes (AwA): A large-scale dataset with 30,475 images across 50 animal classes, featuring 85 binary attributes.

For smaller datasets, the entire CNN can be trained end-to-end. For larger datasets like AwA, the authors demonstrate that freezing the lower layers of a pre-trained CNN and only training the MTL "heads" provides significant computational savings while still outperforming single-task models.

Key Results and Insights

The MTL-CNN consistently outperformed both independent single-task CNNs and previous state-of-the-art methods across both datasets:

  • Clothing Dataset: The proposed model reached 92.82% accuracy, a substantial improvement over the previous state-of-the-art (CRF) at 83.95%.
  • AwA Dataset: The model achieved a Mean Average Precision (mAP) of 81.19%, compared to 75.37% for a standard single-task CNN.

A key insight is the "boost" effect: multi-task learning acts as a form of data augmentation. An under-sampled attribute like "necktie" can improve its feature detection by leveraging shared statistics from more common, visually similar attributes like "striped." Qualitatively, the model successfully identified attributes in cluttered environments—such as stripes on a zebra—by utilizing shared pattern knowledge that single-task models lacked.

Conclusion

The MTL-CNN framework demonstrates that joint learning of attributes significantly enhances prediction accuracy by sharing visual knowledge. While effective, the approach currently relies on human-defined "natural groups." Future research may focus on automating the discovery of these groups and further interpreting the "visual atoms" captured within the latent matrix $L$ to provide more human-readable descriptions of shared features.

Summary updated

Abstract

This paper proposes a joint multi-task learning algorithm to better predict attributes in images using deep convolutional neural networks (CNN). We consider learning binary semantic attributes through a multi-task CNN model, where each CNN will predict one binary attribute. The multi-task learning allows CNN models to simultaneously share visual knowledge among different attribute categories. Each CNN will generate attribute-specific feature representations, and then we apply multi-task learning on the features to predict their attributes. In our multi-task framework, we propose a method to decompose the overall model's parameters into a latent task matrix and combination matrix. Furthermore, under-sampled classifiers can leverage shared statistics from other classifiers to improve their performance. Natural grouping of attributes is applied such that attributes in the same group are encouraged to share more knowledge. Meanwhile, attributes in different groups will generally compete with each other, and consequently share less knowledge. We show the effectiveness of our method on two popular attribute datasets.
BibTeX
@article{abdulnabi2015multi,
  author = {Abrar H. Abdulnabi and Gang Wang and Jiwen Lu and Kui Jia},
  title = {{Multi-task CNN Model for Attribute Prediction}},
  journal = {IEEE Trans. Multimedia},
  year = {2015},
  doi = {10.1109/tmm.2015.2477680},
}
Figure