Multi-Label Classification in Anime Illustrations Based on Hierarchical Attribute Relationships

Ziwen Lan, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel Graph Convolutional Network (GCN) model for multi-label classification of anime illustrations, specifically focusing on capturing the hierarchical relationships between visual attributes. By organizing attributes into a hierarchy and leveraging this structure within the GCN, the model achieves improved accuracy in identifying complex visual features in anime art.

What Problem Does This Paper Address?

Anime illustrations are rich in detail and often feature complex visual attributes that are not always independent. Traditional multi-label image classification methods often treat all labels equally, failing to capture the inherent hierarchical relationships between attributes (e.g., "hair color" is a sub-attribute of "hair," which is a sub-attribute of "character"). This limitation hinders accurate classification, especially in the domain of anime art where creators intentionally highlight specific, often hierarchically related, visual features. The paper addresses the challenge of effectively classifying these multi-label attributes in anime illustrations by explicitly modeling their hierarchical structure.

Methods

The proposed method consists of three core components:

  • Hierarchical Divisive Clustering (HDC): This step uses a hierarchical Generative Adversarial Network (GAN) tree structure to cluster image features. The GAN-tree recursively splits data into two clusters based on discriminative feature differences, aiming to avoid mode collapse and capture nuanced representations.
  • Hierarchical Label Assignment (HLA): After clustering, attributes are assigned hierarchical labels. This involves using a Maximum A Posteriori (MAP) approach to determine the most probable labels and then organizing these labels according to a predefined hierarchical structure. The paper also describes constructing this hierarchy using WordNet for semantic relationships.
  • GCN-Based Classification: The hierarchical feature representations generated by HDC and HLA are fed into a GCN. The GCN is designed to capture both co-occurrence relationships between attributes (using a static GCN) and their hierarchical dependencies (using a dynamic GCN). This allows the model to leverage the rich hierarchical information for final classification.

Datasets

The authors evaluated their method on four publicly available anime illustration datasets:

  • Safebooru [36]: A large dataset with over 1 million illustrations.
  • DAF:re [40]: A crowd-sourced, long-tailed dataset with a focus on character recognition.
  • FG-BG [41]: Used for character background segmentation.
  • iCartoonFace [42]: A benchmark dataset with a large number of characters and annotations.

The paper also details how they constructed hierarchical label structures for these datasets by filtering out less frequent or ambiguous labels and using WordNet to define semantic relationships.

Key Insights and Ideas

  • Hierarchy Matters: The core insight is that explicitly modeling the hierarchical relationships between visual attributes in anime illustrations significantly improves classification accuracy, outperforming methods that treat attributes independently.
  • GCNs for Hierarchical Data: Graph Convolutional Networks are well-suited for capturing complex relationships, and by adapting them to incorporate hierarchical label structures, they become powerful tools for this specific task.
  • GANs for Feature Representation: Using hierarchical GANs for clustering provides robust and discriminative feature representations, which are crucial for the subsequent hierarchical label assignment and GCN classification.
  • WordNet as a Knowledge Source: Leveraging WordNet to build attribute hierarchies demonstrates a practical way to incorporate external semantic knowledge into the classification pipeline.

Key Results

The proposed hierarchical GCN model consistently outperformed several baseline methods, including state-of-the-art GCN-based approaches, across all four tested datasets.

  • Quantitative Performance: The paper reports significant improvements in metrics like Overall Precision (OP), Recall (OR), F1 Score (OF1), and particularly in hierarchical metrics like Hierarchical Precision (HP), Recall (HR), and F1 Score (HF1), which directly benefit from the hierarchical approach. For instance, on the Safebooru dataset, the proposed method achieved an OF1 of 71.1, while the next best (P-GCN) achieved 70.2. The HF1 score was also notably higher.
  • Qualitative Results: Heatmaps visualizing prediction scores showed that the proposed method assigns higher probabilities to true labels and lower probabilities to false labels, especially in cases where hierarchical relationships were crucial (e.g., correctly classifying "black hair" by leveraging the score for "long hair").
  • Efficiency: The model demonstrated competitive computational time and space complexity compared to other GCN-based methods, suggesting a good balance between performance and resource usage.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Focus on Frequent Labels: The current study primarily focuses on the 500 most frequently appearing labels, and its performance on less frequent labels is not evaluated.
  • Manually Constructed Hierarchy: The hierarchical structure was constructed based on predetermined rules and WordNet. It's uncertain if this is the optimal hierarchy for all anime illustration datasets.
  • Data Proportion Impact: The impact of varying training data proportions on the final classification results was not investigated.

Future work will involve extending the evaluation to less frequent labels, exploring automated methods for constructing optimal hierarchical structures, and investigating the influence of different data splits.

Summary updated

Abstract

In this paper, we propose a hierarchical multi-modal multi-label attribute classification model for anime illustrations using a graph convolutional network (GCN). Our focus is on the challenging task of multi-label attribute classification, which requires capturing subtle features intentionally highlighted by creators of anime illustrations. To address the hierarchical nature of these attributes, we leverage hierarchical clustering and hierarchical label assignments to organize the attribute information into a hierarchical feature. The proposed GCN-based model effectively utilizes this hierarchical feature to achieve high accuracy in multi-label attribute classification. The contributions of the proposed method are as follows. Firstly, we introduce GCN to the multi-label attribute classification task of anime illustrations, enabling the capturing of more comprehensive relationships between attributes from their co-occurrence. Secondly, we capture subordinate relationships among the attributes by adopting hierarchical clustering and hierarchical label assignment. Lastly, we construct a hierarchical structure of attributes that appear more frequently in anime illustrations based on certain rules derived from previous studies, which helps to reflect the relationships between different attributes. The experimental results on multiple datasets show that the proposed method is effective and extensible by comparing it with some existing methods, including the state-of-the-art method.
BibTeX
@inproceedings{lan2023multi,
  author = {Ziwen Lan and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Multi-Label Classification in Anime Illustrations Based on Hierarchical Attribute Relationships}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=LBVQX6okZ4},
}
Figure