Interval Type-2 Fuzzy Neural Networks Formulti-Label Classification

Dayong Tian, Li Fei-Fei, Yiwen WeiSSRN 2023 · DOI

TL;DR

This paper introduces a novel multi-label classification (MLC) approach using interval type-2 fuzzy logic to better capture the inherent ambiguity and human-like fuzzy judgments in data. By employing a deep neural network to predict fuzzy memberships and their uncertainty, the model achieves state-of-the-art performance on several MLC benchmarks, outperforming existing methods.

What Problem Does This Paper Address?

Traditional multi-label classification (MLC) models typically assign binary labels (0 or 1) to instances, meaning an object either belongs to a category or it doesn't. However, human perception and real-world data often exhibit more nuanced and fuzzy characteristics. For instance, when classifying an image of a flower, a human might perceive it as primarily a "flower" but also acknowledge the presence of a "woman" with varying degrees of certainty. This inherent fuzziness and the relationships between labels are not well-represented by simple binary assignments. Existing MLC methods struggle to capture this ambiguity, leading to suboptimal performance.

Methods

The proposed method leverages interval type-2 fuzzy logic to model this inherent uncertainty. The core idea is to predict not just a single membership value for each label, but an interval representing a range of possible memberships. This is achieved through a two-part neural network architecture:

  1. Fuzziness Initializer: A deep neural network that predicts "type-1 fuzzy labels." This network uses a custom activation function designed to avoid gradient vanishing and ensure outputs are within the [0, 1] range, similar to Batch Normalization but adapted for fuzzy outputs.
  2. Fuzzifier Estimator: Another neural network that predicts "fuzzifiers" for the type-1 fuzzy labels. These fuzzifiers determine the "thickness" or uncertainty of the interval for each label's membership. Interestingly, this network is trained by first learning to predict the number of true labels for an instance, which is then used to estimate the fuzziness. An autoencoder is used to learn feature representations for this task.

A novel loss function is introduced to measure the dissimilarity between the generated interval type-2 fuzzy labels and the ground truth binary labels. This loss function is a continuous extension of the F1 measure, designed to effectively utilize the fuzzy information. Finally, a defuzzification strategy is employed to convert the interval type-2 fuzzy predictions back into binary labels for evaluation. This defuzzification uses a linear combination of the interval's midpoint and size, controlled by a hyperparameter $\lambda$.

Datasets

The authors evaluated their method on three widely-used multi-label datasets:

  • scene: A dataset for scene classification.
  • mirflickr: A dataset for image annotation.
  • nus-wide: A large-scale dataset for multi-label image classification.

For the nus-wide dataset, they used 128D-cVALD features. For scene and mirflickr, they used pre-extracted features.

Key Insights and Ideas

  • Fuzzy Judgments are Crucial: The paper strongly argues that human judgments are inherently fuzzy, and this fuzziness is critical for accurate multi-label classification, especially when dealing with complex relationships between labels.
  • Interval Type-2 Fuzzy Logic as a Solution: Interval type-2 fuzzy logic provides a powerful framework to represent this uncertainty, going beyond the limitations of traditional binary or even type-1 fuzzy logic.
  • Predicting Fuzziness from Label Count: A clever insight is using the number of true labels for an instance as a proxy for its fuzziness, and training a network to predict this count to estimate the fuzzifiers.
  • Novel Loss Function: The proposed loss function is key to training the model effectively by bridging the gap between fuzzy predictions and binary ground truth.
  • Defuzzification Matters: The choice of defuzzification strategy significantly impacts performance, and a simple midpoint is insufficient; incorporating interval size is beneficial.

Key Results

The proposed interval type-2 fuzzy neural network model achieved state-of-the-art performance across all evaluated metrics (Example-F1, Micro-F1, Macro-F1, and Hamming Accuracy) on the scene, mirflickr, and nus-wide datasets.

  • Example-F1: Improved over the best compared methods by 1.6%, 4.3%, and 3.2% on scene, mirflickr, and nus-wide, respectively.
  • Micro-F1: Improved over the best compared methods by 2.4%, 5.4%, and 5.3% on scene, mirflickr, and nus-wide, respectively.
  • Macro-F1: Improved over the best compared methods by 0.9%, 5.5%, and 9.5% on scene, mirflickr, and nus-wide, respectively.
  • Hamming Accuracy: Improved over the best compared methods by 0.5%, 1.1%, and 0.6% on scene, mirflickr, and nus-wide, respectively.

An ablation study demonstrated that using interval type-2 fuzzy logic substantially improved performance compared to using type-1 fuzzy logic.

Limitations and Future Work

  • Hyperparameter Sensitivity: The model is noted to be sensitive to the $\lambda$ hyperparameter, which controls the influence of interval size in defuzzification. The paper provides experimental results showing optimal ranges for $\lambda$.
  • Future Work: The authors suggest several promising directions:
  • Modeling hierarchical relationships among labels using interval type-2 fuzzy logic.
  • Extending the proposed method to handle multi-modal multi-label classification tasks.

Summary updated

BibTeX
@misc{tian2023interval,
  author = {Dayong Tian and Li Fei-Fei and Yiwen Wei},
  title = {{Interval Type-2 Fuzzy Neural Networks Formulti-Label Classification}},
  year = {2023},
  doi = {10.2139/ssrn.4384431},
  howpublished = {SSRN},
}
Figure