Confusion Graph: Detecting Confusion Communities in Large Scale Image Classification

Ruochun Jin, Yong Dou, Yueqing Wang, Xin NiuIJCAI 2017 · OpenReview

Abstract

For deep CNN-based image classification models, we observe that confusions between classes with high visual similarity are much stronger than those where classes are visually dissimilar. With these unbalanced confusions, classes can be organized in communities, which is similar to cliques of people in the social network. Based on this, we propose a graph-based tool named "confusion graph" to quantify these confusions and further reveal the community structure inside the database. With this community structure, we can diagnose the model's weaknesses and improve the classification accuracy using specialized expert sub-nets, which is comparable to other state-of-the-art techniques. Utilizing this community information, we can also employ pre-trained models to automatically identify mislabeled images in the large scale database. With our method, researchers just need to manually check approximate 3% of the ILSVRC2012 classification database to locate almost all mislabeled samples.

Introduction

Deep Convolutional Neural Networks (CNNs) have revolutionized image classification, but they remain "black boxes" that are difficult to debug. When a model fails, it’s often because it confuses classes with high visual similarity (e.g., a "hen" vs. a "cock"). Traditional diagnostic tools focus on individual misclassified images, which fails to capture these broader, class-level patterns of error.

Additionally, large-scale datasets like ImageNet are prone to "label noise"—images that are incorrectly labeled due to the limitations of crowdsourcing. Identifying these errors manually is an impossible task for humans given the scale. This paper seeks a systematic way to understand model weaknesses and clean up dataset labels simultaneously.

Methods

The authors propose a two-step framework to map and utilize class-level confusions.

Overview of the Confusion Graph pipeline, showing how model predictions are used to build a graph, detect communities, and then improve models or identify mislabeling.
  1. Building the Confusion Graph: They feed a dataset through a pre-trained model and record the top-$\tau$ predictions for each image. They then construct an undirected graph where nodes represent classes and edge weights represent the frequency and confidence with which the model confuses those two classes.
  2. Community Detection: Using the Louvain method (a standard algorithm for finding clusters in social networks), they partition the graph into "communities." Classes within a community are frequently confused with each other but rarely with classes outside the group.
Confusion graph for CIFAR-10 (a) and the resulting communities detected (b), where colors represent distinct confusion groups.

The authors then apply this structure in two ways:

  • Expert Sub-nets (ES): For a specific confusion community (e.g., different types of terriers), they train a small, specialized neural network to distinguish only between those specific classes, which is then "plugged in" to the main model to refine its predictions.
  • Mislabel Detection: They use the community info to flag suspicious images. If a model’s top predictions for an image consistently fall into a different community than its provided label, the image is flagged as likely mislabeled.

Datasets

The researchers evaluated their approach on several standard benchmarks:

  • CIFAR-10 & CIFAR-100: Used for initial proof-of-concept and to analyze how communities form based on shape, color, and semantic co-occurrence.
  • Oxford 102 Flowers: Used to test mislabel detection by intentionally "polluting" the dataset with 15% random incorrect labels.
  • ILSVRC2012 (ImageNet): The primary target for large-scale validation, both for improving state-of-the-art models (AlexNet and VGG-16) and for finding real-world labeling errors.

Key Insights and Ideas

  • Confusions are Structured: Errors aren't random. Classes cluster into communities based on similar shapes (snakes and worms), similar backgrounds (aquatic animals), similar textures (different types of trees), or co-occurrence (beds and wardrobes appearing in bedrooms).
  • Modularity as a Metric: The authors found that communities with high "modularity" (meaning they are very tightly knit and distinct from others) are the best candidates for improvement using expert sub-nets.
  • Human-Level Error in Data: Many errors in ImageNet occur in fine-grained categories (like specific bird species) where crowdsourced workers lack the expertise to distinguish between similar classes. The Confusion Graph effectively highlights these "expert-required" zones.

Key Results

  • Model Improvement: By adding expert sub-nets to handle specific confusion communities, the authors reduced the top-1 error rate of AlexNet by 1.49% and VGG-16 by 3.45% on the targeted classes.
  • Efficient Data Cleaning: On the Oxford Flowers dataset, the method detected 89% of all wrong labels with 72% precision.
  • ImageNet Audit: When applied to a 50,000-image subset of ImageNet, the tool identified 383 true mislabeled samples. Most importantly, the authors showed that researchers only need to manually check about 3% of the database to locate almost all mislabeled samples, drastically reducing the labor required for dataset curation.

Limitations and Future Work

The paper notes that the quality of the confusion graph depends on the quality of the initial pre-trained model; a very weak model might produce a noisy graph. Furthermore, while the expert sub-nets improve accuracy, the current implementation requires a two-stage prediction process (main model then expert sub-net), which adds slight computational overhead.

Future work could involve automating the selection of communities for sub-net training or applying these graphs to other domains like video classification or natural language processing, where semantic "communities" of confusion likely exist as well.

Summary updated

Abstract

For deep CNN-based image classification models, we observe that confusions between classes with high visual similarity are much stronger than those where classes are visually dissimilar. With these unbalanced confusions, classes can be organized in communities, which is similar to cliques of people in the social network. Based on this, we propose a graph-based tool named "confusion graph" to quantify these confusions and further reveal the community structure inside the database. With this community structure, we can diagnose the model's weaknesses and improve the classification accuracy using specialized expert sub-nets, which is comparable to other state-of-the-art techniques. Utilizing this community information, we can also employ pre-trained models to automatically identify mislabeled images in the large scale database. With our method, researchers just need to manually check approximate 3% of the ILSVRC2012 classification database to locate almost all mislabeled samples.
BibTeX
@inproceedings{jin2017confusion,
  author = {Ruochun Jin and Yong Dou and Yueqing Wang and Xin Niu},
  title = {{Confusion Graph: Detecting Confusion Communities in Large Scale Image Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=ryZcbXGuZr},
}
Figure