Fine-tuning Deep Neural Networks by Interactively Refining the 2D Latent Space of Ambiguous Images

Jiafu Wei, Haoran Xie, Chia-Ming Chang, Xi YangIJCAI 2022 · DOI

TL;DR

This paper introduces a novel human-in-the-loop approach to improve the classification accuracy of deep neural networks (DNNs) on ambiguous images. By projecting the DNN's high-dimensional latent space into an interactive 2D workspace, users can manually adjust the positions of misclassified data points, and these adjustments are then used to fine-tune the network, leading to improved performance.

What Problem Does This Paper Address?

Deep neural networks (DNNs) excel at image classification, but they struggle with ambiguous images – those that are visually similar across different classes. Humans, on the other hand, are often better at distinguishing these subtle differences. This paper addresses the challenge of bridging this gap by leveraging human intuition to enhance DNN performance on such difficult cases. The core problem is that current DNNs lack the nuanced understanding that humans possess for certain types of visual ambiguity, leading to classification errors.

Methods

The authors propose a human-in-the-loop system that augments a pre-trained DNN. The key innovation lies in:

  • 2D Latent Space Projection: The high-dimensional latent space learned by the DNN is projected onto a 2D workspace using Principal Component Analysis (PCA). This projection allows for visual representation and interactive manipulation.
  • Interactive User Interface: A custom UI is developed, featuring a function bar, a modification window, and a result window. Users can select data points, drag them to new positions in the 2D space, and use tools like lasso or selection boxes for multiple points.
  • Fine-tuning Loss Function: A novel loss function is introduced that combines the standard classification loss (cross-entropy) with a "distance difference loss." This latter loss penalizes deviations between the original and adjusted coordinates of the manipulated data points in the 2D space. This ensures that the network's fine-tuning is guided by the user's adjustments.
  • Feedback Loop: The user's modifications in the 2D workspace are fed back to the network for retraining. This iterative process allows the network to learn from the user's corrections and improve its classification of ambiguous instances.

Datasets

The primary dataset used for evaluation is the Quick Draw dataset. This is a large, crowd-sourced dataset of over a billion hand-drawn sketches. The authors specifically selected nine categories of images from this dataset that exhibit a degree of similarity, making them challenging for DNNs to classify accurately.

Key Insights and Ideas

  • Human intuition as a powerful supervisory signal: The paper demonstrates that human judgment, even on a small scale, can effectively guide DNNs to overcome their limitations with ambiguous data.
  • Visualizing and manipulating latent space is intuitive: Projecting the complex latent space into a 2D interactive environment makes the network's internal representations more understandable and actionable for users.
  • Targeted fine-tuning for specific weaknesses: Instead of retraining the entire network from scratch, this method allows for precise adjustments to address specific areas where the network struggles, particularly with ambiguous examples.
  • Potential for smaller networks to achieve comparable performance: The authors suggest that by leveraging human input, even smaller networks might achieve performance levels comparable to larger, more complex models on challenging tasks.

Key Results

The paper reports promising results from user studies and experimental evaluations:

  • Accuracy Improvement: In experiments with ambiguous data from the Quick Draw dataset, the proposed method achieved accuracy rates of 86.48% and 86.36%, representing an improvement of 0.15% and 0.03% over the baseline accuracy of 86.33%. While these numbers might seem small, they are significant in the context of fine-tuning and addressing difficult cases.
  • Qualitative Improvements: Visualizations (Figure 4) show that after user intervention, the inter-class distances in the latent space become clearer, and misclassified points (e.g., a "clock" misclassified as "apple") are corrected.
  • User Feedback: Four users found the interface convenient and intuitive. They agreed that the 2D point coordinates effectively conveyed the network's classification, and the method of influencing classification by moving coordinates was novel and promising.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Scale: The current validation has been performed on small datasets. Further evaluation on larger, more diverse datasets is needed.
  • Projection and Distance Matching: The specific methods for projecting the latent space and matching distances between the 2D and original latent spaces require further research.

Future work will focus on addressing these limitations and further exploring the synergy between neural networks and human-in-the-loop approaches to improve classification accuracy and eliminate machine-induced deviations.

Summary updated

BibTeX
@inproceedings{wei2022fine,
  author = {Jiafu Wei and Haoran Xie and Chia-Ming Chang and Xi Yang},
  title = {{Fine-tuning Deep Neural Networks by Interactively Refining the 2D Latent Space of Ambiguous Images}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/861},
}
Figure