L-WISE: Boosting Human Image Category Learning Through Model-Based Image Selection And Enhancement

Morgan B. Talbot, Gabriel Kreiman, James J. DiCarlo, Guy Gaziv

TL;DR

This paper introduces L-WISE, a novel method that leverages robust artificial neural networks (ANNs) to enhance human visual learning. By intelligently selecting training images based on their predicted difficulty and applying subtle image enhancements, L-WISE significantly boosts categorization accuracy and reduces learning time for humans across various domains, including challenging clinical tasks.

What Problem Does This Paper Address?

Learning to recognize new visual categories, especially fine-grained ones, can be a slow and challenging process for humans. This is particularly true in specialized fields like medical diagnosis (e.g., identifying skin lesions or histological anomalies), where extensive training is required. While ANNs have become powerful tools for image analysis and have shown remarkable alignment with human visual processing, their potential to actively improve human learning has not been fully explored. The authors aim to bridge this gap by using ANNs not just as models of vision, but as active tutors to accelerate and improve human visual categorization skills.

Methods

The core of L-WISE lies in two key capabilities derived from robustified ANNs:

  1. Image Difficulty Prediction: The authors found that the "ground truth logit" (the pre-softmax output of an ANN for the correct class) from a robustified ANN serves as a highly accurate predictor of how difficult an image is for humans to categorize. This allows for the creation of a human-aligned difficulty score for any given image.
  2. Image Enhancement: Using the same robustified ANNs, the authors can generate subtle image perturbations that maximize the ground truth logit. This process effectively enhances category-specific features in an image, making it easier for humans to recognize the correct category.

L-WISE combines these capabilities into a learning curriculum:

  • Image Selection: During training, images are selected based on their predicted difficulty, starting with easier examples and gradually progressing to more challenging ones.
  • Image Enhancement: Training images are enhanced to aid recognition, with the magnitude of enhancement decreasing over time as the learner progresses.

This approach is implemented using adversarially trained ResNet-50 models, fine-tuned on specific task datasets.

Datasets

The study utilized several datasets:

  • ImageNet: Used for training robustified ResNet-50 models and for a 16-way basic animal classification task to validate the difficulty prediction and enhancement methods.
  • iNaturalist: Used for fine-tuning models for natural image tasks.
  • HAM10000: A dataset of dermoscopy images used for skin lesion classification.
  • MHIST: A dataset of histology images used for benign vs. pre-cancerous colon tissue classification.

No new benchmarks were introduced; the focus was on applying existing datasets to the novel learning augmentation task.

Key Insights and Ideas

  • ANNs as Tutors: The most significant insight is that ANNs, particularly robustified ones, can be more than just models of human vision; they can actively teach humans by predicting difficulty and enhancing perception.
  • Ground Truth Logit as a Universal Difficulty Metric: The authors demonstrate that the ground truth logit from a robustified ANN is a superior predictor of human image difficulty compared to existing methods. This opens doors for more accurate curriculum design.
  • Enhancement for Humans, Not Just ANNs: While adversarial attacks aim to fool ANNs, this work shows how similar perturbation techniques can be repurposed to help humans by making target features more salient.
  • Synergy of Selection and Enhancement: Combining image selection based on difficulty with targeted image enhancement yields greater benefits than either strategy alone, suggesting a powerful synergy for accelerating learning.
  • Clinical Relevance: The successful application to challenging medical imaging tasks highlights the practical potential of L-WISE beyond general image recognition.

Key Results

  • Accuracy Gains: L-WISE achieved substantial improvements in test-time categorization accuracy, ranging from 33% to 72% relative to control groups on unmodified test images.
  • Reduced Training Time: The augmented learning group completed training 20-23% faster while performing the same number of trials.
  • Difficulty Prediction Accuracy: The ground truth logit from robustified ANNs achieved an AUC of 0.72 for predicting human error rates on a 16-way animal classification task, outperforming prior state-of-the-art methods.
  • Effective Enhancement: Model-based image enhancements significantly increased human accuracy on basic animal categories, outperforming conventional image enhancement techniques.
  • Broad Applicability: The method demonstrated efficacy across natural images (moth species), dermoscopy images (skin lesions), and histology images (colon tissue).

Limitations and Future Work

  • Fixed Schedules: L-WISE currently uses fixed, time-dependent schedules for image selection and enhancement. The authors suggest that adapting these schedules in real-time based on individual learner progress could lead to even more effective augmentation.
  • Homogenizing Effect of Enhancement: High-magnitude enhancements can sometimes make images within a class appear too similar, potentially obscuring subtle but important variations (e.g., in medical images). This could also lead to biased representations if the underlying ANN is biased.
  • "Hallucinations": While enhancements can improve accuracy, they can sometimes introduce features not present in the original image ("hallucinations"). While these might be beneficial for learning, they could also be misleading, especially in critical applications like medical diagnosis. Further investigation is needed to understand and control these effects.
  • Hyperparameter Optimization: The authors did not exhaustively search for optimal hyperparameters for curriculum design or image enhancement.
  • Generalizability to Other Tasks: While effective for categorization, the applicability of L-WISE to other visual learning tasks (e.g., object detection, segmentation) remains to be explored.

Future work should focus on developing adaptive learning strategies, thoroughly investigating the implications of "hallucinations" and potential biases, and exploring the method's utility in a wider range of visual tasks and domains.

Summary updated

Abstract

The currently leading artificial neural network models of the visual ventral stream - which are derived from a combination of performance optimization and robustification methods - have demonstrated a remarkable degree of behavioral alignment with humans on visual categorization tasks. We show that image perturbations generated by these models can enhance the ability of humans to accurately report the ground truth class. Furthermore, we find that the same models can also be used out-of-the-box to predict the proportion of correct human responses to individual images, providing a simple, human-aligned estimator of the relative difficulty of each image. Motivated by these observations, we propose to augment visual learning in humans in a way that improves human categorization accuracy at test time. Our learning augmentation approach consists of (i) selecting images based on their model-estimated recognition difficulty, and (ii) applying image perturbations that aid recognition for novice learners. We find that combining these model-based strategies leads to categorization accuracy gains of 33-72% relative to control subjects without these interventions, on unmodified, randomly selected held-out test images. Beyond the accuracy gain, the training time for the augmented learning group was also shortened by 20-23%, despite both groups completing the same number of training trials. We demonstrate the efficacy of our approach in a fine-grained categorization task with natural images, as well as two tasks in clinically relevant image domains - histology and dermoscopy - where visual learning is notoriously challenging. To the best of our knowledge, our work is the first application of artificial neural networks to increase visual learning performance in humans by enhancing category-specific image features.
BibTeX
@inproceedings{talbot2025l,
  author = {Morgan B. Talbot and Gabriel Kreiman and James J. DiCarlo and Guy Gaziv},
  title = {{L-WISE: Boosting Human Image Category Learning Through Model-Based Image Selection And Enhancement}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=AoIKgHu9Si},
}
Figure