Improving neural network representations using human similarity judgments

Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A. Vandermeulen, Katherine L. Hermann, Andrew K. Lampinen, Simon Kornblith

Improving neural network representations using human similarity judgments

Introduction: The Gap Between AI and Human Perception

Deep neural networks (DNNs) excel at fine-grained object recognition, yet they often lack a "common sense" understanding of how different categories relate. While a standard model can easily distinguish between two different breeds of cats (local structure), it may not naturally group "cats," "dogs," and "raccoons" together under the broader concept of "animals" (global structure). This discrepancy arises because standard training objectives, like ImageNet classification or contrastive learning, focus primarily on ensuring that similar images are close in the representation space without explicitly constraining the broader hierarchy.

Previous attempts to align AI representations with human similarity judgments often resulted in "naive" transformations that improved global structure at the expense of local accuracy. The challenge lies in injecting human-like organization without degrading the high-quality features learned during massive pretraining.

The gLocal transform reshapes the representation space to align with human-like global hierarchies (e.g., animate vs. inanimate) while preserving the original model's fine-grained local clusters.

The gLocal Transform

To address this, researchers developed the gLocal transform, a linear transformation designed to achieve a "best-of-both-worlds" representation. The transform optimizes an objective function with three key components:

  1. Global Alignment Loss: Uses human triplet data (e.g., "A is more similar to B than to C") from the THINGS dataset to reshape the large-scale organization of the space.
  2. Local Contrastive Loss: Acts as a regularizer to ensure that images which were neighbors in the original space remain neighbors after the transformation.
  3. Weight Regularization: Prevents the model from deviating unnecessarily from its original state by shrinking the transformation matrix toward a scaled identity matrix.

By balancing these losses, the gLocal transform "warps" the representation space to match human intuition about broad categories while maintaining the model's ability to distinguish subtle textures and shapes.

Key Results and Performance Gains

The effectiveness of the gLocal transform was evaluated on few-shot learning and anomaly detection tasks. The results demonstrate that human alignment serves as a powerful inductive bias for generalization.

The gLocal transform significantly improves few-shot learning performance across various CLIP models compared to both the original representations and naive alignment methods.
  • Few-Shot Learning: The transform substantially improved 5-shot learning accuracy across multiple datasets. For example, on the Entity-13 dataset, CLIP ViT-L/14 accuracy increased from 65.34% to 71.94%.
  • Anomaly Detection: The method achieved state-of-the-art results on coarse anomaly detection tasks (e.g., CIFAR-100), consistently outperforming original and naively aligned models.
  • Efficiency: Remarkably, these gains were achieved using only ~1,800 images for alignment—orders of magnitude fewer than the millions used in pretraining. This suggests that the necessary concepts are already present in the model and simply require rearrangement.

Conclusion and Future Directions

The gLocal transform proves that aligning AI with human global structure can significantly enhance performance on downstream tasks that require "learning to learn." While the transform did not consistently improve standard ImageNet-1K classification—possibly due to a mismatch in conceptual coverage—it highlights the value of human similarity judgments as a training signal. Future work may involve integrating these human-like biases directly into the initial training objectives of neural networks.

Summary updated

BibTeX
@inproceedings{muttenthaler2023improving,
  author = {Lukas Muttenthaler and Lorenz Linhardt and Jonas Dippel and Robert A. Vandermeulen and Katherine L. Hermann and Andrew K. Lampinen and Simon Kornblith},
  title = {{Improving neural network representations using human similarity judgments}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=Nh5dp6Uuvx},
}
Figure