Improving neural network representations using human similarity judgments

Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A. Vandermeulen, Katherine L. Hermann, Andrew K. Lampinen, Simon Kornblith

TL;DR

Researchers have developed the gLocal transform, a method that aligns the global structure of neural network representations with human similarity judgments while preserving their local details. By making AI "see" the big picture more like humans do—grouping concepts like "animals" or "tools" even if they look different—they significantly improved model performance on few-shot learning and anomaly detection tasks.

What Problem Does This Paper Address?

Deep neural networks are incredibly good at recognizing specific objects, but they often lack a "common sense" understanding of how different categories relate to one another. Standard training objectives (like ImageNet classification or contrastive learning) focus on local structure: they ensure that two pictures of the same tabby cat are close together in the representation space. However, they don't explicitly constrain global structure: the relative positioning of "cats," "dogs," and "raccoons" in the broader space.

While models implicitly learn some hierarchy, it's often suboptimal. Previous attempts to force models to align with human similarity judgments (like "which of these three objects is the odd one out?") often resulted in a "naive" transformation that destroyed the model's fine-grained local accuracy. The challenge is: how do we inject human-like global organization into a model without breaking the high-quality local features it learned during massive pretraining?

Methods

The authors introduce the gLocal transform, a linear transformation designed to achieve a "best-of-both-worlds" representation. It optimizes a objective function with three main components:

  1. Global Alignment Loss: This uses human triplet data (e.g., "A is more similar to B than to C") to reshape the large-scale organization of the representation space.
  2. Local Contrastive Loss: This is the "secret sauce." It acts as a regularizer that forces the transformed space to maintain the same local neighborhoods as the original model. If two images were neighbors before the transform, they should remain neighbors after.
  3. Weight Regularization: It shrinks the transformation matrix toward a scaled identity matrix, ensuring the model doesn't deviate unnecessarily from its original state.

By balancing these, the gLocal transform "warps" the space to match human intuition about broad categories while keeping the model's ability to distinguish between subtle textures and shapes intact.

Datasets

  • Alignment Data: The THINGS dataset, containing 4.7 million human triplet responses for 1,854 natural object images.
  • Downstream Evaluation (Few-Shot Learning): Entity-13 and Entity-30 (from the BREEDS benchmark), CIFAR-100 (coarse and fine), SUN397 (scenes), and DTD (textures).
  • Downstream Evaluation (Anomaly Detection): Standard benchmarks like CIFAR-10, CIFAR-100, and ImageNet30, along with distribution-shifted versions to test robustness.

Key Insights and Ideas

  • Human Alignment as an Inductive Bias: The core "aha moment" is that human visual representations are globally organized in a way that specifically facilitates learning from very few examples. By mimicking this organization, AI becomes better at "learning to learn."
  • Local vs. Global Trade-off: The paper demonstrates that "naive" alignment is destructive. You cannot simply maximize alignment with humans; you must treat the model's original local structure as a precious resource to be preserved.
  • The Power of Small Data: Remarkably, the authors used only ~1,800 images for alignment—three to six orders of magnitude fewer than the images used in the models' original pretraining—yet saw significant gains in generalization. This suggests that the "concepts" are already in the model; they just need to be rearranged.

Key Results

  • Few-Shot Learning: The gLocal transform substantially improved 5-shot learning accuracy across almost all datasets for image/text models (like CLIP). For instance, on the Entity-13 dataset, CLIP ViT-L/14 accuracy jumped from 65.34% to 71.94%.
  • Anomaly Detection: The method yielded state-of-the-art results on the CIFAR-100 coarse anomaly detection task. It consistently outperformed both the original models and the "naively" aligned versions.
  • Preservation of Structure: Visualization via PCA (global) and t-SNE (local) confirmed that gLocal successfully adopted human-like global hierarchies (grouping by "animacy" or "consumability") while maintaining the tight local clusters of the original untransformed model.

Limitations and Future Work

  • ImageNet-1K Inconsistency: The transform did not consistently improve performance on standard ImageNet-1K classification. The authors speculate this is because ImageNet might not capture the specific concepts humans use to organize their own representations, or the 1,854 images used for alignment weren't diverse enough to cover the ImageNet manifold.
  • Cost of Human Data: While the amount of data needed is small, human similarity judgments are much more expensive to collect than raw images or text.
  • Future Directions: The authors suggest that instead of a post-hoc transform, researchers could explore new initialization strategies or training objectives that bake this human-like global structure into the model from day one.

Summary updated

Abstract

Deep neural networks have reached human-level performance on many computer vision tasks. However, the objectives used to train these networks enforce only that similar images are embedded at similar locations in the representation space, and do not directly constrain the global structure of the resulting space. Here, we explore the impact of supervising this global structure by linearly aligning it with human similarity judgments. We find that a naive approach leads to large changes in local representational structure that harm downstream performance. Thus, we propose a novel method that aligns the global structure of representations while preserving their local structure. This global-local transform considerably improves accuracy across a variety of few-shot learning and anomaly detection tasks. Our results indicate that human visual representations are globally organized in a way that facilitates learning from few examples, and incorporating this global structure into neural network representations improves performance on downstream tasks.
BibTeX
@inproceedings{muttenthaler2023improving,
  author = {Lukas Muttenthaler and Lorenz Linhardt and Jonas Dippel and Robert A. Vandermeulen and Katherine L. Hermann and Andrew K. Lampinen and Simon Kornblith},
  title = {{Improving neural network representations using human similarity judgments}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=Nh5dp6Uuvx},
}
Figure