Improving neural network representations using human similarity judgments
Introduction: The Gap Between AI and Human Perception
Deep neural networks (DNNs) excel at fine-grained object recognition, yet they often lack a "common sense" understanding of how different categories relate. While a standard model can easily distinguish between two different breeds of cats (local structure), it may not naturally group "cats," "dogs," and "raccoons" together under the broader concept of "animals" (global structure). This discrepancy arises because standard training objectives, like ImageNet classification or contrastive learning, focus primarily on ensuring that similar images are close in the representation space without explicitly constraining the broader hierarchy.
Previous attempts to align AI representations with human similarity judgments often resulted in "naive" transformations that improved global structure at the expense of local accuracy. The challenge lies in injecting human-like organization without degrading the high-quality features learned during massive pretraining.
The gLocal Transform
To address this, researchers developed the gLocal transform, a linear transformation designed to achieve a "best-of-both-worlds" representation. The transform optimizes an objective function with three key components:
- Global Alignment Loss: Uses human triplet data (e.g., "A is more similar to B than to C") from the THINGS dataset to reshape the large-scale organization of the space.
- Local Contrastive Loss: Acts as a regularizer to ensure that images which were neighbors in the original space remain neighbors after the transformation.
- Weight Regularization: Prevents the model from deviating unnecessarily from its original state by shrinking the transformation matrix toward a scaled identity matrix.
By balancing these losses, the gLocal transform "warps" the representation space to match human intuition about broad categories while maintaining the model's ability to distinguish subtle textures and shapes.
Key Results and Performance Gains
The effectiveness of the gLocal transform was evaluated on few-shot learning and anomaly detection tasks. The results demonstrate that human alignment serves as a powerful inductive bias for generalization.
- Few-Shot Learning: The transform substantially improved 5-shot learning accuracy across multiple datasets. For example, on the Entity-13 dataset, CLIP ViT-L/14 accuracy increased from 65.34% to 71.94%.
- Anomaly Detection: The method achieved state-of-the-art results on coarse anomaly detection tasks (e.g., CIFAR-100), consistently outperforming original and naively aligned models.
- Efficiency: Remarkably, these gains were achieved using only ~1,800 images for alignment—orders of magnitude fewer than the millions used in pretraining. This suggests that the necessary concepts are already present in the model and simply require rearrangement.
Conclusion and Future Directions
The gLocal transform proves that aligning AI with human global structure can significantly enhance performance on downstream tasks that require "learning to learn." While the transform did not consistently improve standard ImageNet-1K classification—possibly due to a mismatch in conceptual coverage—it highlights the value of human similarity judgments as a training signal. Future work may involve integrating these human-like biases directly into the initial training objectives of neural networks.
Summary updated