Learning visual biases from human imagination

Carl Vondrick, Hamed Pirsiavash, Aude Oliva, Antonio TorralbaNeurIPS 2015 · OpenReview

In computer vision, "bias" is usually considered a negative trait. Most researchers spend their time trying to remove dataset bias—the tendency for a model to learn specific quirks of a dataset (like "all cows are on green grass") rather than the actual object. However, not all biases are detrimental. The human visual system has its own set of "natural" biases, such as preferring certain perspectives or grouping parts into wholes, that help us navigate an underconstrained world.

This paper asks a fascinating question: Can we capture these beneficial human biases and transfer them into machine recognition systems to help them generalize better? By converting human visual biases into mathematical priors, researchers have developed a method to "extract" mental images from the human brain, significantly improving machine learning models' ability to recognize objects, especially in "low-data" scenarios.

Extracting Human Bias from Noise

To capture these internal biases, the authors adapt a classic psychophysics technique called classification images. Traditionally, this involves showing a human an image buried in noise and observing how the noise affects their perception. This paper introduces three key innovations to apply this to computer vision:

  1. Hallucination from Pure Noise: Instead of adding noise to real images, they show humans visualizations of pure white noise sampled from computer vision feature spaces (like HOG or CNN layers).
  2. Feature Inversion: Because humans cannot interpret raw feature vectors, the researchers use "feature inversion" algorithms to turn abstract noise into visual patterns a human can actually see.
  3. The "Imagination" Template: Thousands of workers on Amazon Mechanical Turk were asked to look at these noise visualizations and identify if they resembled a specific object, such as a car.
Process of extracting human bias

By averaging the thousands of noise samples that people "hallucinated" as cars, a recognizable template emerged. Finally, these templates were integrated into a Support Vector Machine (SVM) using a novel "orientation constraint." This forces the AI's decision boundary to stay mathematically close to the "direction" of the human mental template, rather than just minimizing errors on a dataset.

Experimental Setup

The researchers validated their approach using several datasets and scenarios:

  • Synthetic Training: 150,000 points of i.i.d. white noise sampled from a standard normal distribution were used to generate the visualizations.
  • Evaluation: The PASCAL VOC 2011 dataset was used to test how well the human-biased models recognized real-world objects.
  • Generalization Tests: Models were trained on Caltech 101 and tested on PASCAL VOC (and vice versa) to see if human bias helps models perform better when moving between different datasets.
  • Custom Data: They collected 63 images of fire hydrants from Flickr to test categories not present in standard benchmarks.

Key Insights

The most striking finding is the "Aha!" moment: if you average enough "wrong" guesses (people seeing a car in random noise), a clear visual structure emerges. The human brain is so primed to see certain shapes that it imposes structure on randomness.

Emergent car template

Other fascinating insights include:

  • Cultural Differences in Imagination: When asked to find "sports balls" in noise, workers in India hallucinated red balls (resembling cricket balls), while US workers hallucinated orange/brown balls (resembling basketballs or American footballs). This demonstrates that the method captures deep-seated cultural mental models.
  • Shape vs. Color: The templates revealed that for some categories (like people), shape is the dominant bias, while for others (like fire hydrants), color is a primary cue.
  • Feature Space Matters: This process fails in raw RGB pixel space because it is too high-dimensional and unstructured. By operating in "feature space" (the internal language of AI), the noise already contains the building blocks (edges, textures) that the human eye can assemble into objects.

Results and Impact

Transferring human bias into machines yielded impressive results, particularly when training data was scarce:

  • Zero-Shot Recognition: Using only the human bias template (with zero real training images), the system achieved 27.5% Average Precision (AP) for cars on PASCAL VOC, compared to a 7.3% chance for a random guess.
  • Boosting Low-Data Learning: When the AI was given only one real positive example of a "person," adding the human bias prior boosted performance from 55.2% to 69.3% AP.
  • Cross-Dataset Generalization: Models trained on one dataset often fail on another. Adding human bias improved this "cross-domain" performance by about 5% AP, suggesting that human imagination is more "universal" than any single dataset.

Limitations and Future Directions

The authors acknowledge that these "imagination templates" are currently quite blurry and low-resolution. While they capture the "gist" of an object, they lack the fine-grained detail of a model trained on millions of real images.

Additionally, the process is currently expensive, requiring thousands of human labels per category. Future work might explore how to automate the extraction of these biases or use them to help AI "discover" rare categories that lack large existing datasets. Ultimately, this approach could also serve as a powerful tool for social psychologists to quantitatively map how different cultures perceive the world.

Summary updated

Abstract

Although the human visual system can recognize many concepts under challengingconditions, it still has some biases. In this paper, we investigate whether wecan extract these biases and transfer them into a machine recognition system.We introduce a novel method that, inspired by well-known tools in humanpsychophysics, estimates the biases that the human visual system might use forrecognition, but in computer vision feature spaces. Our experiments aresurprising, and suggest that classifiers from the human visual system can betransferred into a machine with some success. Since these classifiers seem tocapture favorable biases in the human visual system, we further present an SVMformulation that constrains the orientation of the SVM hyperplane to agree withthe bias from human visual system. Our results suggest that transferring thishuman bias into machines may help object recognition systems generalize acrossdatasets and perform better when very little training data is available.
BibTeX
@inproceedings{vondrick2015learning,
  author = {Carl Vondrick and Hamed Pirsiavash and Aude Oliva and Antonio Torralba},
  title = {{Learning visual biases from human imagination}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
  url = {https://openreview.net/forum?id=HJZtNwZu-H},
}

Loading related content...

Figure