Designing Perceptual Puzzles by Differentiating Probabilistic Programs

Kartik Chandra, Tzu-Mao Li, Joshua B. Tenenbaum, Jonathan Ragan-Kelley

TL;DR

This paper introduces a systematic way to "hack" human vision by automatically generating new visual illusions. By treating illusions as adversarial examples for Bayesian models of human perception and using a novel differentiable probabilistic programming language, the authors successfully synthesized stimuli that trick both computer models and real people regarding color, size, and facial expressions.

What Problem Does This Paper Address?

Human perception is not a direct window into reality; it is an act of "inverse rendering," where our brains try to infer the most likely 3D scene that could have produced the 2D image hitting our retinas. Occasionally, this process fails, resulting in famous illusions like "The Dress."

While researchers have long studied these glitches, finding new ones has traditionally been a matter of luck or painstaking manual craft. Standard AI "adversarial examples" (which fool neural networks) rarely work on humans because deep learning models don't perceive the world the way we do. This paper seeks to bridge that gap by building a tool to systematically search for images that trigger "confidently incorrect" inferences in principled, human-like models of perception.

Methods

The authors propose a framework they call "inverse inverse rendering." If vision is the process of inferring a scene from an image, then designing an illusion is the process of optimizing an image to produce a specific, incorrect scene inference.

To do this efficiently, they developed two key technical contributions:

  1. Differentiable Probabilistic Programming Language (PPL): They built a PPL where the inference process itself—specifically Hamiltonian Monte Carlo (HMC)—is a first-class, differentiable function. This allows them to use gradient descent to "tweak" an input image until the model's output matches a desired (illusory) state.
  2. Reversible Dynamics: Differentiating through thousands of steps of a physics-based simulation (like HMC) is incredibly memory-intensive. The authors adapted "reversible learning" techniques, recomputing intermediate values by running the simulation backward during the gradient calculation to keep the memory footprint manageable.

Datasets

The paper doesn't rely on a massive training dataset. Instead, it uses small, principled generative models:

  • Color Constancy: A model of light (Planckian radiator) and Lambertian surfaces (wedge-top erasers).
  • Size Constancy: A model of 3D table geometry and camera perspective.
  • Face Perception: The Basel Face Model, a 3D morphable model used to generate human face meshes.
  • Human Validation: The authors conducted an online study with 60 participants to confirm that their synthesized color illusions actually produced diverse, conflicting perceptions in humans.

Key Insights and Ideas

  • Illusions as Adversarial Examples: The core "aha moment" is that illusions are to human cognitive models what adversarial examples are to deep neural networks. By optimizing against a model that incorporates human-like priors (like "light usually comes from above" or "tables are usually a certain size"), you can find the specific edge cases of human logic.
  • Differentiating the Inference, Not Just the Model: Most differentiable systems optimize the parameters of a model. This work optimizes the input to the model by backpropagating through the entire Bayesian inference chain.
  • Exploiting Prior Diversity: The authors realized that illusions like "The Dress" are powerful because different people have different internal "priors" (e.g., some assume cool light, others warm). Their system finds images that are highly sensitive to these slight differences in priors, leading to "perceptual bi-stability."

Key Results

  • Color Constancy: The system generated images of erasers where different viewers reported wildly different hues (e.g., one person seeing blue/black while another sees white/gold). In human trials, these "illusory" images showed high variance in reported color, whereas control images were perceived consistently.
  • Size Constancy: The tool automatically generated "forced perspective" scenes. For example, it found a camera angle and table arrangement where two far-apart tables appear to be stacked directly on top of each other.
  • Face Perception: They synthesized 3D face meshes that appear to change expression (e.g., from neutral to smiling) simply by moving the light source. They even 3D-printed these meshes to prove the illusion works with physical flashlights in the real world.

Limitations and Future Work

The current system is limited to continuous random variables. It cannot yet generate illusions involving discrete choices, such as the "Rubin's Vase" (face vs. vase) illusion, because you cannot easily differentiate through a binary "either/or" decision.

Additionally, the "yield" for face illusions was relatively low (about 10% were compelling). The authors attribute this to a mismatch between their model's priors (based largely on European faces) and the vast complexity of human face perception. They suggest that future work could extend this "inverse inverse" approach beyond vision to other domains like intuitive physics (making objects that seem to defy gravity) or social cognition (creating scenarios that lead to specific social misunderstandings).

Summary updated

Abstract

We design new visual illusions by finding "adversarial examples" for principled models of human perception -- specifically, for probabilistic models, which treat vision as Bayesian inference. To perform this search efficiently, we design a differentiable probabilistic programming language, whose API exposes MCMC inference as a first-class differentiable function. We demonstrate our method by automatically creating illusions for three features of human vision: color constancy, size constancy, and face perception.
BibTeX
@inproceedings{chandra2022designing,
  author = {Kartik Chandra and Tzu-Mao Li and Joshua B. Tenenbaum and Jonathan Ragan-Kelley},
  title = {{Designing Perceptual Puzzles by Differentiating Probabilistic Programs}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2022},
  doi = {10.1145/3528233.3530715},
}

Loading related content...

Figure