The Violation of Expectation Paradigm for Commonsense AI

30 concepts21 papers1 proposal

How do we know if a machine truly understands that an object cannot pass through a solid wall, or that a person walks toward a water fountain because they are thirsty? Standard computer vision benchmarks evaluate models on tasks like object detection or next-frame pixel prediction. However, a model can excel at surface-level pixel matching without grasping the underlying laws of physics or psychology.

To bridge this gap, AI researchers have borrowed a powerful methodology from developmental psychology: the Violation-of-Expression (VoE) paradigm. When human infants cannot yet speak, psychologists measure how long they look at an event to gauge their understanding. If an infant stares significantly longer at a ball passing through a wall than a ball bouncing off it, it indicates that the impossible event violated their internal expectations. In AI, this paradigm translates into evaluating models by measuring their computational "surprise" when presented with physically or logically impossible scenarios.

What is Computational Surprise?

To implement VoE, we must first define what it means for a machine to experience surprise. As early as 1987, Ortony et al. clarified that surprise is not merely the failure of an active, explicit forecast. Rather, it often stems from the violation of passive assumptions—background beliefs about how the world works that only become active when they are contradicted.

In modern deep learning, this passive monitoring is operationalized through prediction errors or probabilistic likelihoods. For instance, Smith et al. quantified surprise as the negative log-likelihood of observed visual data given the internal predictions of a probabilistic physics engine. If an object suddenly teleports or vanishes, the likelihood drops precipitously, generating a massive surprise signal.

Similarly, in generative architectures like video diffusion models, Yuan et al. utilized the model's own denoising loss as a proxy for surprise. A physically impossible video requires more denoising correction (higher loss) because it deviates from the natural physical distributions the model has internalized. When a model integrates latent imagination with ongoing observation, as demonstrated by Becker, prediction errors naturally spike the moment an object violates core physical constraints.

Probing Intuitive Physics

A major application of the VoE paradigm is benchmarking "intuitive physics"—the baseline common sense required to navigate a physical world. A typical VoE physics benchmark presents a model with pixel-matched pairs or quadruplets of video sequences. One sequence depicts a physically plausible event (e.g., a ball rolling behind a screen and emerging on the other side), while the other depicts an impossible violation (e.g., the ball disappearing or emerging instantly).

Several benchmarks have been established to test these concepts:

  • IntPhys 2019 and IntPhys 2 (Riochet et al.; Bordes et al.) evaluate core principles like object permanence, solidity, and continuity. If a model understands physics, its internal representation error or plausibility score should clearly differentiate between the two scenarios.
  • GRASP (Jassim et al.) extends this to multimodal language models, revealing that even advanced video-LLMs frequently fail to perform better than a coin flip on these foundational tasks.
  • InfLevel (Weihs et al.) targets the physical reasoning capabilities of a 4.5-month-old infant, warning that standard out-of-distribution statistical metrics can sometimes mistake surface-level visual anomalies for genuine physical understanding.

Despite massive scaling, LeGris et al. note that training models purely on visual data—even developmentally realistic egocentric video—often fails to build an internal physics engine, highlighting the difficulty of learning these latent constraints from pixels alone. Conversely, object-centric models like PLATO (Piloto et al.) show that representing the world as discrete objects rather than raw pixels allows networks to rapidly internalize physics and exhibit human-like surprise when expectations fail. A comprehensive overview of these methodologies is provided in the survey by Duan et al..

Probing Intuitive Psychology

The utility of VoE is not restricted to inanimate objects; it is equally vital for evaluating "intuitive psychology"—the ability to reason about the hidden goals, preferences, and mental states of other agents.

In this context, the VoE setup typically involves a familiarization phase to establish an agent's behavioral patterns or preferences, followed by a test phase showing either a rational or irrational action. For example, the Baby Intuitions Benchmark (BIB) and the AGENT benchmark (Shu et al.) test whether models expect agents to act efficiently to achieve goals.

If an agent takes a circuitous, highly inefficient path to an object without any physical obstacles, a model with a built-in "rationality prior" will experience a spike in prediction error. As shown by Bortoletto et al. with the IRENE architecture, models that learn to expect efficient, goal-directed movement can successfully identify irrational behaviors by leveraging this exact discrepancy between predicted and observed actions.

From Detection to Explanation: Surprise-Driven Refinement

While detecting a violation is a crucial first step, advanced architectures use surprise actively to update and correct their internal world models. When a model encounters a massive prediction error, it faces a choice: flag the universe as broken, or find a hidden variable that explains the anomaly.

In the "Surprise and Explain" (SnE) framework proposed by Nguyen et al., when an object's trajectory deviates from a simple linear path, the system attempts to reconcile the surprise using predefined physical explanations like an unobserved occlusion or collision. If no logical explanation can be found, only then is the event flagged as a true violation of physical laws.

Taking this a step further, Dai introduced the X-VoE benchmark and the Explanation-based Physics Learner (XPL). When XPL encounters an unexpected physical outcome, it treats the surprise as a signal to dynamically revise its mental map. Instead of merely registering an error, the model infers and visually reconstructs hypothetical hidden objects (such as an invisible barrier) to resolve the logical inconsistency. This mirrors human cognitive flexibility, where surprise acts as a catalyst for state refinement and deeper causal learning.

Key Foundational Work

Surprisingness and Expectation Failure: What's the Difference? (1987) — contributed 1 concept elements to this direction.
Learning Unknown Event Models (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_surprise_based_evaluation_7d3cb71a,
  title = {surprise-based evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surprise-based-evaluation/rdr_7d3cb71a985c42d28f5823fcd8ee5c89},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_surprise_based_evaluation_7d3cb71a_v1,
  title = {surprise-based evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surprise-based-evaluation/rdr_7d3cb71a985c42d28f5823fcd8ee5c89/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake