CLEVRER-Humans: Describing Physical and Causal Events the Human Way

Jiayuan Mao, Xuelin Yang, Xikun Zhang, Noah D. Goodman, Jiajun Wu

TL;DR

CLEVRER-Humans is a new video reasoning benchmark that moves beyond synthetic, rule-based causality to capture how humans actually describe and judge physical events. By introducing human-annotated Causal Event Graphs (CEGs) and diverse natural language, the researchers reveal a significant performance gap: state-of-the-art models that "solve" synthetic benchmarks fail to align with human causal intuition and struggle with the richness of real-world language.

What Problem Does This Paper Address?

Most current benchmarks for physical reasoning, such as the original CLEVRER or CATER, rely on synthetic data. While useful, these datasets have two major flaws:

  1. Language Simplicity: Descriptions are generated from rigid templates (e.g., "the red ball hits the blue cube"), lacking the diversity of human expression.
  2. Heuristic Causality: Causal relationships are defined by simple rules—usually, if object A touches object B before an event, A is labeled the "cause."

However, human causal judgment is far more nuanced, often involving complex counterfactual reasoning (e.g., "Would the ball have exited the scene if the cube hadn't been there?"). As shown in the paper, these simple heuristics often contradict human labels. The authors aim to bridge this gap by creating a dataset where both the event descriptions and the causal judgments come directly from humans.

Methods

The authors developed a three-stage pipeline to build the CLEVRER-Humans dataset efficiently:

  1. Iterative Event Cloze Task: To collect diverse descriptions, they used a "cloze" (fill-in-the-blank) task. Starting with a seed event, annotators were asked to describe a cause or an effect (e.g., "The silver object starts to move because ____"). These new descriptions became seeds for the next iteration, building a dense Causal Event Graph (CEG).
  2. Neural Data Augmentation: Because human annotation is expensive, they trained a neural "trajectory-to-text" model. This model takes symbolic object trajectories (position, velocity) and generates candidate event descriptions, which are then filtered by humans for accuracy.
  3. Graded Causal Labeling: Instead of a simple yes/no, humans rated the "responsibility" of one event for another on a scale of 1 to 5. This captures the subjective and graded nature of human causality.
The overall labeling pipeline, showing the transition from video to human-written descriptions, neural augmentation, and final graph construction.

Datasets

  • CLEVRER: The original synthetic video dataset providing the visual foundation.
  • CLEVRER-Humans (New): Contains 1,099 videos, 5,538 unique human-written descriptions, and over 23,000 causal relationship annotations. It features a vocabulary nearly double the size of the original (157 vs. 82 words) and ten times the number of distinct verbs (30 vs. 3).

Key Insights and Ideas

  • Heuristics $\neq$ Human Intuition: The paper highlights cases where heuristic rules fail. For example, a model might label a stationary object as a "cause" simply because it was present, whereas a human recognizes that the object played no active role in the outcome.
  • Language as a Bottleneck: Models pretrained on synthetic data struggle with "out-of-vocabulary" human expressions. Even simple variations like "comes up" vs. "moves upward" can trip up high-performing models.
  • The "Because" Factor: Humans naturally use the connective "because" to link events, implying a deeper understanding of physical dynamics than just temporal succession.
Examples showing the divergence between CLEVRER’s heuristic labels and human judgment. In the second case, humans correctly identify that the sphere would have exited the scene regardless of the metal sphere's presence.

Key Results

The researchers benchmarked several state-of-the-art models, including NS-DR (neuro-symbolic) and ALOE (transformer-based). The results were striking:

  • Performance Drop: Models that achieved near-perfect scores (~98%) on the original CLEVRER dropped to near-chance levels (~54% per-option accuracy) on CLEVRER-Humans.
  • Human Baseline: Humans achieve 84.5% accuracy on the same tasks, highlighting a massive gap between current AI and human-level physical reasoning.
  • Data Efficiency: Large models like ALOE suffered from extreme overfitting due to the smaller size of the human-annotated training set, suggesting that current architectures are not yet "data-efficient" enough to learn from limited human feedback.

Limitations and Future Work

  • Dataset Size: While rich, the dataset is smaller than its synthetic counterparts (1,076 QA pairs), making it primarily a tool for zero-shot or few-shot evaluation rather than large-scale training.
  • Binarization: For the sake of benchmarking, the authors binarized the 1-5 responsibility scores. They note that future models should aim to predict these graded judgments directly.
  • Naturalistic Videos: The current work uses the simplified CLEVRER environment. The authors suggest extending these iterative annotation techniques to complex, real-world videos in the future.

Summary updated

Abstract

Building machines that can reason about physical events and their causal relationships is crucial for flexible interaction with the physical world. However, most existing physical and causal reasoning benchmarks are exclusively based on synthetically generated events and synthetic natural language descriptions of causal relationships. This design brings up two issues. First, there is a lack of diversity in both event types and natural language descriptions; second, causal relationships based on manually-defined heuristics are different from human judgments. To address both shortcomings, we present the CLEVRER-Humans benchmark, a video reasoning dataset for causal judgment of physical events with human labels. We employ two techniques to improve data collection efficiency: first, a novel iterative event cloze task to elicit a new representation of events in videos, which we term Causal Event Graphs (CEGs); second, a data augmentation technique based on neural language generative models. We convert the collected CEGs into questions and answers to be consistent with prior work. Finally, we study a collection of baseline approaches for CLEVRER-Humans question-answering, highlighting the great challenges set forth by our benchmark.
BibTeX
@inproceedings{mao2022clevrer,
  author = {Jiayuan Mao and Xuelin Yang and Xikun Zhang and Noah D. Goodman and Jiajun Wu},
  title = {{CLEVRER-Humans: Describing Physical and Causal Events the Human Way}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=1UqtmRiJIJ},
}
Figure