TL;DR
CLEVRER-Humans is a new video reasoning benchmark that moves beyond synthetic, rule-based causality to capture how humans actually describe and judge physical events. By introducing human-annotated Causal Event Graphs (CEGs) and diverse natural language, the researchers reveal a significant performance gap: state-of-the-art models that "solve" synthetic benchmarks fail to align with human causal intuition and struggle with the richness of real-world language.
What Problem Does This Paper Address?
Most current benchmarks for physical reasoning, such as the original CLEVRER or CATER, rely on synthetic data. While useful, these datasets have two major flaws:
- Language Simplicity: Descriptions are generated from rigid templates (e.g., "the red ball hits the blue cube"), lacking the diversity of human expression.
- Heuristic Causality: Causal relationships are defined by simple rules—usually, if object A touches object B before an event, A is labeled the "cause."
However, human causal judgment is far more nuanced, often involving complex counterfactual reasoning (e.g., "Would the ball have exited the scene if the cube hadn't been there?"). As shown in the paper, these simple heuristics often contradict human labels. The authors aim to bridge this gap by creating a dataset where both the event descriptions and the causal judgments come directly from humans.
Methods
The authors developed a three-stage pipeline to build the CLEVRER-Humans dataset efficiently:
- Iterative Event Cloze Task: To collect diverse descriptions, they used a "cloze" (fill-in-the-blank) task. Starting with a seed event, annotators were asked to describe a cause or an effect (e.g., "The silver object starts to move because ____"). These new descriptions became seeds for the next iteration, building a dense Causal Event Graph (CEG).
- Neural Data Augmentation: Because human annotation is expensive, they trained a neural "trajectory-to-text" model. This model takes symbolic object trajectories (position, velocity) and generates candidate event descriptions, which are then filtered by humans for accuracy.
- Graded Causal Labeling: Instead of a simple yes/no, humans rated the "responsibility" of one event for another on a scale of 1 to 5. This captures the subjective and graded nature of human causality.
Datasets
- CLEVRER: The original synthetic video dataset providing the visual foundation.
- CLEVRER-Humans (New): Contains 1,099 videos, 5,538 unique human-written descriptions, and over 23,000 causal relationship annotations. It features a vocabulary nearly double the size of the original (157 vs. 82 words) and ten times the number of distinct verbs (30 vs. 3).
Key Insights and Ideas
- Heuristics $\neq$ Human Intuition: The paper highlights cases where heuristic rules fail. For example, a model might label a stationary object as a "cause" simply because it was present, whereas a human recognizes that the object played no active role in the outcome.
- Language as a Bottleneck: Models pretrained on synthetic data struggle with "out-of-vocabulary" human expressions. Even simple variations like "comes up" vs. "moves upward" can trip up high-performing models.
- The "Because" Factor: Humans naturally use the connective "because" to link events, implying a deeper understanding of physical dynamics than just temporal succession.
Key Results
The researchers benchmarked several state-of-the-art models, including NS-DR (neuro-symbolic) and ALOE (transformer-based). The results were striking:
- Performance Drop: Models that achieved near-perfect scores (~98%) on the original CLEVRER dropped to near-chance levels (~54% per-option accuracy) on CLEVRER-Humans.
- Human Baseline: Humans achieve 84.5% accuracy on the same tasks, highlighting a massive gap between current AI and human-level physical reasoning.
- Data Efficiency: Large models like ALOE suffered from extreme overfitting due to the smaller size of the human-annotated training set, suggesting that current architectures are not yet "data-efficient" enough to learn from limited human feedback.
Limitations and Future Work
- Dataset Size: While rich, the dataset is smaller than its synthetic counterparts (1,076 QA pairs), making it primarily a tool for zero-shot or few-shot evaluation rather than large-scale training.
- Binarization: For the sake of benchmarking, the authors binarized the 1-5 responsibility scores. They note that future models should aim to predict these graded judgments directly.
- Naturalistic Videos: The current work uses the simplified CLEVRER environment. The authors suggest extending these iterative annotation techniques to complex, real-world videos in the future.
Summary updated