TL;DR
HAPACT introduces a new benchmark for Human-Centric Physical Impact Localization, a task requiring AI to identify exactly when a person is hit in a video and where on their body the impact occurs. By providing over 2.8K video-query pairs from movies with fine-grained SMPL-X mesh annotations, the authors demonstrate that current state-of-the-art models struggle significantly with the brief, forceful nature of physical impacts.
What Problem Does This Paper Address?
Most research in "physical reasoning" for AI has focused on everyday, sustained interactions—think of a person picking up a cup or sitting on a chair. These actions are characterized by relatively long durations and predictable contact points.
However, the real world (and the cinematic one) is full of impact events: short, forceful, and often externally initiated physical interactions like punches, falls, or being hit by an object. These events are critical for understanding the physical consequences of actions, yet they are largely missing from current datasets.
The authors identify a major gap: we lack benchmarks that test a model's ability to reason about these abrupt events both temporally (finding the exact frames) and spatially (identifying the affected body parts).
Methods
The researchers formulated a two-part task:
- Temporal Localization: Given a text query (e.g., "A character is punched"), predict the start and end frames of that specific impact.
- Spatial On-body Localization: For those frames, identify which vertices on a 3D human body model (SMPL-X) are affected.
To tackle the spatial task, where traditional models often fail because they aren't designed for text-conditioned reasoning, the authors proposed a novel baseline: Query-Guided Region Selection. They fine-tuned a Multimodal Large Language Model (MLLM), specifically Cosmos-Reason2, to select affected body regions from a predefined list based on the video frame and the text query.
The dataset construction itself was a multi-stage pipeline involving genre-based filtering of movie clips, candidate generation using Gemini 2.5 Pro, and rigorous human annotation using a custom-built 3D "painting" interface for the body mesh.
Datasets
The paper introduces HAPACT (HumAn-centric imPACT), a benchmark built from high-resolution movie clips.
- Scale: 2,884 video-query pairs across 212 movies.
- Granularity: Over 20K fine-grained human-annotated frames.
- Uniqueness: Unlike standard video grounding datasets where events last several seconds, HAPACT moments are extremely brief, averaging just 0.20 seconds (approx. 4.8 frames).
Key Insights and Ideas
- Impacts are "Temporal Bottlenecks": Because impact events are so short, even a tiny shift in predicted boundaries leads to a massive drop in accuracy (tIoU). Current video moment retrieval models are "lazy"—they are tuned for longer actions and struggle with the sub-second precision required here.
- Contact $\neq$ Impact: The authors tested existing "human-scene contact" models and found they performed poorly. These models can tell if a hand is touching a wall, but they can't distinguish between a hand resting on a wall and a shoulder slamming into it based on a specific text query.
- The Multi-Moment Struggle: General MLLMs (like GPT-4o) are decent at finding the most "salient" (obvious) impact in a clip but struggle significantly when a query describes multiple events (e.g., "punched and slammed"). Their performance drops by over 50% when moving from single to multi-moment queries.
Key Results
- Temporal Performance: Retrieval-based models improved with fine-tuning on HAPACT, but the best scores remained moderate. GPT-5.4 achieved the highest zero-shot performance (42.64 [email protected]), yet it still struggled with precise boundary detection for very short impacts.
- Spatial Performance: Traditional contact-based baselines (like BSTRO and DECO) achieved F1 scores below 9%. In contrast, the proposed MLLM region-selection baseline reached an F1 of 18.59%, showing that text-conditioning is essential for this task, even if the output is currently limited to coarse body regions.
- Duration Sensitivity: There is a clear performance degradation as impacts get shorter. Models are significantly more accurate on "long" impacts (7+ frames) than on "short" ones (1-2 frames).
Limitations and Future Work
The authors acknowledge several key limitations:
- Annotation Subjectivity: Determining exactly which vertices are hit during a fast-moving, occluded impact is inherently subjective, leading to some variance among human annotators.
- Domain Bias: The dataset is movie-based. While movies provide rich physical interactions, the "staged" nature of cinematic impacts might not perfectly translate to real-world scenarios like sports injuries or car accidents.
- Coarse Spatial Reasoning: The current best spatial baseline works at the "region" level (e.g., "left arm") rather than the "vertex" level. Moving toward precise, mesh-level impact estimation remains an open challenge.
Future work suggests expanding the dataset to more diverse video domains and developing architectures specifically designed for high-frequency, short-duration physical reasoning.
Summary updated