HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies

Youngrae Kim, Yejin Jang, Eunho Kim, Youjin Sung, Kun-Woo Song, Sangpil Kim, Sang Ho YoonarXiv 2026 · arXiv

TL;DR

HAPACT introduces a new benchmark for Human-Centric Physical Impact Localization, a task requiring AI to identify exactly when a person is hit in a video and where on their body the impact occurs. By providing over 2.8K video-query pairs from movies with fine-grained SMPL-X mesh annotations, the authors demonstrate that current state-of-the-art models struggle significantly with the brief, forceful nature of physical impacts.

What Problem Does This Paper Address?

Most research in "physical reasoning" for AI has focused on everyday, sustained interactions—think of a person picking up a cup or sitting on a chair. These actions are characterized by relatively long durations and predictable contact points.

However, the real world (and the cinematic one) is full of impact events: short, forceful, and often externally initiated physical interactions like punches, falls, or being hit by an object. These events are critical for understanding the physical consequences of actions, yet they are largely missing from current datasets.

The authors identify a major gap: we lack benchmarks that test a model's ability to reason about these abrupt events both temporally (finding the exact frames) and spatially (identifying the affected body parts).

Figure 1: Human-Centric Impact Localization. Given a video and a text query describing a target impact event, the task requires localizing when the queried impact affects the person and which body regions are affected in each frame.

Methods

The researchers formulated a two-part task:

  1. Temporal Localization: Given a text query (e.g., "A character is punched"), predict the start and end frames of that specific impact.
  2. Spatial On-body Localization: For those frames, identify which vertices on a 3D human body model (SMPL-X) are affected.

To tackle the spatial task, where traditional models often fail because they aren't designed for text-conditioned reasoning, the authors proposed a novel baseline: Query-Guided Region Selection. They fine-tuned a Multimodal Large Language Model (MLLM), specifically Cosmos-Reason2, to select affected body regions from a predefined list based on the video frame and the text query.

The dataset construction itself was a multi-stage pipeline involving genre-based filtering of movie clips, candidate generation using Gemini 2.5 Pro, and rigorous human annotation using a custom-built 3D "painting" interface for the body mesh.

Datasets

The paper introduces HAPACT (HumAn-centric imPACT), a benchmark built from high-resolution movie clips.

  • Scale: 2,884 video-query pairs across 212 movies.
  • Granularity: Over 20K fine-grained human-annotated frames.
  • Uniqueness: Unlike standard video grounding datasets where events last several seconds, HAPACT moments are extremely brief, averaging just 0.20 seconds (approx. 4.8 frames).
Figure 3: Data Distribution Overview. (a) Distribution of impact occurrences on body mesh, where 1 indicates high and 0 indicates low occurrence probability; (b) Distribution of start-end positions of impact moments, normalized from 0 to 1 relative to the total video length; (c) Distribution of movie genres; (d) Distribution of moment lengths by frame count; (e) Distribution of single- and multi-moment queries; (f) Word cloud of verbs and nouns in the description of HAPACT. It shows that verbs such as “strike” and nouns such as “man” appear frequently, emphasizing that the dataset focuses on physical interactions. More detailed statistical information can be found in Section A.5.

Key Insights and Ideas

  • Impacts are "Temporal Bottlenecks": Because impact events are so short, even a tiny shift in predicted boundaries leads to a massive drop in accuracy (tIoU). Current video moment retrieval models are "lazy"—they are tuned for longer actions and struggle with the sub-second precision required here.
  • Contact $\neq$ Impact: The authors tested existing "human-scene contact" models and found they performed poorly. These models can tell if a hand is touching a wall, but they can't distinguish between a hand resting on a wall and a shoulder slamming into it based on a specific text query.
  • The Multi-Moment Struggle: General MLLMs (like GPT-4o) are decent at finding the most "salient" (obvious) impact in a clip but struggle significantly when a query describes multiple events (e.g., "punched and slammed"). Their performance drops by over 50% when moving from single to multi-moment queries.

Key Results

  • Temporal Performance: Retrieval-based models improved with fine-tuning on HAPACT, but the best scores remained moderate. GPT-5.4 achieved the highest zero-shot performance (42.64 [email protected]), yet it still struggled with precise boundary detection for very short impacts.
  • Spatial Performance: Traditional contact-based baselines (like BSTRO and DECO) achieved F1 scores below 9%. In contrast, the proposed MLLM region-selection baseline reached an F1 of 18.59%, showing that text-conditioning is essential for this task, even if the output is currently limited to coarse body regions.
  • Duration Sensitivity: There is a clear performance degradation as impacts get shorter. Models are significantly more accurate on "long" impacts (7+ frames) than on "short" ones (1-2 frames).
Performance drops sharply for the shortest impact events, highlighting a major challenge for current architectures.

Limitations and Future Work

The authors acknowledge several key limitations:

  • Annotation Subjectivity: Determining exactly which vertices are hit during a fast-moving, occluded impact is inherently subjective, leading to some variance among human annotators.
  • Domain Bias: The dataset is movie-based. While movies provide rich physical interactions, the "staged" nature of cinematic impacts might not perfectly translate to real-world scenarios like sports injuries or car accidents.
  • Coarse Spatial Reasoning: The current best spatial baseline works at the "region" level (e.g., "left arm") rather than the "vertex" level. Moving toward precise, mesh-level impact estimation remains an open challenge.

Future work suggests expanding the dataset to more diverse video domains and developing architectures specifically designed for high-frequency, short-duration physical reasoning.

Summary updated

Abstract

Physical reasoning about humans in videos has largely focused on everyday interactions, where contact, pressure, pose, and motion support embodied perception and contact-aware applications. Still, short and forceful properties like impact events remain relatively underexplored, despite their relevance to broader understanding of how abrupt physical events affect the human body. To fill this gap, we introduce human-centric impact localization, a new task that localizes target physical impacts temporally within a video and spatially on the human body. We further present HAPACT, a HumAn-centric imPACT dataset and benchmark built from movie clips, containing over 2.8K video-query pairs with more than 20K fine-grained human-annotated frames. We built HAPACT around viewer-grounded annotation, text-conditioned localization, and supervision at frame-level temporal and SMPL-X-based spatial granularity. To demonstrate the effectiveness of our benchmark, we conduct extensive evaluations of existing methods, employing video moment retrieval models for temporal localization and human-scene contact models for spatial localization. For the spatial task, we propose a text-conditioned baseline by reformulating localization as query-guided body-region selection. Benchmark results show that physical impact localization remains challenging for existing methods, motivating HAPACT as a dedicated benchmark to support physical impact reasoning in videos.
BibTeX
@misc{kim2026hapact,
  author = {Youngrae Kim and Yejin Jang and Eunho Kim and Youjin Sung and Kun-Woo Song and Sangpil Kim and Sang Ho Yoon},
  title = {{HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies}},
  year = {2026},
  eprint = {2610.22895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.22895},
}
Figure