Preliminary Action Recognition

Yasufumi Kawano, Yoshiki Nagasaki, Kensho Hara, Yoshimitsu Aoki, Hirokatsu KataokaJournal of the Japan Society for Precision Engineering 2023 · DOI

Abstract

In this paper, we propose preliminary action recognition, an action anticipation method designed for real-time use. Preliminary action recognition detects a person's intent before they perform an action by identifying specific "preliminary actions" that precede the main event. Conventional anticipation methods face two primary challenges: they require long video contexts (10-20 seconds) for short-term predictions and often suffer from "leakage" in training datasets where action transitions are artificially predictable. Our proposed method addresses these issues, enabling real-time operation without long-term buffers. Furthermore, we introduce a model that discriminates between visually similar action and preliminary action classes by increasing the latent distance between temporally adjacent classes.

Introduction: The Challenge of Real-Time Anticipation

Traditional action anticipation—predicting a future activity based on current observations—is often computationally prohibitive for real-world deployment. Most existing models rely on extensive video history (10 to 20 seconds) to predict just one second ahead. Additionally, standard benchmarks often contain "leakage," where models learn to predict transitions based on clip boundaries rather than actual visual cues.

Humans, by contrast, do not need long histories to anticipate actions; we recognize the "wind-up" or preliminary motions (予備動作) that lead to a specific task. The researchers developed Preliminary Action Recognition (PAR) to mimic this human capability, focusing on the immediate moments preceding an activity.

Comparison of action recognition, anticipation, and the proposed preliminary action recognition (PAR) framework.

Methodology

The PAR framework treats anticipation as a simultaneous recognition task. Instead of extrapolating into the future, the model identifies explicitly defined "preliminary action" classes (e.g., pre-cut tomato) that occur just before the "action" class (cut tomato).

Architecture and Contrastive Learning

The system utilizes a Video Swin Transformer backbone to extract spatio-temporal features from short video segments (approximately 4 seconds). A key challenge is that preliminary actions and the actions themselves are often visually nearly identical. To solve this, the authors introduced a Siamese-style network architecture.

The proposed model architecture featuring a Transformer encoder and a contrastive loss ($L_{similarity}$) to separate adjacent temporal classes in the latent space.

The core innovation is the $L_{similarity}$ loss function. This loss increases the distance in the latent space between temporally adjacent classes (the "Anchor" preliminary action and the "Negative" action). By pushing these representations apart, the model learns to detect the subtle "micro-motions" that signal a transition, effectively reducing temporal confusion.

Key Results and Insights

The proposed method was evaluated on major benchmarks, including EpicKitchens-100, 50-Salads, UT-Kinect-Action, and Watch-n-Patch.

  • Performance Gains: On EpicKitchens-100, the model achieved a 15.29% Top-1 Accuracy for action anticipation, outperforming the standard Video Swin Transformer baseline.
  • Precision in Transitions: The model showed a significant boost in identifying the "pre-action" phase specifically, reaching 4.60% Top-1 on EpicKitchens compared to the baseline's 4.25%.
  • Simpler Dataset Success: On the UT-Kinect-Action dataset, accuracy reached 87.78%, a dramatic improvement over the 73.23% baseline.
  • Real-Time Viability: By eliminating the need for 20-second video buffers, the model is suitable for low-latency edge devices in robotics and smart surveillance.

Conclusion and Future Work

Preliminary Action Recognition shifts the paradigm of anticipation from "predicting the future" to "recognizing the preparation." While highly effective, the authors note that extremely subtle motions in low-resolution video remain a challenge. Future research will explore multi-action transitions—where one preliminary motion could lead to several possible outcomes—and apply the framework to autonomous driving and human-robot collaboration.

Summary updated

Abstract

In this paper, we propose preliminary action recognition which is an action anticipation method that can be used in real-time. Preliminary action recognition as a solution to action anticipation, which is the task of detecting the action of a person before the person performs the action. We define preliminary action class before action class that is anticipated by recognizing the preliminary actions specific to the action before it occurs, just as humans do. Conventional action anticipation methods have two problems for real-time use. First, it requires 10 to 20 seconds of video to anticipate 1 second ahead, and it can only anticipate every 10 seconds. Second, the video input in training always switches actions 1 second after the end of the video. This problem setting can be a leakage because there is no action break in real-time operation. Proposed method solves this problem and makes it possible to operate action prediction in real-time. In addition, since action classes and preliminary action classes tend to share a lot of similarities, it is difficult to distinguish between them. We propose a model that can discriminate the subtle differences in motion by increasing the latent distance between adjacent classes in terms of time.
BibTeX
@article{kawano2023preliminary,
  author = {Yasufumi Kawano and Yoshiki NAGASAKI and Kensho Hara and Yoshimitsu Aoki and Hirokatsu Kataoka},
  title = {{Preliminary Action Recognition}},
  journal = {Journal of the Japan Society for Precision Engineering},
  year = {2023},
  doi = {10.2493/jjspe.89.915},
}
Figure