Persistent-Transient Duality in Human Behavior Modeling

Hung Tran, Vuong Le, Svetha Venkatesh, Truyen Tran

Introduction

Predicting human motion is relatively straightforward when an individual is simply walking across a room. However, the task becomes significantly more complex when that person suddenly reaches out to grab a coffee cup or move a chair. Current Human-Object Interaction (HOI) models typically employ a single, fixed mechanism to track all entities in a scene. The primary limitation of this approach is that human behavior is inherently "dual." We maintain a background "plan" (e.g., walking) while occasionally deviating to perform "emerging tasks" (e.g., picking up an object).

Existing models often fail to capture this discrete switch, treating a small object with the same relational weight as the entire room. This lack of differentiation leads to "sluggish" adaptations and unrealistic motion predictions. To address this, researchers have developed Persistent-Transient Duality (PTD) Networks, a "parent-child" neural architecture that mimics how humans switch between general movement and focused interaction.

Methods: The PTD Architecture

The PTD framework factorizes behavior into two distinct processes that operate at different scales:

  1. The Persistent Channel (The Parent): This recurrent relational network maintains a "global view" of the scene. It tracks the long-term, slow-changing dynamics of all entities—both humans and objects—across space and time.
  2. The Transient Channel (The Child): When a human begins interacting with an object, the system "zooms in." This channel creates a temporary, egocentric graph centered on the human. By transforming coordinates so motion is relative to the subject, it filters out global noise and focuses strictly on the interaction.
  3. The Transient Switch: This neural unit monitors the distance and state of nearby objects. It automatically "switches on" a transient process when an interaction is imminent and "switches off" once the task is complete, returning control to the persistent channel.
Overview of the Persistent-Transient Duality (PTD) framework, showing the switching mechanism between global scene dynamics and localized interactions.

The model is trained end-to-end, learning to discover this duality structure automatically without requiring manual labels for interaction start or end points.

Egocentric Representation

A key innovation in the Transient Channel is the use of egocentricity. By shifting to a human-centric coordinate system during interactions, the model maps various motions into the same aligned space. This makes learning significantly more efficient than attempting to understand interactions from a static, global camera perspective.

The Transient Channel focuses on a specific human-object interaction using an egocentric representation to isolate the task from global scene noise.

Evaluation and Results

The model was evaluated using a subset of the KIT Whole-Body Human Motion Database (WBHM), featuring 233 videos with 20 different entity classes. The researchers utilized 3D skeleton poses (18 joints) for humans and 3D bounding boxes for objects.

Quantitative Performance

PTD consistently outperformed several baselines, including GRU, Structural RNN (SRNN), and the previous state-of-the-art, Context-aware RNN (CRNN). The performance gap was most pronounced in long-term predictions (up to 2 seconds into the future), where standard models typically drift significantly. PTD maintained a lower joint position error (mm) across all tested scenarios.

Qualitative Insights

Visual analysis revealed that while older models like CRNN exhibited "sluggish" attention maps that failed to react quickly to hand-object contact, the PTD model’s attention map snapped instantly to the object of interest. This resulted in much more realistic and stable motion paths during the critical moments of interaction.

Conclusion and Future Work

While PTD represents a significant advancement in HOI prediction, several areas for future research remain. The "Transient Switch" currently relies on distance-based factors; optimizing these thresholds for cluttered environments is a priority. Additionally, future work will explore "high-dexterity" tasks involving multiple simultaneous transient channels, such as a human using both hands to manipulate different objects at once. Despite these challenges, the persistent-transient duality offers a robust framework for more human-like robotic perception and motion planning.

Summary updated

Abstract

We propose to model the persistent-transient duality in human behavior using a parent-child multi-channel neural network, which features a parent persistent channel that manages the global dynamics and children transient channels that are initiated and terminated on-demand to handle detailed interactive actions. The short-lived transient sessions are managed by a proposed Transient Switch. The neural framework is trained to discover the structure of the duality automatically. Our model shows superior performances in human-object interaction motion prediction.
BibTeX
@misc{tran2022persistent,
  author = {Hung Tran and Vuong Le and Svetha Venkatesh and Truyen Tran},
  title = {{Persistent-Transient Duality in Human Behavior Modeling}},
  year = {2022},
  eprint = {2204.09875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.09875},
}
Figure