From Static Snapshots to Continuous Interaction Sequences

22 concepts21 papers1 proposal

To understand or synthesize how humans interact with the physical world, computer vision and robotics have traditionally relied on static snapshots. We predict a 3D mesh of a hand grasping a bottle, or generate a static avatar positioned on a sofa. While these models can produce visually impressive single frames, they treat interaction as a fixed state rather than an ongoing process.

In reality, physical manipulation is inherently dynamic. A single frame cannot differentiate whether a hand is approaching an object, actively manipulating an articulated part, or releasing its grip. As Kris Kitani et al. observed in early action recognition work, static analysis fails to capture the evolution of hand-object interactions over time, making it difficult to distinguish between complex, multi-stage tasks that share similar static poses. To resolve this, a major research direction has emerged to shift interaction modeling away from isolated configurations and toward continuous temporal sequences.

The Representation Challenge: Mapping Space and Time

How do we represent an interaction so that a machine can reason about its temporal flow? Early spatial representations focused heavily on geometry. For instance, Korrawe Karunratanakul et al. introduced the Grasping Field (GF), an implicit representation mapping 3D points to their distances from both the hand and the object. While GF successfully generates realistic static grasps, the authors noted that modeling the full trajectory of reaching, grasping, and moving an object remains an open challenge for implicit spatial representations.

To inject differentiability and fluid dynamics into these representations, Théo Morales proposed CHOIR (Coarse Hand-Object Interaction Representation). By combining unsigned distance fields with a Gaussian-based contact representation, CHOIR allows diffusion models to synthesize and refine interactions. Yet, as the author emphasizes, a lack of explicit temporal awareness still limits these models to static or near-static scenarios, failing to capture how a hand approaches and manipulates an object over time.

Static Representation:  [ Hand/Object Pose ] ---> (Single Snapshot)
                                 vs.
Temporal Sequence:      [ Approach ] ---> [ Contact/Grasp ] ---> [ Manipulation Trajectory ]

Synthesizing Trajectories via Diffusion and Semantic Control

If the underlying representation can support motion, how do we synthesize the actual sequence? The answer has increasingly involved conditional generative models, such as diffusion frameworks, which excel at modeling complex, multi-modal trajectories over time.

The primary challenge here is control: ensuring the generated sequence adheres to human intent or text instructions. Kaifeng Zhao et al. developed COINS, a framework that synthesizes human-scene interactions from high-level semantic commands by composing atomic action-object pairs. However, the authors acknowledged that transitioning from these compositional snapshots to continuous transitions between different interaction states remains an unresolved hurdle.

For hand-object manipulation, recent frameworks attempt to guide the temporal sequence using fine-grained natural language. Zhongqun Zhang et al. introduced NL2Contact, using a staged diffusion model to generate intent-driven contact maps, noting that moving from single-frame predictions to dynamic sequences is vital to capture the fluid nature of tool use.

Similarly, Guangyi Han et al. proposed TOUCH to move beyond simple stable grasping toward free-form interactions like poking, pushing, and rotating. They highlight that current generative frameworks produce static snapshots rather than continuous temporal sequences, preventing them from capturing fluid actions like sliding a finger along a surface over time.

Grounding Motion in Intent and Affordances

Why does a human move their hand along a specific trajectory? The motion is dictated by the object's affordance—its functional purpose. To train models to generate realistic temporal sequences, researchers need datasets that capture this underlying intent rather than just geometric alignment.

Lixin Yang et al. introduced OakInk, a massive knowledge repository designed to bridge the gap between affordance perception and functional manipulation. They noted that existing datasets focus on steady-state poses rather than the continuous dynamic motions required to operate articulated tools. To address part-level functional interactions (like twisting a cap or pulling a handle), Juntao Jian et al. developed AffordPose, explicitly stating that transitioning from static configurations to dynamic trajectories is necessary to model complex, continuous manipulations.

The Universal Need for Temporal Coherence

This push for temporal continuity extends far beyond hand-object manipulation; it is a universal frontier across 3D generation, avatar synthesis, and environmental simulation:

  • Human-Scene Avatars: Yan Zhang et al. and Minseok Kim et al. both emphasize that while populating 3D scenes with static human poses has progressed, synthesizing coherent sequences of movement—such as navigating a room or the physical process of sitting down—remains an open challenge.
  • Tactile and Sensor Simulation: In tactile sensing, Jiahang Tu et al. developed TextToucher for static tactile images but noted that modeling continuous deformation and friction patterns for tactile video generation is the next critical step. In autonomous driving simulation, Yang Wu et al. introduced Text2LiDAR for point cloud generation, highlighting that single-frame synthesis lacks the temporal coherence and environmental persistence required for continuous simulation.

The shift from static configurations to continuous temporal sequences is driven by a simple realization: physical interaction is defined by its transitions. Whether a model is generating a hand turning a key, an avatar walking through a room, or a LiDAR sensor capturing a moving vehicle, the underlying physics and human intent can only be fully captured when we model the continuous flow of time.

Key Foundational Work

Understanding Hand-Object Manipulation with Grasp Types and Object Attributes (2016) — contributed 1 concept elements to this direction.
Learning Object Arrangements in 3D Scenes using Human Context (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_interaction_modeling_69fce5d6,
  title = {temporal interaction modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-interaction-modeling/rdr_69fce5d6e893481b8c022aa61c94b3ca},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_interaction_modeling_69fce5d6_v1,
  title = {temporal interaction modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-interaction-modeling/rdr_69fce5d6e893481b8c022aa61c94b3ca/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake