CoRelate: Coordinated Human-Object Motion Synthesis through Interaction Relations

Zhiwen Qiu, Daisy Lin, Diyu Zhou, Jialin Wu, Rosita Ji, Saleh KalantariarXiv 2026 · arXiv

Generating realistic animations of a person interacting with an object—like picking up a lamp or opening a refrigerator—is surprisingly difficult for AI. While current models can often get the person to the right place and the object to its destination, they frequently fail at the "handshake": the hand might hover inches from a handle, or the object might start moving before the hand even touches it.

Researchers from Cornell University have introduced CoRelate, a new diffusion-based framework that solves this coordination problem by treating the relationship between the hand and the object as a first-class citizen.

TL;DR

CoRelate is a motion synthesis framework that ensures sustained, realistic hand contact during human-object interactions by using explicit "interaction relations" and task feedback. By jointly refining both human and object trajectories during the AI's generation process, it achieves a nearly 2x improvement in interaction success over previous state-of-the-art methods.

What Problem Does This Paper Address?

In human-object motion synthesis, the goal is usually twofold: the object must reach a target (the "task"), and the human must move naturally to make that happen (the "interaction").

The challenge lies in coordination. If you only optimize for the object reaching its goal, the human might "telekinetically" move it. If you only optimize for hand contact, the human might fail to actually move the object to the target. Previous methods often treated these as separate constraints or relied on simple proximity. This leads to common failure cases like "late contact" (the object moves too early) or "contact not sustained" (the hand slips off mid-motion), as shown in Figure 1.

Comparison of common failure cases in existing models versus the coordinated results from CoRelate.

Methods

CoRelate introduces a three-stage approach to ensure the human and object move in lockstep:

  1. Interaction Relations ($\mathcal{R}$): Instead of just looking for "contact," the model defines a specific relationship. This includes which hand is used, which part of the object it targets, the exact relative position (the "wrist reference"), and the timing (when to establish, hold, and release contact).
  2. Feedback-Driven Joint Refinement: This is the core of the framework. During the diffusion denoising process (where the AI turns noise into motion), a temporal refiner looks at two signals:
  3. Relation Feedback: How far is the hand from its intended relative position on the object?
  4. Task Residuals: How far is the object from its final goal? The refiner then updates both the human and the object trajectories simultaneously to minimize these errors.

  5. Temporal Motion Correction: After the main motion is generated, a final optimization step fixes minor "slips" or penetrations by slightly adjusting the arm's joints while keeping the object's path and the rest of the body's motion fixed.

The CoRelate architecture, showing how interaction relations and task feedback are used to refine motion during the denoising process.

Datasets

The authors evaluated CoRelate on two primary benchmarks:

  • FullBodyManipulation: A dataset containing roughly 10 hours of paired human-object motion involving 15 different rigid objects.
  • ParaHome: A more complex dataset featuring household interactions with both rigid objects (like chairs) and articulated parts (like cabinet doors and drawers).

Key Insights and Ideas

  • Joint Updates are Essential: One of the "aha" moments in the paper is that you can't just fix the human to match the object. Sometimes, the most natural way to fix a contact error is to move the object slightly toward the hand. CoRelate’s refiner is trained to decide which one to move.
  • Relations vs. Proximity: Simply telling a model to "be close to the object" isn't enough. By defining a "hold" phase where the hand must maintain a fixed relative pose to a specific part (like a drawer handle), the motion becomes much more stable.
  • Articulated Objects: The framework treats a drawer handle or a fridge door as a "part" with its own local coordinate system. This allows the same math to work for picking up a box (rigid) or pulling a lever (articulated).

Key Results

The researchers used a metric called Complete Interaction Success (CIS), which measures the percentage of outputs that satisfy the task goal, maintain proper contact, and look physically plausible.

  • Rigid Objects: On the FullBodyManipulation dataset, CoRelate achieved a 71.2% CIS, a massive jump from the previous best baseline (MaMi-HOI) at 36.8%.
  • Articulated Objects: On the ParaHome dataset, CoRelate reached 36.0% CIS, more than doubling the performance of specialized baselines like SK-HOI (16.2%).

Qualitatively, the model produces much more convincing "pulling" and "carrying" motions where the hand stays glued to the interaction point throughout the entire movement (see Figure 3).

Qualitative comparisons showing CoRelate maintaining contact while baselines like CHOIS and MaMi-HOI suffer from hand-object separation.

Limitations and Future Work

While CoRelate is a significant step forward, the authors note a few areas for improvement:

  • Fixed Relations: Currently, the "who, what, and when" of the interaction are determined early in the process. The model can't easily decide to "regrasp" an object if the initial plan becomes awkward.
  • Physics: The model is primarily geometric. It doesn't explicitly calculate contact forces or friction, which means it might occasionally generate motions that are geometrically perfect but physically impossible (e.g., lifting a very heavy object with one finger).
  • Complexity: The current representation doesn't support complex multi-stage interactions, like picking up a key, putting it in a lock, and then turning it, within a single motion window.

Summary updated

Abstract

Human-object motion synthesis requires coordinated human participation as objects move toward specified goals. However, reaching object targets does not ensure that contact is established and maintained, while adjustments that improve contact may interfere with task completion. We introduce CoRelate, a diffusion framework that coordinates full-body human-object motion through explicit interaction relations and task feedback for rigid and articulated objects. We represent intended participation through hand-part relations that specify the participating hand, its part-relative reference position, and participation interval. These relations serve both as generation conditions and as references for measuring interaction realization. At each denoising step, a temporal refiner trained jointly with the generator combines relation deviations, task residuals, and motion features from the current prediction to jointly update human and object motion across the sequence. After generation, temporal motion correction addresses remaining contact failures by optimizing local motion segments while preserving object trajectories and protected human motion. Experiments demonstrate improved object-goal satisfaction, contact F1, and complete interaction success over the evaluated baselines. Ablation studies support the benefits of relation conditioning and the complementary roles of task and relation feedback.
BibTeX
@misc{qiu2026corelate,
  author = {Zhiwen Qiu and Daisy Lin and Diyu Zhou and Jialin Wu and Rosita Ji and Saleh Kalantari},
  title = {{CoRelate: Coordinated Human-Object Motion Synthesis through Interaction Relations}},
  year = {2026},
  eprint = {2611.25166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25166},
}
Figure