Generating realistic animations of a person interacting with an object—like picking up a lamp or opening a refrigerator—is surprisingly difficult for AI. While current models can often get the person to the right place and the object to its destination, they frequently fail at the "handshake": the hand might hover inches from a handle, or the object might start moving before the hand even touches it.
Researchers from Cornell University have introduced CoRelate, a new diffusion-based framework that solves this coordination problem by treating the relationship between the hand and the object as a first-class citizen.
TL;DR
CoRelate is a motion synthesis framework that ensures sustained, realistic hand contact during human-object interactions by using explicit "interaction relations" and task feedback. By jointly refining both human and object trajectories during the AI's generation process, it achieves a nearly 2x improvement in interaction success over previous state-of-the-art methods.
What Problem Does This Paper Address?
In human-object motion synthesis, the goal is usually twofold: the object must reach a target (the "task"), and the human must move naturally to make that happen (the "interaction").
The challenge lies in coordination. If you only optimize for the object reaching its goal, the human might "telekinetically" move it. If you only optimize for hand contact, the human might fail to actually move the object to the target. Previous methods often treated these as separate constraints or relied on simple proximity. This leads to common failure cases like "late contact" (the object moves too early) or "contact not sustained" (the hand slips off mid-motion), as shown in Figure 1.
Methods
CoRelate introduces a three-stage approach to ensure the human and object move in lockstep:
- Interaction Relations ($\mathcal{R}$): Instead of just looking for "contact," the model defines a specific relationship. This includes which hand is used, which part of the object it targets, the exact relative position (the "wrist reference"), and the timing (when to establish, hold, and release contact).
- Feedback-Driven Joint Refinement: This is the core of the framework. During the diffusion denoising process (where the AI turns noise into motion), a temporal refiner looks at two signals:
- Relation Feedback: How far is the hand from its intended relative position on the object?
-
Task Residuals: How far is the object from its final goal? The refiner then updates both the human and the object trajectories simultaneously to minimize these errors.
-
Temporal Motion Correction: After the main motion is generated, a final optimization step fixes minor "slips" or penetrations by slightly adjusting the arm's joints while keeping the object's path and the rest of the body's motion fixed.
Datasets
The authors evaluated CoRelate on two primary benchmarks:
- FullBodyManipulation: A dataset containing roughly 10 hours of paired human-object motion involving 15 different rigid objects.
- ParaHome: A more complex dataset featuring household interactions with both rigid objects (like chairs) and articulated parts (like cabinet doors and drawers).
Key Insights and Ideas
- Joint Updates are Essential: One of the "aha" moments in the paper is that you can't just fix the human to match the object. Sometimes, the most natural way to fix a contact error is to move the object slightly toward the hand. CoRelate’s refiner is trained to decide which one to move.
- Relations vs. Proximity: Simply telling a model to "be close to the object" isn't enough. By defining a "hold" phase where the hand must maintain a fixed relative pose to a specific part (like a drawer handle), the motion becomes much more stable.
- Articulated Objects: The framework treats a drawer handle or a fridge door as a "part" with its own local coordinate system. This allows the same math to work for picking up a box (rigid) or pulling a lever (articulated).
Key Results
The researchers used a metric called Complete Interaction Success (CIS), which measures the percentage of outputs that satisfy the task goal, maintain proper contact, and look physically plausible.
- Rigid Objects: On the FullBodyManipulation dataset, CoRelate achieved a 71.2% CIS, a massive jump from the previous best baseline (MaMi-HOI) at 36.8%.
- Articulated Objects: On the ParaHome dataset, CoRelate reached 36.0% CIS, more than doubling the performance of specialized baselines like SK-HOI (16.2%).
Qualitatively, the model produces much more convincing "pulling" and "carrying" motions where the hand stays glued to the interaction point throughout the entire movement (see Figure 3).
Limitations and Future Work
While CoRelate is a significant step forward, the authors note a few areas for improvement:
- Fixed Relations: Currently, the "who, what, and when" of the interaction are determined early in the process. The model can't easily decide to "regrasp" an object if the initial plan becomes awkward.
- Physics: The model is primarily geometric. It doesn't explicitly calculate contact forces or friction, which means it might occasionally generate motions that are geometrically perfect but physically impossible (e.g., lifting a very heavy object with one finger).
- Complexity: The current representation doesn't support complex multi-stage interactions, like picking up a key, putting it in a lock, and then turning it, within a single motion window.
Summary updated