Back to the Future: Predicting Long-Horizon Dynamics with Retrospective Correction for Disturbance-Aware Retrieval

Jiwoong Choi, Haoyu Li, Yiwei LyuarXiv 2026 · arXiv

TL;DR

DARE (Disturbance-Aware REtrieval) is a new framework that allows robots to retrieve objects from dense clutter by intentionally—and safely—interacting with surrounding items. By combining a geometry-aware graph neural network with a novel "retrospective correction" mechanism, DARE can predict complex contact cascades and fix long-horizon simulation drift using sparse, noisy observations.

What Problem Does This Paper Address?

Imagine trying to grab a specific bottle of juice from the back of a packed refrigerator. You can’t lift it over the other items because of the shelf above, so you have to slide it out, inevitably bumping into other jars and cartons. If you aren't careful, a small nudge to a mustard jar might tip over a delicate wine glass nearby.

For robots, this is a nightmare. Traditional "collision avoidance" planners simply give up if there's no clear path. "Direct retrieval" planners might reach the goal but leave a trail of toppled objects in their wake. While learned dynamics models can predict these interactions, they suffer from accumulated drift: small errors in predicting the first contact compound over time, making long-term plans unreliable. Furthermore, real-world sensor data is often sparse and noisy, making it difficult to simply "reset" a simulation to the current observed state without introducing physical inconsistencies.

Collision avoidance vs. DARE. (a) Collision avoidance fails. (b) Direct retrieval causes damage. (c) DARE plans a safe path through controlled contact.

Methods

The authors introduce DARE, which tackles these issues through three main components:

  1. Geometry-Aware Relational Dynamics: Instead of using slow, high-resolution meshes or overly simple bounding boxes, DARE decomposes objects into convex pieces and encodes them into compact descriptors. These descriptors are fed into a Propagation Network (a type of Graph Neural Network) that models how forces ripple through a chain of objects.
  2. Retrospective Correction: This is the core innovation. When the robot receives a new (potentially noisy) observation, it doesn't just overwrite its current state. Instead, a learned "corrector" looks back at the discrepancy between the prediction and the observation. It goes "back to the future" by amending an earlier predicted state and rerunning the dynamics forward. This ensures the resulting trajectory is physically consistent with the model's learned rules.
  3. Joint Predictor-Corrector Learning: The dynamics model and the corrector are trained together. This encourages the dynamics model to produce rollouts that are "correctable" and the corrector to suggest amendments that actually improve future predictions, not just fit the current data point.

These predictions feed into a Model Predictive Path Integral (MPPI) planner that balances the progress toward the target object against the predicted "disturbance" (displacement, tilt, and rotation) of bystanders.

The DARE pipeline: from geometry encoding to interaction graph propagation and disturbance-aware planning.

Datasets

The researchers evaluated DARE across three increasingly complex simulated environments:

  • Planar Puck Table: A 2D environment with five pucks used for basic dynamics validation.
  • RoboCasa Tabletop: 3D scenes with diverse household objects.
  • RoboCasa Kitchen Cabinets: The most challenging setup, featuring 13 different compartment geometries and varying levels of clutter (up to 20 objects).

The evaluation specifically included "unseen" objects—items the model never encountered during training—to test its ability to generalize based on geometry alone.

Key Insights and Ideas

  • The "Correctability" Objective: Training the model to be corrected is a powerful paradigm. By supervising the "rerolled" trajectory against a ground-truth future, the model learns to prioritize amendments that fix the root cause of an error rather than just smoothing over the symptoms.
  • Physics-Consistent Filtering: Simply snapping a simulation to a noisy observation often breaks the physics (e.g., objects might suddenly overlap). By amending a past state and rerunning the learned dynamics, DARE keeps the objects in a physically plausible configuration.
  • Geometry over Resolution: The paper demonstrates that compact descriptors derived from convex hulls are sufficient for predicting complex contact dynamics, avoiding the massive computational overhead of mesh-based Graph Neural Networks.

Key Results

DARE significantly outperformed existing baselines in both prediction accuracy and retrieval success:

  • Lower Error: In long-horizon (40-step) predictions, DARE achieved the lowest normalized terminal translation and rotation errors across all RoboCasa test sets.
  • Higher Success in Clutter: In "Hard" cabinet scenarios with 20 objects, DARE maintained a success rate of 0.90 (with clean observations), whereas Collision Avoidance dropped to 0.10 and Direct Retrieval to 0.20.
  • Noise Robustness: Even with significant observation noise (2cm position error and 13° rotation error), DARE's retrospective correction allowed it to maintain high success rates where "Direct Observation" updates failed due to noise injection.
Comparison of dynamics prediction under noise. DARE (bottom) maintains alignment with objects far better than uncorrected or directly updated models.

Limitations and Future Work

While DARE is a major step forward, the authors note some areas for growth:

  • Geometry Knowledge: The current system assumes the robot has access to the collision geometry of the objects (though it generalizes to new shapes). Integrating this with real-time vision-based geometry estimation would be a logical next step.
  • Tactile Feedback: While the model uses visual observations for correction, incorporating high-frequency tactile or force feedback during the "contact-rich" phases could further improve the precision of the retrieval.
  • Computational Scaling: While more efficient than mesh-based methods, the GNN's message-passing cost still scales with the number of objects, which might limit its use in extremely high-density environments (e.g., a bin of hundreds of small parts).

Summary updated

Abstract

Retrieving objects from densely packed storage requires anticipating how contact disturbs surrounding objects when limited clearance prevents lifting the target over clutter. However, contact can propagate beyond the target's immediate neighbors, while accumulated prediction errors make learned dynamics unreliable over long planning horizons. We introduce DARE (Disturbance-Aware REtrieval), a geometry-aware relational dynamics framework that uses retrospective correction to improve long-horizon prediction for disturbance-aware retrieval. DARE encodes compact object descriptors and predicts action-conditioned disturbance propagation through object-level message passing. When a sparse observation becomes available, a learned corrector uses the prediction discrepancy to amend an earlier predicted state and reruns the intervening dynamics, propagating the correction through the predicted contact chain. These predictions guide a receding-horizon planner that balances retrieval progress against displacement and rotation of surrounding objects, permitting necessary contact while discouraging harmful disturbance. Experiments in planar and RoboCasa environments evaluate long-horizon prediction, generalization, and robustness for disturbance-aware retrieval in dense clutter. See our project website: https://the-back-to-the-future.github.io
BibTeX
@misc{choi2026back,
  author = {Jiwoong Choi and Haoyu Li and Yiwei Lyu},
  title = {{Back to the Future: Predicting Long-Horizon Dynamics with Retrospective Correction for Disturbance-Aware Retrieval}},
  year = {2026},
  eprint = {2611.02073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02073},
}
Figure