Robots that learn from human video increasingly use an intermediate step. Instead of mapping pixels straight to motor commands, they first predict how points in the scene will move, and only then turn those predicted paths into actions. Every system of this kind assumes a clean camera. A real robot's camera gets water droplets, dust, mud, glare and fog, and the human videos it learns from were mostly filmed in good conditions. We propose to train the motion predictor on deliberately corrupted frames while supervising it with point paths extracted from the matching clean frames. Its loss has three parts. The first rewards accurate paths where a point is visible. The second lets the model report how uncertain each predicted path is, so it can signal low confidence wherever the lens hides the scene instead of guessing. The third asks it to mark which image regions are contaminated. The downstream controller then reads both the predicted paths and their uncertainties.
The bet is that point motion is the right place to put robustness. Lens dirt and weather change what pixels look like, but they do not change where a cup handle moves when it is pushed. A model asked for motion rather than appearance has a better-posed target. The classification–distortion–perception analysis supports this: On The Classification-Distortion-Perception Tradeoff proves that restoring a degraded signal cannot add task-relevant information beyond what the raw degraded signal already contains. If the controller needs motion, it is better to predict motion directly than to first repaint a clean image.
Core Claim
A track-prediction policy trained with clean-frame track supervision, uncertainty-weighted track loss and a contamination-mask head will keep most of its clean-camera success under lens soiling, rain and fog. It will also beat three alternatives under the same training budget: restoring frames before planning, augmenting the input with corruptions while keeping a plain regression loss, and end-to-end behavior cloning on corrupted frames. The gap should widen as the contaminated fraction of the image grows. The part that is new is the uncertainty and visibility structure of the loss and a controller that consumes it. Training on corrupted images is not new; ordinary augmentation already does that.
Where This Sits
The method we build on is Any-point Trajectory Modeling, or ATM (Any-point Trajectory Modeling for Policy Learning). ATM runs an off-the-shelf point tracker over unlabeled human and robot videos. It then trains a transformer to forecast future 2D paths of arbitrary image points from one frame and a language instruction. Finally, it learns a small controller that follows those paths, using only a few action-labeled demonstrations. ATM reports 39% success on LIBERO-Long versus 16% for behavior cloning. In human-to-robot transfer it reports 60–63% on a UR5 arm. It is about 500x faster at inference than video-diffusion planners.
ATM supervises tracks with plain regression and assumes the tracker sees clean frames. Our proposal changes the supervision and the controller's inputs, not the architecture.
Two other lines of work predict future pixels rather than tracks. Learning Universal Policies via Text-Guided Video Generation generates a whole future video and then infers actions from it. Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models generates a single subgoal image. Prediction with Action: Visual Policy Learning via Joint Denoising Process denoises future frames and actions together. These motivate our choice of representation in two ways. They inherit any corruption in the conditioning frame: UniPi tiles the current frame across all diffusion steps, so a soiled frame propagates into the whole plan. They are also slow, which rules out cascading them with a restorer.
The degradation literature supplies our tools. Let's Get Dirty: GAN Based Data Augmentation for Camera Lens Soiling Detection in Autonomous Driving synthesizes realistic opaque and semi-transparent soiling with pixel-exact masks. It reports that segmentation drops by roughly 22% on soiled lenses. We reuse its synthesis approach to produce paired clean and soiled frames with masks. RainSD and Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix supply rain and fog synthesis.
Two papers shape the loss design. Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving weights sensor streams by estimated uncertainty. RLNet: Adaptive Fusion of 4D Radar and Lidar for 3D Object Detection shows that randomly dropping a modality during training improves robustness, by about 1% 3D detection mean average precision (mAP). Our version applies this reasoning per point within one camera rather than per sensor.
Application of a Two-Dimensional Regression Network Algorithm Model Based on Local Constraints in Human Motion Recognition shows that a composite loss with structural terms stabilizes keypoint prediction under occlusion. That supports adding structure beyond plain regression, though its skeleton prior does not carry over to general objects.
Two papers constrain the proposal. Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal restores weather-degraded video and reaches 22.57 dB peak signal-to-noise ratio (PSNR) on unseen weather, against 21.57 dB for the next best method. It still needs about 1.2 seconds per frame. Its results also show that offline training on fixed synthetic weather generalizes imperfectly, so our evaluation must hold out degradation types. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning finds that naive supervised pretraining can hurt later adaptation. That is a warning that flooding training with corruptions could damage clean-condition performance, so we measure clean performance as well.
Research Program
Data. We pair every training frame from the human and robot videos with a corrupted copy. Corruptions come from a bank covering opaque soiling, semi-transparent droplets, rain streaks and fog, each at a sampled severity. Each copy comes with a mask of affected pixels. The point tracker runs only on the clean copy, which gives target paths that are reliable even where the corrupted copy hides the scene. For every query point and future step, we record whether it falls under the contamination mask.
Loss. The track transformer sees only the corrupted frame. For query point $i$ at future step $t$, it outputs a predicted position $\hat p_{i,t}$, a scale $\sigma_{i,t}$ and a contamination probability $\hat m_{i,t}$. It is trained with
$$ \mathcal{L} = \sum_{i,t} \left[ \frac{\lVert p_{i,t} - \hat p_{i,t} \rVert^2}{2\sigma_{i,t}^2} + \log \sigma_{i,t} \right] + \lambda_m \sum_{i,t} \mathrm{BCE}(\hat m_{i,t}, m_{i,t}) + \lambda_c \, \mathcal{L}_{\mathrm{clean}}. $$
The terms are as follows:
- $p_{i,t}$ is the clean-frame track.
- $m_{i,t}$ is the mask label, and BCE is binary cross-entropy.
- $\mathcal{L}_{\mathrm{clean}}$ is the same track loss computed on the clean frame, which keeps clean performance from eroding.
- $\lambda_m$ and $\lambda_c$ are weights.
The first term is a Gaussian likelihood with a learned scale. Where the model cannot see, it can inflate $\sigma$ rather than being forced into a confident wrong answer.
One limitation should be stated plainly. The predicted position is still a conditional mean, so it will blur under ambiguity, exactly as the plain-regression critique predicts. What changes is that the uncertainty is reported. If the experiments show that the controller needs several distinct hypotheses rather than one blurred one, we will replace the Gaussian with a small mixture.
Controller. The path-following controller from ATM is retrained to take $(\hat p, \sigma, \hat m)$ for each point. During training, its attention over points is randomly masked in proportion to $\hat m$. This is the per-point version of modality dropout: it teaches the controller to work from whichever points remain trustworthy.
Analysis. We will measure the following:
- how predicted $\sigma$ tracks true path error, as a calibration check;
- how success falls as contaminated area grows from 0% to roughly 60% of the image;
- whether the controller's reliance on high-$\sigma$ points actually drops.
Baselines
The naive reference is ATM exactly as published, trained on clean data and evaluated on corrupted frames. It shows how large the problem is. No reported result was found on Lacuna for ATM or any track or video policy under camera degradation, so this number has to be measured. The clean-condition anchor is ATM's reported 39% success on LIBERO-Long (Any-point Trajectory Modeling for Policy Learning), which our reimplementation must reproduce approximately before any comparison is trusted.
The strongest competitor is ATM trained on the same corrupted frames with the same clean-frame track targets, but with ATM's original plain regression loss and an unchanged controller. Call it augmented ATM. It isolates exactly our contribution, the uncertainty and mask terms and the uncertainty-aware controller, and a skeptical reviewer would expect it to capture most of the gain. It gets identical corruption banks, data, training steps and a matched hyperparameter search, for example eight configurations each. Beating it means statistically separated success rates across seeds on held-out degradation types, plus a gap that grows with contamination area. Matching it on light corruption alone would not count.
The pipeline alternative restores each frame with the video weather-removal model from Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal, with its test-time adaptation enabled, and then feeds clean-trained ATM. Its published figure is restoration quality: 22.57 dB PSNR on unseen weather at about 1.2 seconds per frame. We report task success both with the controller allowed to wait for restoration and under a matched real-time control budget.
Two further baselines complete the set. An end-to-end visual behavior-cloning policy trained on the same corrupted frames tests whether the track intermediate matters at all; ATM reports 16% on LIBERO-Long for behavior cloning on clean data. PAD-style joint frame-and-action denoising (Prediction with Action: Visual Policy Learning via Joint Denoising Process), trained on the same corrupted data, tests whether predicting pixels is more or less fragile than predicting tracks. PAD reports 72.5% average success on MetaWorld in clean conditions.
Three ablations each remove one ingredient: the uncertainty term (fixed $\sigma$), the mask head ($\lambda_m = 0$), and the controller's use of $\sigma$ and $\hat m$, where uncertainty is predicted but ignored.
Why not just…?
Why not just restore the frames and run an ordinary planner? The theory in On The Classification-Distortion-Perception Tradeoff says restoration cannot recover task information the degraded frame lacks. Restorers tuned for realism can also invent texture at exactly the edges a controller relies on. Latency compounds this: about 1.2 seconds per frame for Diff-TTA is far slower than a 10–50 Hz manipulation loop. The restore-then-ATM baseline tests this directly. If it wins when given unlimited time, we will say so and report the cost in time.
Why not just augment training with synthetic dirt and keep the original loss? This is the real threat, and augmented ATM is built to test it. If it matches our method on every held-out corruption and severity, the honest conclusion is that clean-target augmentation is sufficient, and the paper becomes a benchmark plus that finding. We expect the separation to appear at high contamination, where plain regression is forced into confident averaged paths. There, the calibration analysis and the controller-ignores-uncertainty ablation show whether uncertainty is doing the work.
Why not just use behavior cloning with heavy image augmentation? ATM already shows behavior cloning falling to 16% on LIBERO-Long and 0–30% in human-to-robot transfer under clean conditions. Separately, soiling cuts segmentation by about 22% (Let's Get Dirty: GAN Based Data Augmentation for Camera Lens Soiling Detection in Autonomous Driving). The augmented behavior-cloning baseline checks whether corruption narrows or widens the gap with track-based methods.
Proposed Experiments
The main testbed is LIBERO (Spatial, Object, Goal and Long suites), rendered with the corruption bank. One corruption family is held out per run: for example, train on opaque soiling, rain and fog, and test on semi-transparent droplets. This guards against the offline-generalization failure that Diff-TTA documents.
The cross-domain test follows ATM's human-to-robot protocol on BridgeData V2 and real Franka or UR5 arms. Tracks are pretrained on human video, including filtered Something-Something V2 clips, and evaluation uses a real camera with physically applied water spray and dust. This checks that synthetic soiling transfers to real optics, the case Sistu et al. flag as hardest for semi-transparent contaminants.
Metrics are as follows:
- task success rate across five or more seeds;
- success as a function of contaminated area;
- clean-condition success, to catch the harm LIBERO warns about;
- track error and $\sigma$ calibration, using clean-frame tracks as ground truth;
- control rate in Hz.
The decisive check is the comparison of success against contaminated area between our method and augmented ATM on held-out corruptions. If the two curves overlap, the proposed loss structure is not justified.
References
- On The Classification-Distortion-Perception Tradeoff
- Any-point Trajectory Modeling for Policy Learning
- Learning Universal Policies via Text-Guided Video Generation
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
- Prediction with Action: Visual Policy Learning via Joint Denoising Process
- Let's Get Dirty: GAN Based Data Augmentation for Camera Lens Soiling Detection in Autonomous Driving
- RainSD: Rain style diversification module for image synthesis enhancement using feature-level style distribution
- Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix
- Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving
- RLNet: Adaptive Fusion of 4D Radar and Lidar for 3D Object Detection
- Application of a Two-Dimensional Regression Network Algorithm Model Based on Local Constraints in Human Motion Recognition
- Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning