Clip-Level Cycle Consistency for Viewpoint-Robust Rewards from Third-Person Video

A robot that learns from video of an expert usually watches that video from a camera it does not share. One practical recipe trains a visual "task progress" score from expert videos and uses it as a reward for reinforcement learning. Each frame is mapped to an embedding, and the reward is how close the robot's current frame is to the embedding of a finished task. This works across different bodies, but it breaks when the camera moves. Camera pose is the single most damaging factor for visual imitation: in Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation, a real-world pick task fell from 91.7% to 45.8% success under a camera shift, while a background change only reduced it to 88.9%.

We argue that part of this failure is a timing problem disguised as a geometry problem. From a new camera angle, one still frame often cannot say which phase of the task it shows. A gripper moving toward the camera looks like one moving away. A reach looks like the retraction after the grasp. A short clip settles the question, because the direction of motion survives the change of view even when the appearance does not. The proposal is to learn progress embeddings from short clips rather than single frames. Clips from different, unpaired camera views are aligned in time with a cycle-consistency loss, and a time-reversal penalty forces the encoder to read motion direction instead of copying static scenery. The resulting reward is then tested on camera views never seen in training.

Core Claim

Frame-level progress rewards trained on several source viewpoints fail on a new viewpoint mainly because single-frame progress becomes many-to-one there. Several task phases project to similar images. Replacing frames with short clips, and adding an explicit penalty that separates a clip from its time-reversed copy, restores a single-valued progress estimate across views. That should yield RL success on held-out cameras that frame-level and adversarial methods do not reach. The claim can be falsified: if the gains come entirely from seeing more pixels (clip stacking) and not from the reversal term, or if the frame-level encoder already ranks phases correctly on held-out views, the hypothesis is wrong.

Relation to Prior Work

The method builds directly on XIRL: Cross-embodiment Inverse Reinforcement Learning. XIRL trains frame embeddings with Temporal Cycle-Consistency Learning (TCC). TCC matches each frame of one video to its nearest neighbor in another video of the same task, then requires the match to cycle back to the starting frame. XIRL uses the resulting embedding distance to the goal as a reward and generalizes to an unseen embodiment. Its authors report that it works best when camera viewpoints are consistent. We keep XIRL's reward recipe and change two things: the unit of embedding becomes a clip, and the held-out factor becomes the camera rather than the body.

Two lines of work motivate the temporal ingredient. Third Person Imitation Learning found that a discriminator given single frames cannot tell expert from novice when their positions coincide but their velocities are opposite, so it feeds frame pairs. Action Tubelet Detector for Spatio-Temporal Action Localization resolves "sitting down" versus "standing up" by regressing over six-frame stacks. A caution from He constrains the design: models given multiple frames often still fall back on static background shortcuts. That observation is why clip input alone is not our method, and why the reversal penalty exists.

Adversarial approaches are the main competitors and solve a different subproblem. Third Person Imitation Learning adds domain confusion, with an adversary that penalizes features revealing which camera was used. Domain Adaptive Imitation Learning learns an explicit mapping between domains. Third-Person Imitation Learning via Image Difference and Variational Discriminator Bottleneck (Student Abstract) subtracts frames to cancel static background and limits how much information the discriminator can use. These methods align distributions of views but impose no ordering of task phases. Domain confusion is also known to be sensitive to its weighting.

Generative view augmentation is a third route. View-Invariant Policy Learning via Zero-Shot Novel View Synthesis (VISTA) renders demonstrations from synthetic cameras. EgoDemoGen repairs reprojected video with a diffusion model. VISTA synthesizes frame by frame and names temporally consistent synthesis as open. Our method needs no generator at all, which is a cost advantage we must justify against VISTA's accuracy rather than assume.

Research Program

The setting is third-person reward learning. Expert videos come from several source cameras, without actions and without synchronization across views. A learner then runs reinforcement learning from a camera that never appeared in the expert data. Simulation gives exact control over camera pose and ground-truth task phase.

The encoder $\phi$ maps a clip $c_t = (o_{t-k}, \dots, o_t)$ of $k+1$ frames ending at time $t$ to an embedding. It is trained with the TCC loss computed between clips from different source cameras, plus a reversal term:

$$\mathcal{L} = \mathcal{L}_{\text{TCC}}(\phi) + \beta \, \mathbb{E}_t \left[ \max\left(0,\; m - \lVert \phi(c_t) - \phi(\bar{c}_t) \rVert \right) \right]$$

Here $\bar{c}_t$ is the same clip played backward, $m$ is a margin, and $\beta$ weights the term. The reversal term costs nothing to compute, since it needs no labels and no second view. It cannot be satisfied by static appearance, because a clip and its reverse contain identical pixels. The reward follows XIRL: the negative distance between the current clip embedding and the mean embedding of final clips. The clip length $k$ is swept from 1 (frame-level XIRL) up to about 8, the range where Action Tubelet Detector for Spatio-Temporal Action Localization found six frames best for disambiguating motion.

The analysis side is a phase-ambiguity probe. On held-out cameras, we take frames whose ground-truth arm states nearly coincide but whose task phases differ, such as approach versus retract, or motion toward versus away from the camera. We then measure whether the learned progress score orders them correctly. We also compute the rank correlation between learned progress and true phase along full held-out-view expert videos, and the reward's sign on time-reversed expert videos, which should decrease. These diagnostics test the mechanism directly, separate from RL noise.

Task selection matters for the argument. The hypothesis predicts gains on tasks whose trajectories revisit configurations or move along the camera's depth axis. It predicts little gain on tasks with monotonic, view-parallel motion. We therefore split tasks into these two groups in advance and report them separately.

Baselines

The naive reference is frame-level XIRL (XIRL) trained on the same source-view videos and evaluated on the held-out camera. A sparse true-success reward under the same RL budget is a lower bound, and XIRL evaluated in a source view is an upper reference. The file reports no XIRL numbers under viewpoint shift, so no reported result was found on Lacuna for XIRL on held-out cameras, and we will produce this number ourselves.

The strongest temporal-plus-viewpoint competitor is the frame-pair domain-confusion method of Third Person Imitation Learning, together with the image-difference variant with a capacity-limited discriminator from Third-Person Imitation Learning via Image Difference and Variational Discriminator Bottleneck (Student Abstract). Neither paper reports numeric success rates in the file. Third Person Imitation Learning describes linear decay as perspectives diverge, so no reported success number was found on Lacuna for either. Both get the same encoder, the same source views, and the same tuning budget for the confusion weight as we spend on $\beta$ and $m$.

The strongest augmentation competitor is VISTA (View-Invariant Policy Learning via Zero-Shot Novel View Synthesis). It reports 32% success versus about 10% for a single-view baseline on Robomimic Threading under camera shift. In the real world it reports 40% versus 10% full-task completion on cup-on-saucer across five unseen camera poses. We run it as VISTA-augmented expert videos fed to frame-level XIRL, with the same RL budget. Synthesis compute is reported separately.

Two ablations isolate the key ingredient. Clip-input XIRL without the reversal term tests whether more pixels alone explain any gain. Frame-level XIRL plus reversal is impossible by construction, so the second ablation instead removes cross-view pairing (TCC within a single view only) to test whether the reversal term works without alignment. Beating the field means higher held-out-camera RL success than all three competitors on the ambiguous task group. The clip-without-reversal ablation must also fall clearly short of the full method there.

Why not just…?

Why not just stack frames and let the network figure out motion? Stacking is exactly our no-reversal ablation, so the experiment answers this directly. The prior reason to doubt it is He's finding that multi-frame models still latch onto static correlations. MS-TCN likewise found raw feature stacking inferior to structured temporal signals. If stacking matches the full method on the phase probe and on RL, we drop the reversal term and report stacking as the answer.

Why not just use heavy 2D augmentation such as random crops? The file shows this is insufficient for real viewpoint change. Cropping helped textures and distractors in Decomposing the Generalization Gap, but camera shift remained the largest drop. VISTA's single-view baseline stayed near 10% on Threading. We nonetheless apply crops to every method, so that any gain is measured on top of augmentation rather than in place of it.

Why not just synthesize more views with VISTA? This is the real threat. Novel-view synthesis attacks the geometric gap directly and has reported numbers. Our counter-hypothesis is that per-frame synthesis cannot fix phase ambiguity, because ambiguity exists even in perfectly rendered single frames. Flicker in synthesized frames may also corrupt cycle alignment. The experiment settles it in two ways. If VISTA plus frame-level XIRL matches our method on the ambiguous task group, the temporal story is wrong. We also run VISTA plus our clip encoder, which tests whether the two are complementary.

Why not just train on many more source cameras? More views may make frame-level embeddings invariant enough on their own. We sweep the number of source cameras for both frame-level XIRL and our method. The prediction is that the gap persists on ambiguous tasks even as views grow, because adding views does not make a single frame reveal motion direction.

Proposed Experiments

The primary testbed is Factor World ( Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation), using its camera-pose factor across its Meta-World-based tasks. Expert videos come from a set of training camera poses. Rewards are evaluated with SAC on interpolated and extrapolated held-out poses, and results are split into the pre-registered ambiguous and monotonic task groups. A secondary testbed reuses the modified Pointmass and Reacher third-person tasks from Third Person Imitation Learning, where the adversarial baselines were designed. We also rerun X-MAGICAL from XIRL to confirm that clip embeddings do not hurt cross-embodiment transfer. Robomimic Threading under camera shift provides a policy-level comparison point on VISTA's own ground, where 32% is the reported figure to beat.

The decisive check is the phase-ambiguity probe on held-out cameras, run before any RL. If frame-level XIRL already orders ambiguous phases correctly, the premise fails, and we stop and report that the viewpoint gap is purely geometric. If the probe shows the predicted ordering errors for frames and their correction by reversal-trained clips, the RL results test whether the fix matters for control. Three outcomes would together support the claim: held-out-camera success that beats all baselines on ambiguous tasks, little difference on monotonic tasks, and a no-reversal ablation that lands with frame-level XIRL. All results use matched encoders, matched source data, five or more seeds, and equal hyperparameter search budgets per method.