Generative Trajectory Augmentation for Robust Policy Learning

275 concepts204 papers3 proposals

In traditional machine learning, data augmentation is straightforward: you can flip an image, inject noise into an audio clip, or swap words with synonyms. However, for embodied agents navigating physical spaces or robotic arms manipulating objects, data consists of trajectories—highly correlated, sequential dependencies of states and actions.

If you inject random noise into a trajectory, you break the underlying physics. The transition from state $s_t$ to $s_{t+1}$ via action $a_t$ must obey the laws of the environment. To overcome data scarcity without violating physical reality, researchers use generative trajectory augmentation. Instead of perturbing individual data points, these techniques synthesize entirely new, physically and causally coherent sequences.

Bridging the Distribution Gap in Offline RL

Offline reinforcement learning relies on static datasets, which often suffer from narrow coverage. If an agent encounters a state even slightly outside its training distribution, its value estimates can degrade catastrophically.

To expand dataset coverage safely, sequence models like Transformers can be trained directly on historical data to "hallucinate" valid paths. As shown by Kerong Wang et al. in Bootstrapped Transformer for Offline Reinforcement Learning, generating high-quality synthetic trajectories effectively converts out-of-distribution regions into in-distribution data. This sequence-level generation allows the model to respect environment physics far better than traditional noise-based augmentations.

Alternatively, agents can use generative world models to simulate entire rollouts. Marc Rigter demonstrates in World Models via Policy-Guided Trajectory Diffusion that trajectory-level diffusion can generate coherent sequences of states and actions guided by the agent's current policy. This approach bypasses the compounding errors that plague step-by-step autoregressive world models.

Furthermore, Mianchu Wang et al. in GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models use model-based reanalysis to "stitch" together states and goals from entirely different recorded sequences. This cross-trajectory synthesis allows agents to learn from transitions and shortcuts that were never explicitly demonstrated in the original data.

Standard Augmentation:  [State 1] -> [State 2 + Noise (Broken Physics)] -> [State 3]
Trajectory Synthesis:  [State 1] -> [Generative Model / World Model]   -> [Valid State 2]

Multimodal Alignment: Pairing Actions with Intent

For tasks like Vision-and-Language Navigation (VLN), data scarcity is a double-edged sword: agents require both realistic spatial trajectories and corresponding natural language instructions.

The core intuition behind solving this bottleneck is to decouple the collection of environmental paths from human annotation. By using generative "speaker" models, researchers can automatically label raw, unannotated trajectories with synthetic instructions. Daniel Fried et al. pioneered this in Speaker-Follower Models for Vision-and-Language Navigation, demonstrating that training a navigation agent on hallucinated instructions for unlabeled paths significantly improves its ability to ground language in visual landmarks.

This paradigm scales remarkably well. Zun Wang et al. leveraged this concept in Scaling Data Generation in Vision-and-Language Navigation to synthesize millions of instruction-trajectory pairs across photo-realistic environments, effectively closing the performance gap between seen and unseen spaces. To ensure these synthetic pairs remain effective, modern generators incorporate explicit 3D spatial relationships (Muraleekrishna Gopinathan et al. in Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation) or utilize adversarial frameworks to maximize instruction diversity (Niyati Rawal et al. in AIGeN: An Adversarial Approach for Instruction Generation in VLN).

Counterfactual Hallucination and Recovery

Why wait for an agent to fail in the real world when you can hallucinate the failure and its recovery beforehand? A major strength of synthetic trajectory generation is its ability to teach robustness through counterfactual scenarios.

Consider a robot navigating a crowded space. Instead of collecting endless hours of real-world collision data, Zizhao Wang in Dyna-LfLH: Learning Agile Navigation in Dynamic Environments from Learned Hallucination proposes a self-supervised pipeline that takes a robot's successful open-space movements and hallucinates moving obstacles around them. By framing the past path as an optimal avoidance maneuver, the agent learns complex navigation behaviors from minimal unconstrained robot motion.

Similarly, when a student policy makes an error during training, generative models can synthesize expert recovery paths starting precisely from that failure point. Michelle Torok demonstrates this rollout efficiency in Greedy-DAgger - A Student Rollout Efficient Imitation Learning Algorithm, allowing agents to learn recovery behaviors from simulated expert demonstrations without requiring physical environment resets.

Furthermore, the authors of IntervenGen show that automated synthesis of corrective trajectories from a small seed of human interventions provides dense coverage of failure modes that are difficult to capture manually, drastically increasing policy robustness.

Grounding Synthesis in Physical Invariants

To prevent generative models from hallucinating physically impossible behaviors, synthetic trajectories must be constrained by structural or kinematic invariants.

One approach is to bridge neural networks with physics simulators. Zhilu Zhang et al. demonstrate this in PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesis by building digital twins from short videos and using physics simulators to generate thousands of counterfactual interaction sequences. This allows highly accurate dynamics models to be trained on deformable objects like cloth without extensive real-world handling.

In robotic manipulation, kinematic retargeting serves as a powerful constraint. Jinhan Li et al. demonstrate in OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation that by combining foundation vision models with object-aware retargeting, successful trajectories can be synthesized across randomized object positions. This allows a humanoid robot to generalize a manipulation skill to entirely new environmental configurations based on a single video demonstration.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_synthetic_trajectory_generation_66d5aefa,
  title = {synthetic trajectory generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-trajectory-generation/rdr_66d5aefa671f4c8c9f521a31c7700396},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_synthetic_trajectory_generation_66d5aefa_v1,
  title = {synthetic trajectory generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-trajectory-generation/rdr_66d5aefa671f4c8c9f521a31c7700396/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake