Standard Reinforcement Learning (RL) relies on a simple feedback loop: an agent takes an action and receives a reward signal $r$. In the offline setting, where agents learn from static datasets rather than active exploration, this dependency becomes a significant bottleneck. While we have access to massive logs of robot interactions, human videos, and text corpora, these datasets are often "reward-blind"—they record what happened, but not how "good" it was.
As Springenberg et al. note, obtaining dense reward annotations for every frame of a massive dataset is a major scalability barrier. The research direction emerging from these papers focuses on bridging this gap, enabling agents to extract meaningful behaviors from unlabeled trajectories.
The Reward Labeling Bottleneck
In many real-world domains, such as healthcare or robotics, environment interaction is risky, and manual reward engineering is prohibitively expensive. Yu et al. highlight that even when large-scale logs are available, the lack of task-specific labels prevents standard RL algorithms from distinguishing between successful and unsuccessful behaviors.
One surprisingly effective baseline, proposed by Yu et al., is Unlabeled Data Sharing (UDS): simply labeling all unlabeled data with a reward of zero. This works because it allows the agent to learn the underlying transition dynamics of the environment from the unlabeled data, effectively reducing sampling error even if the reward signal is biased.
Inferring Rewards from Sparse Signals
When explicit rewards are missing, they can often be reconstructed from auxiliary information. Several approaches focus on "auto-labeling" the unlabeled transitions:
- Optimal Transport: Luo introduces a method to align unlabeled offline data with a few expert demonstrations using Optimal Transport. By measuring the similarity between the unlabeled snippets and the expert's path, the system can generate intrinsic rewards that guide the agent toward expert-level performance.
- Graph Propagation: The TRAIN framework, proposed by Qu et al., treats state-action pairs as nodes in a graph. By propagating rewards from a tiny set of human-annotated samples to the rest of the dataset, the agent can learn a policy in "reward-blind" scenarios.
- Preference Learning: Instead of absolute rewards, Dragan et al. and Sestini et al. use human preferences to rank trajectories. This creates a learnable reward signal that can be used to "stitch" together unrelated data snippets into novel, successful behaviors.
- LLM-as-a-Reward-Generator: Recent work by Lyu et al. leverages Large Language Models to generate reward code for unlabeled trajectories, providing the mathematical scorecard that offline RL agents otherwise lack.
Unsupervised Pretraining and Skill Discovery
A more radical approach is to remove the need for rewards during the initial learning phase entirely. The intuition is that an agent can learn how the world works (dynamics) and what behaviors are possible (skills) before ever being told what to do (the task).
Xie et al. propose the Pretrained Decision Transformer (PDT), which conditions actions on latent representations of the future rather than rewards. By learning to predict and reach future states, the model undergoes large-scale unsupervised pretraining that generalizes to new tasks. Similarly, the authors of Foundation Policies with Hilbert Representations treat the environment's temporal structure as a geometric space, allowing agents to learn a library of behaviors that can be "prompted" zero-shot.
For robotics, Mezghani et al. and Risi et al. focus on self-supervised reward shaping and skill discovery. By extracting "skill atoms" from unstructured data, agents can solve complex, long-horizon tasks that standard RL fails to handle (Asadi).
Trajectory Stitching and Generative Augmentation
Unlabeled data is often fragmented. Trajectory stitching is the process of combining these fragments to solve a task the agent never saw in its entirety. Singh demonstrated that model-free offline RL can automatically chain behaviors—like clearing an obstruction before opening a drawer—even if those actions were never performed together in the training data.
To make this stitching more robust, researchers are turning to generative models. Lee et al. use diffusion models to refine the transitions between stitched segments, maximizing state-space coverage. Mazoure et al. show that conditional diffusion models can even estimate value functions directly from state sequences, bypassing the need for action labels or rewards entirely.
By treating unlabeled data not as "incomplete" but as a rich source of environmental dynamics and behavioral primitives, this research direction enables RL to scale beyond the constraints of human annotation.
Key Foundational Work
Sequence Prediction with Unlabeled Data by Reward Function Learning (2017) — contributed 1 concept elements to this direction.
Reinforcement Learning from Imperfect Demonstrations (2018) — contributed 1 concept elements to this direction.