In the standard reinforcement learning (RL) paradigm, an agent learns by interacting with an environment and receiving a scalar reward signal. However, in many real-world applications—from robotics to healthcare—collecting online experience is risky, and the "ground truth" reward for a specific action is often unknown or expensive to annotate.
This has led to a surge in research focused on leveraging large-scale, unlabeled offline datasets. These datasets might be logs of past behavior, expert demonstrations without success markers, or even passive videos of humans performing tasks. The central challenge is a "data-scarcity paradox": we have an abundance of behavioral data but a lack of the reward signals necessary to optimize a policy.
The Bottleneck of Explicit Rewards
Traditional offline RL algorithms, such as Conservative Q-Learning (CQL), typically require every transition in a dataset to be paired with a reward. As Tianhe Yu and Yevgen Chebotar point out, obtaining these labels for thousands of interactions is often prohibitively expensive.
When rewards are missing, practitioners often fall back on Behavioral Cloning (BC), which simply mimics the data. However, Aviral Kumar and Sergey Levine demonstrate that offline RL can significantly outperform BC by "stitching" together suboptimal trajectories to find a path to a goal—provided a reward signal can be found or inferred.
Imputing Rewards from Sparse Labels
One intuitive solution is to treat reward estimation as a supervised learning problem. If we have a tiny fraction of labeled data, can we guess the rest?
- Reward Networks: Lijun Wu and Tao Qin proposed training a secondary "reward network" to estimate sequence quality, enabling RL to bridge the gap between scarce labeled data and plentiful unlabeled text.
- The 1% Rule: Andrew D. Bagdanov and Carlo Romeo showed that labeling just 1% of a dataset and using a neural network to "impute" the remaining 99% of rewards allows agents to achieve near-baseline performance.
- The Zero-Reward Strategy: Surprisingly, Tianhe Yu found that simply labeling all unlabeled data with a reward of zero (Unlabeled Data Sharing) often outperforms complex models. This works because it reduces sampling error, effectively telling the agent: "I don't know if this is good, but it's a valid way the world moves."
Learning from Preferences and Rankings
When absolute scalar rewards are unavailable, human preferences offer a powerful alternative. Instead of saying "this action is worth 0.75," a human (or a model) says "Trajectory A is better than Trajectory B."
Yinglun Xu and David Zhu introduced Binary Reward Labeling (BRL), which bridges preference-based RL and standard offline RL by labeling "chosen" steps as $+1$ and "rejected" steps as $-1$. Similarly, Alessandro Sestini developed TROFI, which learns to rank trajectories to create a learnable reward signal that often proves more stable than hand-engineered functions.
For more automated scaling, Sreyas Venkataraman and Yufei Wang demonstrated that Vision-Language Models (VLMs) like GPT-4o can act as "automatic reward annotators," providing preference labels for robotic tasks directly from visual data.
Unsupervised Pretraining and Latent Intentions
The most ambitious direction in this field is learning from data with no rewards and no actions—essentially "learning by watching."
The intuition here is that even without rewards, trajectories contain the "physics" of the world. By treating every state reached in a dataset as a potential goal, agents can learn a "sense of direction."
- Goal Chaining: Yevgen Chebotar and colleagues used "goal chaining" to stitch unlabeled trajectories together, allowing robots to perform complex manipulation skills without manual reward engineering.
- Intention-Conditioned Value Functions (ICVF): Dibya Ghosh and Sergey Levine developed frameworks to learn from passive data (like YouTube videos). By modeling the likelihood of future outcomes under latent "intentions," the agent learns representations that capture environment dynamics, which can then be fine-tuned for specific tasks.
- Future-Conditioning: The Pretrained Decision Transformer (PDT) conditions actions on latent representations of the future rather than explicit rewards. This allows the model to learn the structure of decision-making from raw trajectory data, matching the performance of supervised methods during downstream task adaptation.
Bridging the Action Gap
A final hurdle is that much of the world's data (like video) lacks the motor control signals (actions) used to generate it. Marvin Alles and Philip Becker-Ehmck addressed this with Latent Action World Models (LAWM), which map both labeled robot trajectories and unlabeled videos into a shared latent action space. This allows an agent to learn the "intent" of a movement from a video and translate it into a control signal it can actually use.
By moving away from a dependency on dense, manual labels, these methods allow RL agents to tap into the vast "dark matter" of unlabeled data, transforming passive observation into actionable intelligence.