In the standard reinforcement learning (RL) paradigm, an agent receives a reward signal after every action, like a coach providing constant feedback during a drill. However, real-world tasks rarely work this way. Whether it is a robot navigating a complex building, a model solving a multi-step math problem, or an agent browsing the web, the "reward" usually only appears at the very end—a binary signal of success or failure.
This creates the credit assignment problem: when an agent takes 1,000 steps and finally succeeds, which of those 1,000 steps actually caused the success? Without a way to distribute that final reward back to the intermediate actions, learning becomes incredibly inefficient, as the agent must stumble upon the exact sequence of correct moves by pure chance.
Decomposing the Trajectory
The most direct way to solve reward sparsity is to break a single "trajectory-wise" reward into smaller, per-step proxies. Tengyu Xu, Yingbin Liang, and Yue Wang introduced PARTED, an offline RL algorithm that mathematically decomposes a single end-of-episode score into per-step rewards. By using a pessimistic penalty to handle uncertainty in the data, they show that agents can learn efficiently even when the environment never provides a mid-task signal.
Similarly, the RESLOPE algorithm, developed by Hal Daumé III and John Langford, automatically estimates "residual" losses for each step. Instead of waiting for the final outcome to update the policy, the model predicts how much each action contributes to the final loss, effectively performing its own reward shaping.
Retrospective Criticism and Hindsight
If an agent cannot get feedback during the task, it can still learn by looking back at what it just did. This is the intuition behind the "retrospective critic" in CriticSearch. Yaocheng Zhang and colleagues use a frozen Large Language Model (LLM) to review a completed reasoning path. By comparing intermediate steps against the known "gold" answer, the critic provides dense, turn-level rewards that tell the agent exactly where it went off the rails.
This "hindsight" approach is also central to STIR², where Fabien Moutarde proposes relabeling an agent's own successful episodes as expert demonstrations. By treating past successes as a curriculum, the agent learns to bridge the gap between its current behavior and the sparse goal.
Internal Guidance and Auxiliary Signals
When external rewards are missing, agents can generate their own internal signals to guide exploration. Joshua Achiam demonstrated that Surprise-Based Intrinsic Motivation allows agents to solve tasks like Venture by rewarding them for reaching states that are difficult to predict. This "curiosity" keeps the agent moving through the environment until it eventually bumps into the sparse external reward.
Another approach is to provide the agent with a "sense of the world" that doesn't depend on the final goal. Kunal Pratap Singh and researchers at AI2 introduced Scene Graph Contrastive (SGC) Learning, which uses the structural relationships between objects in a room as an auxiliary supervisory signal. By learning to predict the layout of its environment, the agent develops a representation that makes navigating toward a sparse goal much easier.
Hierarchical Decomposition
Finally, complex tasks can be made less sparse by breaking them into a hierarchy. In Composite Task-Completion Dialogue Policy Learning, Baolin Peng and Lihong Li use Hierarchical Deep RL (HDRL) to manage multi-step tasks like travel planning. A high-level "manager" selects subgoals, and a low-level "worker" receives rewards for completing those subgoals. This transforms a single, distant goal into a series of immediate, achievable milestones, solving the credit assignment problem by design.