Abstract
Reinforcement learning (RL) promises to solve long-horizon tasks even when training data contains only short fragments of the behaviors. This experience stitching capability is often viewed as the purview of temporal difference (TD) methods. However, outside of small tabular settings, trajectories never intersect, calling into question this conventional wisdom. Moreover, the common belief is that Monte Carlo (MC) methods should not be able to recombine experience, yet it remains unclear whether function approximation could result in a form of implicit stitching. The goal of this paper is to empirically study whether the conventional wisdom about stitching actually holds in settings where function approximation is used. We empirically demonstrate that Monte Carlo (MC) methods can also achieve experience stitching. While TD methods do achieve slightly stronger capabilities than MC methods (in line with conventional wisdom), that gap is significantly smaller than the gap between small and large neural networks (even on quite simple tasks). We find that increasing critic capacity effectively reduces the generalization gap for both the MC and TD methods. These results suggest that the traditional TD inductive bias for stitching may be less necessary in the era of large models for RL and, in some cases, may offer diminishing returns. Additionally, our results suggest that stitching, a form of generalization unique to the RL setting, might be achieved not through specialized algorithms (temporal difference learning) but rather through the same recipe that has provided generalization in other machine learning settings (via scale).
Introduction
In reinforcement learning, "experience stitching" refers to the ability to piece together fragments of behavior to solve complex tasks. For example, if an agent learns to navigate from point A to B and separately from B to C, stitching allows it to infer the path from A to C without ever seeing the full sequence.
Traditionally, this capability has been attributed exclusively to Temporal Difference (TD) methods because they use bootstrapping—updating current estimates based on future predictions. Monte Carlo (MC) methods, which learn from full returns, were thought to be incapable of this. However, this assumption relies on trajectories actually intersecting at exact states. In high-dimensional or continuous spaces, trajectories almost never visit the exact same state twice.
As illustrated above, if trajectories do not literally intersect, "stitching" must rely on generalization through function approximation rather than simple tabular updates. This paper investigates whether the TD inductive bias is truly necessary or if neural network scale is the primary driver of stitching.
Defining Stitching Regimes
To isolate the effects of stitching, the researchers defined three distinct regimes of generalization:
- Standard Generalization: Learning to reach a new goal $g'$ from a new state $s'$.
- Exact Stitching: Recombining trajectories that share a specific, identical waypoint state $w$.
- Generalized Stitching: Recombining trajectories that pass through similar regions of the state space ($w$ and $w'$) but do not overlap exactly.
Methods and Benchmarks
The study compared TD-based methods (e.g., DQN) against MC-based baselines (e.g., Contrastive RL) using a minimalist pick-and-place grid benchmark. Two primary settings were used:
- The Quarters Setting: The agent learns to move boxes between adjacent quadrants and must stitch these behaviors to move a box to a diagonal quadrant at test time.
- The Few-to-Many Setting: The agent is trained on moving 1 or 2 boxes and tested on moving 3 or 4, requiring the composition of multiple sub-behaviors.
Key Insights
MC Methods Can Stitch
Contrary to long-standing dogma, MC methods successfully performed stitching. They achieve this implicitly through the neural network's representation: because the network maps similar states to similar values, the "value" of a goal bleeds over to nearby starting points of other trajectories, enabling a form of generalized stitching.
Scale is the Real Hero
The research found that the performance gap between a small TD model and a large TD model was significantly larger than the gap between TD and MC algorithms. Increasing the capacity of the "critic" (the value-estimating component) effectively reduced the generalization gap for both algorithm types. In some MC methods, scaling the architecture was the difference between a 20% and an 80% success rate.
The Brittleness of TD
While TD methods excelled at "Exact Stitching" in simple scenarios, their performance often collapsed as task complexity increased (e.g., increasing the number of objects). In the 4-box Quarters task, DQN's success rate dropped to 0% despite high training performance, suggesting that the traditional advantages of TD may diminish in complex, high-dimensional environments.
Conclusion
The findings suggest that experience stitching—a form of generalization unique to RL—might not require specialized temporal difference algorithms. Instead, stitching can be achieved through the same recipe that has driven progress in other machine learning fields: model scale and high-capacity function approximation. This challenges the necessity of the TD inductive bias in the era of large-scale reinforcement learning.