TL;DR
Instead of generating optimal behaviors from scratch, this paper introduces Counterfactual Transport Flows, a framework that refines existing low-performing trajectories by locally shifting them toward better alternatives in a latent space. By using source-conditioned flow matching on local preference pairs, the method achieves a highly controllable, conservative trade-off between behavior preservation and performance improvement in offline reinforcement learning.
What Problem Does This Paper Address?
In many real-world decision-making scenarios—such as clinical treatment planning, portfolio management, or robotics—we rarely want to throw away an existing proposal and generate a completely new strategy from scratch. Instead, the goal is to refine an initial candidate plan based on historical feedback.
Standard offline reinforcement learning (RL) typically approaches policy improvement globally, optimizing reactive mappings from states to actions or generating entirely new trajectories via diffusion models. However, these methods face two major hurdles:
- Over-extrapolation: They risk generating physically or behaviorally implausible trajectories that drift far from the support of the offline dataset.
- Lack of Counterfactual Reasoning: They don't explicitly show how a specific low-performing trajectory could have been locally modified to achieve a better outcome.
The authors tackle this gap by formulating offline policy improvement as an instance-level, source-conditioned trajectory refinement problem.
Methods
The proposed framework, Counterfactual Transport Flows, operates entirely in a learned latent trajectory space and consists of three main steps:
- Trajectory Encoding: Full sequences of states and actions are mapped into a latent space using a trained trajectory autoencoder.
- Local Preference Pair Construction: For any given low-feedback "source" trajectory ($z^-$), the method searches its immediate latent neighborhood (using $k$-nearest neighbors) for a "target" trajectory ($z^+$) that yielded a higher return. This creates a weak local preference pair $(z^-, z^+)$ grounded in actual historical data.
- Source-Conditioned Flow Matching: Instead of predicting the target in a single step, the model learns a continuous transport process. It trains a neural vector field to push the source trajectory toward the target along a linear interpolation path. Crucially, the vector field is conditioned on the source trajectory itself ($v_\theta(zs, s | z^-)$), ensuring the refinement direction is tailored to that specific instance.
At inference time, a user can input a low-performing trajectory and adjust a refinement strength parameter ($\alpha \in [0, 1]$). Integrating the vector field up to $\alpha$ yields a partially or fully refined latent state, which is then decoded back into a valid trajectory.
Datasets
The authors evaluated their method on standard offline RL benchmarks from the D4RL suite:
- AntMaze: A long-horizon navigation task characterized by sparse rewards and highly heterogeneous trajectory quality.
- MuJoCo (HalfCheetah): A continuous locomotion task featuring dense return feedback.
Key Insights and Ideas
- Locality is Key to Conservatism: By restricting target selection to a small neighborhood ($k=3$), the model avoids making massive, unrealistic jumps across the data distribution. This keeps the refined trajectories behaviorally plausible and safely within the offline data support.
- Instance-Conditioned vs. Reward-Conditioned: Unlike reward-conditioned generation (which asks "how do I get a reward of 100 from scratch?"), this method asks "how do I fix this specific trajectory to make it slightly better?". This yields highly targeted, conservative edits.
- Trajectory-Level Interpretability: Because the model learns a continuous vector field, it traces an explicit path from the bad trajectory to the good one. This provides a "counterfactual" explanation of what the agent could have done differently to succeed.
Key Results
The method was compared against several baselines, including Nearest Improved Neighbor (directly replacing the source with its neighbor) and Non-local Flow Matching (removing the neighborhood constraint).
As shown in Table 1, the proposed local source-conditioned refinement achieved the best improvement-deviation trade-off across both domains:
- AntMaze: The method improved the proxy feedback score by +69.44 while maintaining the lowest action deviation (1.41) and latent deviation (27.00). Non-local flow matching actually degraded performance (-11.48), proving that locality is essential for stable learning.
- MuJoCo: It achieved a massive feedback boost of +1180.20 with exceptionally low deviation metrics compared to direct neighbor replacement.
The authors also demonstrated that adjusting the refinement strength $\alpha$ allows for a smooth, monotonic control over how aggressively the trajectory is modified:
Limitations and Future Work
While promising, the authors note several limitations and avenues for future research:
- Neighborhood Selection Dependency: The performance relies heavily on $k$-NN retrieval in the latent space. If the latent space doesn't preserve meaningful behavioral structures, the retrieved pairs might be sub-optimal.
- No Feasibility Guarantees: Transporting points through latent space does not mathematically guarantee that the decoded trajectory will perfectly satisfy environment dynamics.
- Offline Proxy Evaluation: The method was validated using an offline proxy return predictor rather than closed-loop environment rollouts. Testing how well these refined trajectories execute in live environments remains an open next step.
Summary updated