Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

Lena Krieger, Xuan Zhao, Zhuo Cao, Qin Wang, Hanno Scharr, Ira Assent

TL;DR

Instead of generating optimal behaviors from scratch, this paper introduces Counterfactual Transport Flows, a framework that refines existing low-performing trajectories by locally shifting them toward better alternatives in a latent space. By using source-conditioned flow matching on local preference pairs, the method achieves a highly controllable, conservative trade-off between behavior preservation and performance improvement in offline reinforcement learning.

What Problem Does This Paper Address?

In many real-world decision-making scenarios—such as clinical treatment planning, portfolio management, or robotics—we rarely want to throw away an existing proposal and generate a completely new strategy from scratch. Instead, the goal is to refine an initial candidate plan based on historical feedback.

Standard offline reinforcement learning (RL) typically approaches policy improvement globally, optimizing reactive mappings from states to actions or generating entirely new trajectories via diffusion models. However, these methods face two major hurdles:

  1. Over-extrapolation: They risk generating physically or behaviorally implausible trajectories that drift far from the support of the offline dataset.
  2. Lack of Counterfactual Reasoning: They don't explicitly show how a specific low-performing trajectory could have been locally modified to achieve a better outcome.

The authors tackle this gap by formulating offline policy improvement as an instance-level, source-conditioned trajectory refinement problem.

Methods

The proposed framework, Counterfactual Transport Flows, operates entirely in a learned latent trajectory space and consists of three main steps:

  1. Trajectory Encoding: Full sequences of states and actions are mapped into a latent space using a trained trajectory autoencoder.
  2. Local Preference Pair Construction: For any given low-feedback "source" trajectory ($z^-$), the method searches its immediate latent neighborhood (using $k$-nearest neighbors) for a "target" trajectory ($z^+$) that yielded a higher return. This creates a weak local preference pair $(z^-, z^+)$ grounded in actual historical data.
  3. Source-Conditioned Flow Matching: Instead of predicting the target in a single step, the model learns a continuous transport process. It trains a neural vector field to push the source trajectory toward the target along a linear interpolation path. Crucially, the vector field is conditioned on the source trajectory itself ($v_\theta(zs, s | z^-)$), ensuring the refinement direction is tailored to that specific instance.
Source-conditioned trajectory refinement in latent trajectory space.

At inference time, a user can input a low-performing trajectory and adjust a refinement strength parameter ($\alpha \in [0, 1]$). Integrating the vector field up to $\alpha$ yields a partially or fully refined latent state, which is then decoded back into a valid trajectory.

Datasets

The authors evaluated their method on standard offline RL benchmarks from the D4RL suite:

  • AntMaze: A long-horizon navigation task characterized by sparse rewards and highly heterogeneous trajectory quality.
  • MuJoCo (HalfCheetah): A continuous locomotion task featuring dense return feedback.

Key Insights and Ideas

  • Locality is Key to Conservatism: By restricting target selection to a small neighborhood ($k=3$), the model avoids making massive, unrealistic jumps across the data distribution. This keeps the refined trajectories behaviorally plausible and safely within the offline data support.
  • Instance-Conditioned vs. Reward-Conditioned: Unlike reward-conditioned generation (which asks "how do I get a reward of 100 from scratch?"), this method asks "how do I fix this specific trajectory to make it slightly better?". This yields highly targeted, conservative edits.
  • Trajectory-Level Interpretability: Because the model learns a continuous vector field, it traces an explicit path from the bad trajectory to the good one. This provides a "counterfactual" explanation of what the agent could have done differently to succeed.

Key Results

The method was compared against several baselines, including Nearest Improved Neighbor (directly replacing the source with its neighbor) and Non-local Flow Matching (removing the neighborhood constraint).

As shown in Table 1, the proposed local source-conditioned refinement achieved the best improvement-deviation trade-off across both domains:

Table 1. Feedback improvement and deviation comparison on D4RL trajectories. Feedback ∆measures improvement over the source trajectory, action and latent deviation measure conservatism. Source-conditioned local refinement (ours) achieves the best improvement-deviation trade-off, outperforming retrieval baselines on conservatism and non-local flow matching on both metrics.
  • AntMaze: The method improved the proxy feedback score by +69.44 while maintaining the lowest action deviation (1.41) and latent deviation (27.00). Non-local flow matching actually degraded performance (-11.48), proving that locality is essential for stable learning.
  • MuJoCo: It achieved a massive feedback boost of +1180.20 with exceptionally low deviation metrics compared to direct neighbor replacement.

The authors also demonstrated that adjusting the refinement strength $\alpha$ allows for a smooth, monotonic control over how aggressively the trajectory is modified:

Table 3. Effect of refinement strength α on improvement and conservatism. As α increases, feedback improvement (Feedback∆) and deviations (ActionDev, LatentDev) increase monotonically. Our main experiments use α = 1.0, k = 3.

Limitations and Future Work

While promising, the authors note several limitations and avenues for future research:

  • Neighborhood Selection Dependency: The performance relies heavily on $k$-NN retrieval in the latent space. If the latent space doesn't preserve meaningful behavioral structures, the retrieved pairs might be sub-optimal.
  • No Feasibility Guarantees: Transporting points through latent space does not mathematically guarantee that the decoded trajectory will perfectly satisfy environment dynamics.
  • Offline Proxy Evaluation: The method was validated using an offline proxy return predictor rather than closed-loop environment rollouts. Testing how well these refined trajectories execute in live environments remains an open next step.

Summary updated

Abstract

Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback. A key difficulty is improving observed behavior without extrapolating beyond what the offline data supports. We propose \emph{counterfactual transport flows}, a source-conditioned trajectory refinement framework for offline decision-making guided by world feedback. Given a low-feedback candidate trajectory, we construct local preference pairs from offline data by retrieving nearby trajectories in latent trajectory space with higher task-specific feedback, and use them as weak supervision for conservative refinement. The framework learns instance-specific refinement directions: at inference time, a refinement strength parameter controls how far the candidate trajectory is transported, enabling a trade-off between preserving the original behavior and applying stronger improvement. Experiments on D4RL benchmarks, including AntMaze and MuJoCo tasks, show that our method improves behavior from historical returns as world feedback, while providing interpretable trajectory-level refinement paths.
BibTeX
@inproceedings{krieger2026counterfactual,
  author = {Lena Krieger and Xuan Zhao and Zhuo Cao and Qin Wang and Hanno Scharr and Ira Assent},
  title = {{Counterfactual Transport Flows for Offline Conservative Trajectory Refinement}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=ZyezfuKZj0},
}
Figure