TL;DR
RIFT (Reward Improvement Field Transfer) is a novel framework for cross-domain offline reinforcement learning that addresses dynamics mismatches between source and target environments. Instead of transferring raw data points, RIFT learns the "improvement field"—the transformation that turns low-reward behaviors into high-reward ones—and applies it to scarce target data to synthesize high-quality training samples.
What Problem Does This Paper Address?
In Offline Reinforcement Learning (RL), we aim to learn an optimal policy from a fixed dataset without further environment interaction. This is hard enough, but it becomes a nightmare when the data we have (the "source" domain, like a simulator) has different physics than where we want to deploy (the "target" domain, like a real-world robot). This is known as dynamics shift.
If you simply train on simulation data and run it on a real robot with different gravity or friction, the robot will likely fail because the same action $(a)$ in the same state $(s)$ leads to a different next state $(s')$. Existing methods try to "correct" these transitions or filter out the bad ones, but when target data is extremely scarce, there isn't enough information to model the new physics accurately. RIFT asks: What if we don't transfer the transitions themselves, but rather the "direction" of how to get better?
Methods
The core of RIFT is the Reward Improvement Field. The process follows four main steps:
- Shared Latent Encoding: The authors train a Variational Autoencoder (VAE) to map both source and target transitions into a shared latent space. They use a clever trick—reward-conditioned regularization—to ensure that the "direction" from low to high reward is aligned across both domains.
- Learning the Shift: In the source domain (where data is abundant), they identify low-reward and high-reward transitions. They use Optimal Transport (OT) to pair these up, essentially finding the most efficient way to "move" the low-reward distribution to the high-reward distribution.
- Fitting the Field: They fit a simple affine transformation (a linear map plus an offset) to these displacement vectors. This map represents the "improvement field"—a mathematical rule for how behavior improves in this task.
- Target Augmentation: They take the scarce, low-reward data available in the target domain ("anchors"), apply the source-learned improvement field to them in latent space, and decode the result. This generates synthetic, high-reward transitions that are "anchored" in the target domain's physics.
Finally, they use Implicit Q-Learning (IQL) to train a policy on this augmented dataset.
Datasets
The authors evaluated RIFT using the ODRL benchmark, which is built on the standard D4RL MuJoCo locomotion tasks (HalfCheetah, Hopper, Walker2d, and Ant). They tested three types of dynamics shifts:
- Gravity shifts: Reducing gravitational acceleration by 50%.
- Morphology shifts: Changing the physical size of body parts (e.g., shortening a thigh or lengthening a foot).
- Friction shifts: Halving the friction coefficients of the robot's joints and feet.
They used a massive source dataset (~1 million transitions) but limited the target data to just 5,000 transitions, simulating a "data-starved" real-world scenario.
Key Insights and Ideas
- Structure is more stable than Samples: While the exact $(s, a, s')$ tuple changes when gravity shifts, the relationship between a clumsy step and a graceful stride often remains similar. RIFT exploits this structural invariance.
- Target Anchoring vs. Source Transplanting: A major "aha moment" is that it’s safer to start with a real target transition (which obeys the correct physics) and "push" it toward a better state than to take a high-reward source transition and try to "fix" its physics.
- The Affine Assumption: The paper demonstrates that a relatively simple affine field is surprisingly effective at capturing improvement directions in dense-reward locomotion tasks.
Key Results
- State-of-the-Art Performance: RIFT achieved the highest aggregate normalized score (1318.7) across gravity and morphology shifts, representing a 26.2% improvement over the strongest baseline (STC).
- Morphology Mastery: RIFT was particularly dominant in morphology shifts, where changing body proportions usually breaks traditional transition-transfer methods.
- Data Efficiency: The method showed significant gains even when target data was extremely limited, though performance dropped when target data fell below 1,000 transitions (likely due to poor VAE representation learning).
- Better than Interpolation: RIFT outperformed standard data augmentation techniques like Mixup or SMOTE, proving that the source-derived "improvement logic" provides more value than simple data smoothing.
Limitations and Future Work
- Dense vs. Sparse Rewards: RIFT was tested on dense-reward locomotion. In environments with sparse or delayed rewards, "improvement" is harder to define locally, and the authors suggest that longer-horizon return labels might be necessary.
- Complexity of the Field: The current model uses an affine field. While effective here, more complex tasks might require non-linear improvement fields (e.g., learned via a neural network).
- Compatibility Checks: The method assumes a "shared class geometry." If the dynamics shift is so radical that the way one improves in the source is fundamentally different from the target, RIFT might struggle.
- Future Directions: The authors aim to extend this to manipulation tasks and explore how to detect when a source field is "incompatible" with a target domain to avoid negative transfer.
Summary updated