TL;DR
Researchers have found that reward-driven reinforcement learning (RL) and self-supervised learning (SSL) capture fundamentally different, "orthogonal" aspects of a system's state. By pretraining a world model with SSL to capture underlying system health and then training an RL policy on top to handle immediate control, we can create agents that are both high-performing and capable of critical downstream tasks like predictive maintenance.
What Problem Does This Paper Address?
In the world of Reinforcement Learning, there is a long-standing debate: What signal should shape an agent's internal "world model"?
One camp, famously summarized by the "Reward is Enough" hypothesis, argues that maximizing a reward signal is sufficient to force an agent to learn everything it needs to know about the world. The opposing camp argues that rewards are too sparse and task-specific; they believe agents should instead learn by predicting the "observation stream" (what they see and feel) in a self-supervised way.
This isn't just an academic argument. In safety-critical domains like aviation, we need agents that don't just "fly well" to get a high reward, but also understand the underlying health of the engine. If an agent's internal model only tracks what it needs to maximize immediate performance, it might discard vital information about long-term mechanical wear—information that human operators need for maintenance planning and anomaly detection.
Methods
The authors "arbitrate" this debate using TurboSens, a sophisticated turbofan engine simulator. Unlike standard benchmarks, TurboSens exposes the "ground truth" latent states of the engine:
- Wear ($s$): Slow-drifting, underlying mechanical health.
- Fouling ($c$): Fast-changing deposits (dust/dirt) that mask the wear signal but can be cleared by maintenance actions.
The researchers compared two primary approaches:
- PPO (Proximal Policy Optimization): A standard RL algorithm driven purely by reward.
- JEPA (Joint-Embedding Predictive Architecture): A self-supervised method that learns by predicting future latent representations from current ones, without any reward signal.
To "audit" what these models actually learned, they used linear probes. They froze the trained encoders and tried to reconstruct the ground truth (wear and fouling) using simple linear regression. If a simple linear model can find the information, it means the encoder has successfully captured and organized that feature in its latent space.
Datasets
The study utilizes the TurboSens environment (based on the OpenDeckSMR turbofan simulator).
- Scenario T2: A multi-region tier featuring 16 different operating contexts per step.
- Data Volume: Approximately $10^3$ episodes, each spanning 14,000 timesteps, recording sensor channels, maintenance actions, rewards, and the hidden ground-truth states.
- Comparison to existing benchmarks: Unlike the classic CMAPSS or N-CMAPSS datasets, TurboSens is interactive (the agent's actions affect the engine's state) and provides the latent "audit" states necessary for this research.
Key Insights and Ideas
- Orthogonal Representations: The most striking finding is that reward and SSL shape encoders along different axes. PPO gets very good at tracking fouling (because it needs to know when to clean the engine to get a reward) but discards wear. JEPA does the exact opposite: it recovers wear (because wear is a predictable drift) but is invariant to fouling.
- The "Collapse" of the Recurrent State: In reward-driven RL, the deeper you go into the neural network, the more information is discarded. The authors observed a "sharp collapse" where the LSTM hidden state throws away almost all information that isn't strictly necessary for the current policy, resulting in an $R^2$ near zero for the full latent state.
- Sequential Composition (The "Recipe"): The authors found that co-training (using both losses at once) is unstable because the signals compete. However, they compose "cleanly" if done sequentially: Pretrain with JEPA $\rightarrow$ Freeze the Encoder $\rightarrow$ Train PPO on top.
- Action-Relevance vs. System-Faithfulness: A model that is "good enough" to achieve a high reward is not necessarily a "faithful" model of the system. For safety-critical AI, "reward is enough" might be a dangerous assumption.
Key Results
- PPO Performance: On its own, PPO achieves near-perfect tracking of fouling ($R^2 \approx 0.97$) but fails to capture underlying wear.
- JEPA Performance: Standalone JEPA reaches an $R^2 \approx 0.51$ on wear (near the "oracle" ceiling of 0.55) but ignores fouling ($R^2 \approx 0.10$).
- The Winning Recipe: The "JEPA + PPO" recipe matched the operational return of end-to-end PPO while maintaining the ability to recover wear.
- Forecasting: Only the JEPA-based models were able to successfully forecast future engine wear ($R^2 \approx 0.30$ at a 250-step horizon). PPO-based models were "uniformly large negative," meaning they had no predictive power for long-term health.
Limitations and Future Work
- Cross-Task Flexibility: The authors noted a "cross-task caveat." In one specific scenario (the "wash penalty" task), the frozen JEPA encoder slightly underperformed end-to-end PPO. This suggests that freezing an encoder might limit the agent's ability to learn highly specific action timings required by certain reward functions.
- Substrate Specificity: The "Recipe" is designed for systems with a specific structure: a slow state of interest masked by a fast nuisance variable. While common (HVAC systems, batteries, agriculture), it may not apply to all RL problems.
- Other Signals: The paper focuses on reward vs. observation. Future work could explore how other signals—like sparse safety alarms or lagged economic reports—could further refine world models.
Summary updated