When Internal Feedback Goodharts: SAE Rewards Fail to Improve Robot Success

RLxF @ ICML 2026 · OpenReview

TL;DR

Researchers found that while Sparse Autoencoders (SAEs) can extract "interpretable" features from robot policies that correlate with success, using those features as a reward signal to pick actions actually makes the robot perform worse. This study serves as a concrete warning of Goodhart’s Law: an internal feature that looks like a great diagnostic tool can become a misleading target once you start optimizing for it.

What Problem Does This Paper Address?

In robotics, "world feedback" (knowing if you actually finished the task) is usually sparse and delayed. A robot might perform thousands of tiny movements before finally getting a single "success" bit. To speed up learning, researchers often look for "dense" proxy signals—internal variables that provide constant feedback on whether the robot is on the right track.

Recently, Sparse Autoencoders (SAEs) have become a popular tool for "peeking inside" AI models. They decompose complex neural activations into sparse, human-understandable features. If an SAE finds a feature that consistently fires when a robot is successfully grabbing a cup, it’s incredibly tempting to use that feature as a reward signal: "Just do whatever makes the 'grabbing' feature go up!"

The authors of this paper test whether this tempting shortcut actually works or if it falls victim to Goodharting—where optimizing for a proxy measure destroys its relationship with the actual goal.

Methods

The researchers used Octo-Base, a large-scale "diffusion policy" for robotics. Diffusion policies are unique because they generate actions through a denoising process, making them a perfect testbed for "reranking."

  1. Feature Discovery: They trained SAEs on the internal activations of the Octo model. They identified specific features that correlated with task success (e.g., features that peak during the middle of a movement).
  2. No-Training Reranking: Instead of expensive fine-tuning, they used a "gate" test. At every step, the robot samples 8 different possible action chunks. It then uses an SAE-based reward function to "score" these chunks and executes the one with the highest score.
  3. The Controls: To ensure any improvement wasn't just due to luck, they compared the SAE-rewarded robot against:
  4. Native Octo: The original, un-intervened model.
  5. Random Chunk: Picking one of the 8 samples at random.
  6. Random Feature: Reranking based on a random set of internal features instead of the "interpretable" ones.

Datasets

The study was conducted in SimplerEnv, a simulated environment for robot manipulation.

  • Task: move_near (moving a robot arm near an object).
  • Evaluation: They focused on a "targeted slice" of 4 specific episode seeds where the native Octo model is known to succeed. This was a "deliberately favorable" setup—if the proxy reward couldn't even maintain success on easy tasks, it certainly wouldn't work on hard ones.

Key Insights and Ideas

  • Diagnostic Validity $\neq$ Control Validity: This is the paper's most important "aha moment." A feature can be a fantastic diagnostic (it predicts success when the robot is acting naturally) without being a valid control handle (it fails when you force the robot to maximize it).
  • The Proxy–Success Gap: The researchers successfully "optimized" the proxy. The SAE reward scores went up significantly (from -0.48 to 2.12). However, the actual success rate in the world plummeted. The robot was "gaming" its own internal features while failing the task.
  • Interpretability is a Trap: Just because a feature looks "semantic" (e.g., it seems to track the "grasping" phase) doesn't mean it captures the causal essence of the task. Under optimization, the robot might find "weird" ways to trigger that feature that have nothing to do with actually moving the arm correctly.

Key Results

  • Optimization worked, but success didn't: The SAE-feature reranker successfully picked chunks with much higher internal scores than random controls.
  • The Failure: Despite the high internal scores, the SAE-reranked robot only succeeded in 1 out of 4 episodes.
  • Matching the Noise: The SAE-reranker performed exactly as poorly as the random-chunk and random-feature controls (also 1/4 success).
  • The Baseline: The native Octo model succeeded in 4/4 cases, and a manual sampler (without reranking) succeeded in 3/4. Adding the "interpretable" SAE reward actually broke a working policy.

Limitations and Future Work

The authors acknowledge this is a "workshop-scale" study with a small sample size (4 seeds). However, they argue that for a signal to be a robust reward, it shouldn't collapse so easily on tasks the robot can already solve.

Future Work: * Instead of using SAE features as a direct reward, could they be used as a regularizer or an auxiliary loss? * Can we develop "grounding" tests to filter out features that are prone to Goodharting before we use them for training? * The authors propose a reporting checklist for future RLxF (Reinforcement Learning from Feedback) research, requiring researchers to show that their proxy optimization actually beats random-feature controls.

Summary updated

Abstract

World feedback for robot learning is sparse and delayed, motivating dense proxy signals for filtering, credit assignment, or fine-tuning. Sparse autoencoders (SAEs) expose dense internal features in diffusion robot policies, but it is unclear whether such features are useful feedback or merely interpretable correlates. We report a controlled negative result testing this boundary in Octo-Base on SimplerEnv. SAE features show task and phase structure, native-state local action selectivity, and outcome association in a small pilot. They are also optimizable: a no-training action-chunk reranker can select candidates with much higher SAE reward than random chunk controls. However, this internal reward does not improve closed-loop task success. On four episode seeds where native Octo succeeds in all cases and our manual sampler succeeds in three of four, SAE-feature reranking succeeds in only one of four, matching random-chunk and random-feature reranking. This proxy–success gap is the main contribution: sparse policy features can be valid diagnostics and optimizable proxies while still failing as grounded world feedback under naive optimization.
BibTeX
@inproceedings{2026when,
  title = {{When Internal Feedback Goodharts: SAE Rewards Fail to Improve Robot Success}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=h0uehSJqbm},
}
Figure