Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

Qinqing Zheng, Mikael Henaff, Amy Zhang, Aditya Grover, Brandon Amos

TL;DR

The authors introduce ONI, a distributed reinforcement learning framework that simultaneously trains an agent and learns an intrinsic reward function using online feedback from an asynchronous LLM server. By distilling LLM feedback on the fly from the agent's own experience, ONI eliminates the need for massive pre-collected offline datasets while scaling to billions of environment steps, matching or exceeding state-of-the-art performance on challenging NetHack tasks.

What Problem Does This Paper Address?

Reinforcement learning (RL) excels at optimization but notoriously struggles when rewards are sparse—such as in complex games or open-ended environments where an agent might wander for thousands of steps before receiving any feedback. To fix this, researchers use intrinsic rewards (extra reward signals) to guide exploration.

Recently, leveraging the vast prior knowledge of Large Language Models (LLMs) to automatically generate these rewards has become a hot topic. However, existing methods suffer from a major dilemma:

  1. They generate reward code directly, which struggles to capture complex, unstructured semantic features (like text logs or visual descriptions).
  2. They query an LLM at every single environment step, which is computationally impossible when scaling to the millions or billions of steps required for complex tasks.
  3. They rely on a complex, multi-stage offline process (like the prior state-of-the-art method, Motif) that requires a massive, pre-collected dataset of diverse agent experiences just to train a reward model before RL training even begins.

ONI tackles this bottleneck by integrating LLM feedback directly into the online training loop, allowing the agent to learn both the policy and the reward function concurrently from its own live experience.

Figure 1: Overview of Motif (Klissarov et al., 2023) and ONI.

Methods

The core architecture of ONI is built on top of Sample Factory's high-throughput Asynchronous Proximal Policy Optimization (APPO) framework.

To prevent the slow inference times of LLMs from bottlenecking the fast-paced RL environment workers, ONI decouples the components using an Asynchronous LLM Worker:

Figure 2: Overall system diagram of ONI. Our additions to Sample Factory are highlighted in blue. We added an asynchronously executing LLM server and learned reward function, and connect them back into the main learning process in a way that does not hurt the overall throughput of the policy and value learning.

As the agent explores, text captions of its observations are placed into a Last-In-First-Out (LIFO) queue. An independent worker sends these captions via HTTP/REST to a remote server running vLLM (hosting LLaMA-3.1-8B-Instruct).

Instead of using the LLM directly as the reward function, ONI distills this online feedback into a smaller, local reward model. The authors explore three different algorithmic flavors for this distillation:

  1. Retrieval: Hashes binary "helpful/unhelpful" LLM labels directly into a table. It is simple and hyperparameter-free but cannot generalize to unseen text.
  2. Classification: Trains a local binary classifier to predict the LLM's sentiment label from the observation, enabling generalization to similar, unseen captions.
  3. Ranking: Sends pairs of captions to the LLM for preference labeling and trains a ranking model using a Bradley-Terry preference model, adapted with running statistics to handle the continuously changing online dataset.

Datasets

The framework was evaluated on the NetHack Learning Environment (NLE), a notoriously difficult, procedurally generated terminal-based dungeon crawler. NetHack features extremely long horizons, sparse rewards, and highly complex game dynamics.

The authors tested ONI across four specific tasks:

  • Score: A dense reward task based on the in-game score.
  • Oracle: A sparse reward task requiring the agent to find a specific character deep in the dungeon.
  • StaircaseLvl3 & StaircaseLvl4: Sparse tasks requiring the agent to descend multiple levels.

They also evaluated the agents in a completely reward-free setting, tracking raw game progress metrics: experience level, dungeon level, gold collected, and unique locations explored (scout).

Key Insights and Ideas

  • Online Learning Eliminates Privileged Data: ONI proves that you don't need a pre-existing dataset collected by a top-tier agent to bootstrap an intrinsic reward model. Learning purely from the agent's own evolving, online experience is highly effective.
  • Generalization Saves Compute: In ablation studies where LLM annotations were heavily restricted (subsampled down to 1%), the parametric Classification model maintained strong performance, whereas the non-parametric Retrieval model plummeted. This highlights the value of distilling feedback into a model that can generalize.
  • Don't Deduplicate Preferences: For the ranking-based approach, deduplicating captions actually ruined the reward model's ability to learn. Keeping the natural distribution of data (including frequent, mundane messages like blank screens) is crucial for the model to accurately calibrate what events are truly rare and valuable.

Key Results

  • Matches Offline SOTA: ONI-classification successfully matched or closely tracked the performance of Motif across all sparse NetHack tasks, despite Motif requiring a pre-collected dataset from an agent already trained for 1 billion steps.
  • Outperforms in Reward-Free Settings: In completely reward-free environments, ONI-classification outperformed Motif on 3 out of 4 game progression metrics.
  • High System Efficiency: Despite adding an LLM into the loop, ONI retains 80-95% of Sample Factory's original throughput, achieving an average of 26k to 30k environment interactions per second on a single GPU setup.

Limitations and Future Work

  • Caption Dependency: ONI relies on the environment providing textual captions of observations. For purely visual environments (like Atari or robotics), it would require integrating an auxiliary image-captioning model.
  • Basic Queueing Strategy: The current architecture uses a simple LIFO queue to decide which observations to send to the LLM. The authors note that developing more sophisticated active learning or uncertainty-based prioritization schemes could make significantly better use of the limited LLM queries.

Summary updated

Abstract

Automatically synthesizing dense rewards from natural language descriptions is a promising paradigm in reinforcement learning (RL), with applications to sparse reward problems, open-ended exploration, and hierarchical skill design. Recent works have made promising steps by exploiting the prior knowledge of large language models (LLMs). However, these approaches suffer from important limitations: they are either not scalable to problems requiring billions of environment samples, due to requiring LLM annotations for each observation, or they require a diverse offline dataset, which may not exist or be impossible to collect. In this work, we address these limitations through a combination of algorithmic and systems-level contributions. We propose ONI, a distributed architecture that simultaneously learns an RL policy and an intrinsic reward function using LLM feedback. Our approach annotates the agent's collected experience via an asynchronous LLM server, which is then distilled into an intrinsic reward model. We explore a range of algorithmic choices for reward modeling with varying complexity, including hashing, classification, and ranking models. Our approach achieves state-of-the-art performance across a range of challenging tasks from the NetHack Learning Environment, while removing the need for large offline datasets required by prior work. We make our code available at https://github.com/facebookresearch/oni.
BibTeX
@inproceedings{zheng2025online,
  author = {Qinqing Zheng and Mikael Henaff and Amy Zhang and Aditya Grover and Brandon Amos},
  title = {{Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback}},
  booktitle = {RLC},
  year = {2025},
  url = {https://openreview.net/forum?id=Q1mUbfMTcp},
}
Figure