TL;DR
The authors introduce ONI, a distributed reinforcement learning framework that simultaneously trains an agent and learns an intrinsic reward function using online feedback from an asynchronous LLM server. By distilling LLM feedback on the fly from the agent's own experience, ONI eliminates the need for massive pre-collected offline datasets while scaling to billions of environment steps, matching or exceeding state-of-the-art performance on challenging NetHack tasks.
What Problem Does This Paper Address?
Reinforcement learning (RL) excels at optimization but notoriously struggles when rewards are sparse—such as in complex games or open-ended environments where an agent might wander for thousands of steps before receiving any feedback. To fix this, researchers use intrinsic rewards (extra reward signals) to guide exploration.
Recently, leveraging the vast prior knowledge of Large Language Models (LLMs) to automatically generate these rewards has become a hot topic. However, existing methods suffer from a major dilemma:
- They generate reward code directly, which struggles to capture complex, unstructured semantic features (like text logs or visual descriptions).
- They query an LLM at every single environment step, which is computationally impossible when scaling to the millions or billions of steps required for complex tasks.
- They rely on a complex, multi-stage offline process (like the prior state-of-the-art method, Motif) that requires a massive, pre-collected dataset of diverse agent experiences just to train a reward model before RL training even begins.
ONI tackles this bottleneck by integrating LLM feedback directly into the online training loop, allowing the agent to learn both the policy and the reward function concurrently from its own live experience.
Methods
The core architecture of ONI is built on top of Sample Factory's high-throughput Asynchronous Proximal Policy Optimization (APPO) framework.
To prevent the slow inference times of LLMs from bottlenecking the fast-paced RL environment workers, ONI decouples the components using an Asynchronous LLM Worker:
As the agent explores, text captions of its observations are placed into a Last-In-First-Out (LIFO) queue. An independent worker sends these captions via HTTP/REST to a remote server running vLLM (hosting LLaMA-3.1-8B-Instruct).
Instead of using the LLM directly as the reward function, ONI distills this online feedback into a smaller, local reward model. The authors explore three different algorithmic flavors for this distillation:
- Retrieval: Hashes binary "helpful/unhelpful" LLM labels directly into a table. It is simple and hyperparameter-free but cannot generalize to unseen text.
- Classification: Trains a local binary classifier to predict the LLM's sentiment label from the observation, enabling generalization to similar, unseen captions.
- Ranking: Sends pairs of captions to the LLM for preference labeling and trains a ranking model using a Bradley-Terry preference model, adapted with running statistics to handle the continuously changing online dataset.
Datasets
The framework was evaluated on the NetHack Learning Environment (NLE), a notoriously difficult, procedurally generated terminal-based dungeon crawler. NetHack features extremely long horizons, sparse rewards, and highly complex game dynamics.
The authors tested ONI across four specific tasks:
Score: A dense reward task based on the in-game score.Oracle: A sparse reward task requiring the agent to find a specific character deep in the dungeon.StaircaseLvl3&StaircaseLvl4: Sparse tasks requiring the agent to descend multiple levels.
They also evaluated the agents in a completely reward-free setting, tracking raw game progress metrics: experience level, dungeon level, gold collected, and unique locations explored (scout).
Key Insights and Ideas
- Online Learning Eliminates Privileged Data: ONI proves that you don't need a pre-existing dataset collected by a top-tier agent to bootstrap an intrinsic reward model. Learning purely from the agent's own evolving, online experience is highly effective.
- Generalization Saves Compute: In ablation studies where LLM annotations were heavily restricted (subsampled down to 1%), the parametric Classification model maintained strong performance, whereas the non-parametric Retrieval model plummeted. This highlights the value of distilling feedback into a model that can generalize.
- Don't Deduplicate Preferences: For the ranking-based approach, deduplicating captions actually ruined the reward model's ability to learn. Keeping the natural distribution of data (including frequent, mundane messages like blank screens) is crucial for the model to accurately calibrate what events are truly rare and valuable.
Key Results
- Matches Offline SOTA: ONI-classification successfully matched or closely tracked the performance of Motif across all sparse NetHack tasks, despite Motif requiring a pre-collected dataset from an agent already trained for 1 billion steps.
- Outperforms in Reward-Free Settings: In completely reward-free environments, ONI-classification outperformed Motif on 3 out of 4 game progression metrics.
- High System Efficiency: Despite adding an LLM into the loop, ONI retains 80-95% of Sample Factory's original throughput, achieving an average of 26k to 30k environment interactions per second on a single GPU setup.
Limitations and Future Work
- Caption Dependency: ONI relies on the environment providing textual captions of observations. For purely visual environments (like Atari or robotics), it would require integrating an auxiliary image-captioning model.
- Basic Queueing Strategy: The current architecture uses a simple LIFO queue to decide which observations to send to the LLM. The authors note that developing more sophisticated active learning or uncertainty-based prioritization schemes could make significantly better use of the limited LLM queries.
Summary updated