Integrating Offline Pre-Training with Online Fine-Tuning: A Reinforcement Learning Approach for Robot Social Navigation

Run Su, Hao Fu, Shuai Zhou, Yingao Fu

Introduction

Navigating a robot through a human crowd is more than a simple obstacle avoidance task; it is a complex "social" challenge. The robot must not only reach its destination but also respect human personal space and predict the fluid, often unpredictable movements of pedestrians. Traditional Reinforcement Learning (RL) approaches generally fall into two categories: Online RL, which is adaptive but dangerous during the learning phase, and Offline RL, which is safe but often fails due to "distribution shift" when the robot encounters real-world scenarios not present in its training data.

This paper introduces the Offline-to-Online Fine-Tuning Reinforcement Learning (OTOFRL) algorithm. OTOFRL bridges the gap between safe pre-training and adaptive deployment by integrating a spatio-temporal transformer that predicts "Return-to-Go" (RTG) values in real-time, allowing the robot to adjust its behavior dynamically as crowd conditions change.

The Problem: The Gap Between Training and Reality

In social navigation, the primary hurdle for Offline RL is the static nature of the datasets. Models like the Decision Transformer typically rely on a fixed target reward (Return-to-Go) to guide behavior. However, in a dynamic crowd, a fixed target is insufficient. If the environment becomes more complex than what was seen during training, a fixed RTG can lead to "frozen" behavior—where the robot becomes overly hesitant—or aggressive maneuvers that risk collisions.

Conversely, Online RL requires the robot to learn through trial and error in the physical world. For social robots, this is unacceptable, as the learning process involves potentially dangerous interactions with pedestrians. The challenge, therefore, is to create a system that starts with a safe, pre-trained foundation and fine-tunes itself online without compromising safety or efficiency.

Methodology: The OTOFRL Framework

OTOFRL addresses these challenges through three primary technical innovations:

1. Dynamic RTG Prediction (RTGP)

Instead of using a hard-coded target reward, the authors developed a spatio-temporal fusion transformer. This model jointly encodes the temporal patterns of pedestrian motion and the spatial dynamics of the crowd. By predicting the expected cumulative return in real-time, the robot can align its expectations with the actual complexity of the crowd it is currently facing. This turns the "target" into a moving goalpost that adapts to environmental difficulty.

2. Causal Transformer Architecture

The navigation policy is framed as a sequence modeling task. Much like how Large Language Models predict the next word in a sentence, the OTOFRL robot analyzes the history of states, actions, and predicted RTGs to decide its next move. This allows the robot to maintain a "memory" of recent pedestrian trajectories, leading to more informed decision-making.

3. Hybrid Experience Sampling and Dual Timescale Updates

To stabilize the transition from offline data to online interaction, the system utilizes a hybrid buffer that mixes "expert" offline data with new "interactive" data. Furthermore, the authors implement a dual timescale update strategy: the RTG predictor updates rapidly to catch up with environmental changes, while the navigation policy updates more slowly to maintain behavioral stability and prevent erratic movements.

Experimental Results

The OTOFRL algorithm was evaluated in a simulated "circle crossing" environment and validated on a physical mobile robot equipped with RPLIDAR-A1 sensors.

Quantitative Performance

OTOFRL significantly outperformed six state-of-the-art baselines, including ORCA, SARL, and the Online Decision Transformer (ODT):

  • Success Rate: OTOFRL achieved a 99.6% success rate, compared to 95.0% for ODT and only 61.2% for the traditional ORCA method.
  • Safety: The model recorded a collision rate of only 0.4%, the lowest among all tested methods.
  • Efficiency: The system improved sampling efficiency by nearly 14% over ODT and 31% over traditional online DRL methods like SARL, meaning the robot reaches peak performance with significantly less data.

Qualitative Observations

Beyond the numbers, OTOFRL demonstrated "socially compliant" behavior. Unlike previous models that might maintain high speeds in dense crowds, OTOFRL learned to decelerate when approaching crowded areas. This controlled deceleration improved safety without sacrificing the ultimate success rate, showing a sophisticated understanding of social navigation norms.

Conclusion and Future Work

OTOFRL provides a robust framework for transitioning robots from the lab to the real world. By treating navigation as a dynamic sequence modeling task and predicting rewards in real-time, the system overcomes the distribution shifts that typically plague offline learning.

While the current results in circular crossing scenarios are promising, future research will focus on higher-density "crush" conditions and more unstructured environments, such as busy hospital corridors or outdoor festivals. Additionally, the authors aim to investigate long-term adaptation to ensure the model does not suffer from "catastrophic forgetting" as it remains deployed in the field for extended periods.

Summary updated

BibTeX
@inproceedings{su2025integrating,
  author = {Run Su and Hao Fu and Shuai Zhou and Yingao Fu},
  title = {{Integrating Offline Pre-Training with Online Fine-Tuning: A Reinforcement Learning Approach for Robot Social Navigation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11245919},
}
Figure