Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision

Ashvin V. Nair, Brian Zhu, Gokul Narayanan, Eugen Solowjow, Sergey Levine

Introduction

In industrial robotics, "insertion" tasks—such as plugging a USB-C cable or fitting specialized connectors—are notoriously difficult. These tasks are highly sensitive to geometry, lighting, and positioning. Traditional industrial robots rely on high-precision localization, which often fails in unstructured environments like messy factory floors where exact socket coordinates are unknown.

While Reinforcement Learning (RL) allows robots to learn from experience, it typically requires a manually defined "reward function" for every new object. To overcome this bottleneck, researchers have developed Learning on the Job (LOTJ), a system that enables robots to master never-before-seen connectors using only visual input and self-generated rewards.

The LOTJ Framework

The LOTJ system follows a four-phase pipeline designed to transition a robot from general knowledge to task-specific mastery in approximately 50 minutes.

The LOTJ framework consists of offline pretraining on 50 connector domains followed by self-supervised online finetuning.
  1. Offline Pretraining: The process begins with a large-scale dataset of 70,000 trajectories involving 50 different connector types, ranging from standard USBs to specialized industrial NEMA plugs.
  2. Self-Rewarding System: A reward classifier is trained to recognize successful insertions from camera feeds. This exploits the insight that recognizing success is significantly easier for a model than performing the task itself.
  3. Domain Adversarial Information Bottleneck (DAIB): This novel representation learning technique splits visual features into domain-invariant components (general shapes) and domain-specific components (unique connector details), preventing the model from overfitting to irrelevant background details.
  4. Online Finetuning: When presented with a new connector, the robot uses its pretrained policy to attempt the task, grades itself using the reward classifier, and updates its behavior via Implicit Q-Learning (IQL).

Technical Innovation: DAIB

A core challenge in cross-domain robotics is preventing the model from "cheating" by focusing on the robot's gripper or the table color. The DAIB architecture addresses this by using an adversarial loss to force the domain-invariant representation ($z_I$) to "forget" the specific connector identity. Simultaneously, the domain-specific representation ($z_S$) is passed through an information bottleneck to filter out noise, ensuring the policy focuses only on semantically meaningful relationships between the connector and the socket.

Experimental Results

The system was evaluated using two 7-DoF Sawyer robots on "held-out" connectors never seen during training.

Success Rates and Robustness

In scenarios with ±10mm of positioning error, LOTJ achieved a success rate exceeding 75%, significantly outperforming standard RL methods like SAC. In even more challenging "offset" scenarios (10-20mm error), where the initial offline policy often had a 0% success rate, online finetuning allowed the robot to reach success rates as high as 95%.

Visual Attention

One of the most compelling results of the finetuning process is the shift in the robot's visual attention. Grad-CAM visualizations demonstrate that as the robot learns, its internal "focus" moves from uninformative regions to the critical contact point between the connector and the socket.

Grad-CAM heatmaps show the robot's attention shifting to the precise contact point after finetuning.

Conclusion and Future Work

LOTJ demonstrates that robots can autonomously adapt to new, complex manipulation tasks in under an hour. However, the system currently relies heavily on the accuracy of the reward model. If the classifier produces false positives, the robot may learn incorrect behaviors. Future research will explore incorporating active learning—where the robot asks for human help when uncertain—and lifelong learning to continuously expand the robot's global knowledge base.

Summary updated

Abstract

Learning-based methods in robotics hold the promise of generalization, but what can be done if a learned policy does not generalize to a new situation? In principle, if an agent can at least evaluate its own success (i.e., with a reward classifier that generalizes well even when the policy does not), it could actively practice the task and finetune the policy in this situation. We study this problem in the setting of industrial insertion tasks, such as inserting connectors in sockets and setting screws. Existing algorithms rely on precise localization of the connector or socket and carefully managed physical setups, such as assembly lines, to succeed at the task. But in unstructured environments such as homes or even some industrial settings, robots cannot rely on precise localization and may be tasked with previously unseen connectors. Offline reinforcement learning on a variety of connector insertion tasks is a potential solution, but what if the robot is tasked with inserting previously unseen connector? In such a scenario, we will still need methods that can robustly solve such tasks with online practice. One of the main observations we make in this work is that, with a suitable representation learning and domain generalization approach, it can be significantly easier for the reward function to generalize to a new but structurally similar task (e.g., inserting a new type of connector) than for the policy. This means that a learned reward function can be used to facilitate the finetuning of the robot's policy in situations where the policy fails to generalize in zero shot, but the reward function generalizes successfully. We show that such an approach can be instantiated in the real world, pretrained on 50 different connectors, and successfully finetuned to new connectors via the learned reward function. Videos can be viewed at https://sites.google.com/view/learningonthejob
BibTeX
@inproceedings{nair2023learning,
  author = {Ashvin V. Nair and Brian Zhu and Gokul Narayanan and Eugen Solowjow and Sergey Levine},
  title = {{Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161491},
}
Figure