Table-Balancing Cooperative Robot Based on Deep Reinforcement Learning

Yewon Kim, Dae-Won Kim, Bo-Yeong Kang

TL;DR

This paper introduces a novel deep reinforcement learning (DRL) approach for a humanoid robot to cooperatively balance a table with a human. By using a camera to recognize human actions and the table's state, the robot learns to perform precise balancing maneuvers, achieving 90% operational precision in real-world tests. This work advances human-robot collaboration by enabling robots to participate in dynamic, shared physical tasks.

What Problem Does This Paper Address?

Many everyday tasks, like moving furniture, require cooperation between individuals. While reinforcement learning (RL) has been successful in enabling robots to perform tasks autonomously, most research has focused on single-robot operations. Cooperative tasks, especially those involving delicate balancing, present a significant challenge: how can a robot effectively collaborate with a human in real-time, adapting to unpredictable human movements and maintaining stability? Existing methods often rely on high-performance, specialized equipment (like motion capture systems) or require extensive human intervention for calibration and operation. This paper aims to bridge this gap by developing a DRL-based system that allows a humanoid robot to learn and execute table-balancing actions cooperatively with a human, using only a standard camera.

Methods

The core of this research lies in a Deep Q-Network (DQN) based deep reinforcement learning approach. The system works as follows:

  • Human Action Recognition: A camera on the humanoid robot captures images of the table. The robot analyzes these images to classify the human's actions into five categories based on how they are raising or lowering the table (e.g., "up a lot," "up," "keep," "down," "down a lot"). This classification is achieved by observing changes in the table's apparent shape in the robot's view.
  • DRL for Balancing: A DQN agent is trained to predict the optimal robot action based on the recognized human action and the current table state. The robot's actions are primarily controlled by adjusting its leg angles to maintain balance.
  • Reward System: The robot receives positive rewards for successfully balancing the table and negative rewards for failures. This feedback loop guides the learning process.
  • Robot Control: The robot's physical actions are translated into precise movements of its legs, specifically adjusting hip, knee, and ankle angles to counteract imbalances. The paper details the kinematic equations used for this control.
  • Parameter Optimization: Extensive experiments were conducted to find optimal hyperparameters for the DQN, including learning rate, discount factor, and exploration strategy (e.g., epsilon-greedy policies).

Datasets

The paper introduces a new dataset for training and evaluating the robot's table-balancing capabilities.

  • Custom Dataset: A dataset of 7174 table images was collected using the robot's camera. These images were captured while humans performed various table-balancing actions, tagged with the five human action states. The images were preprocessed into different sizes (128x170, 84x112, 84x84) and color formats (RGB, grayscale) for experimentation.

Key Insights and Ideas

  • Vision-based Human Action Recognition for Dynamic Tasks: The ability to infer human intent and action solely from visual cues of the shared object (the table) is crucial for real-time cooperative task execution. The paper demonstrates that changes in the table's appearance can effectively signal human actions.
  • DRL for Fine-Grained Physical Collaboration: Applying DRL to a dynamic physical task like table balancing, where continuous adaptation is needed, shows the power of these algorithms beyond game-playing. The DQN's ability to learn complex state-action mappings is key.
  • Exploration vs. Exploitation Balance: The study highlights the importance of exploration (e.g., using 10-greedy or 20-greedy policies) in achieving optimal policy convergence. Pure exploitation (greedy policy) can lead to suboptimal solutions by not exploring alternative actions sufficiently.
  • Image Preprocessing Matters: The choice of image size and color format significantly impacts the learning performance and computational cost. The paper empirically determined that 128x170 RGB images offered the best balance for their DQN training.

Key Results

  • Optimal Policy Convergence: The proposed DQN-based method achieved an optimal policy convergence rate (OPCR) of 90% with specific hyperparameters: a learning rate of 0.005, a discount factor of 0.9, and a 10-greedy exploration policy.
  • Hardware Performance: In real-time hardware (H/W) demonstrations, the trained DQN-based robot achieved an operation precision of 90% in balancing the table, significantly outperforming a baseline Q-learning with AlexNet approach (67% precision).
  • Robustness: The robot demonstrated the ability to correctly analyze human action states and perform appropriate balancing actions, even when the table was initially unbalanced. For instance, it could correct angles of up to 7.47° and -2.64°.

Limitations and Future Work

  • Limited Human Action States: The current system classifies human actions into five discrete states. Future work could expand this to a more continuous or granular representation of human intent.
  • Simplified Balancing Model: The current model focuses on table slope and assumes the table movement is consistent. More complex scenarios involving dynamic table movement or varying surface properties could be explored.
  • Generalization: While the robot performed well in tests, further research is needed to assess its generalization capabilities to different table types, environments, and human collaborators.
  • Future Directions: The authors suggest expanding the range of human action states and robot actions, implementing more delicate balancing methods, and applying the learned model to real-life scenarios. They also mention ongoing work on transfer learning and continual learning for handling a wider variety of objects.

Summary updated

BibTeX
@inproceedings{kim2023table,
  author = {Yewon Kim and Dae-Won Kim and Bo-Yeong Kang},
  title = {{Table-Balancing Cooperative Robot Based on Deep Reinforcement Learning}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=dxPvjz9mYh},
}
Figure