Network Layer Analysis for a RL-Based Robotic Reaching Task

Benedikt Feldotto, Heiko Lengenfelder, Florian Röhrbein, Alois C. KnollFrontiers Robotics AI 2022 · DOI

Introduction

The internal mechanisms of reinforcement learning (RL) networks in robotics often remain a "black box," unlike the well-understood feature hierarchies in computer vision. This research investigates the internal organization of Deep Q-Learning (DQN) networks trained for robotic reaching tasks. The study explores how information is distributed across layers, identifies redundant neurons for network pruning, and examines whether networks trained on different robot kinematics develop similar internal representations.

Methodology and Experimental Setup

The researchers trained DQN agents to control simulated robot manipulators with 2, 3, and 4 degrees of freedom (DOF) within a 2D vertical plane.

Figure 1: Robotic manipulator models used in the study. (A) shows the initial configuration in a vertical plane, and (B) shows the transferability to a robotic simulation.

To analyze these networks, several novel techniques were employed:

  • Activation Distance Metric: Neurons were compared based on their activation history over entire movement episodes rather than static weights.
  • Hierarchical Clustering: This metric allowed for grouping similar-behaving neurons, highlighting network redundancy.
  • Heuristic Pruning: Redundant neurons were merged into single units, and the resulting smaller network was re-initialized with original weights and retrained.
  • Linear Projection Mapping: To compare different networks (e.g., 3-joint vs. 4-joint), linear regression with Lasso regularization was used to determine if activations in one network could predict those in another.

The custom simulated environment featured homogeneous manipulators where the task was to move the fingertip to a random 2D location within 60 control steps. Results were validated across 300 test episodes.

Key Insights and Results

The analysis revealed an "hourglass" specialization pattern: input and output layers contain highly distinct, specialized activations, while inner layers are significantly redundant. This redundancy allows for substantial network compression. For a 3-joint robot, a specific cluster threshold ($\tau=0.2$) reduced the network size by approximately 50% without compromising performance or increasing retraining time.

Furthermore, the study found that kinematic similarity is a strong predictor of internal representation similarity. Networks trained on robots with similar joint counts (e.g., 3 vs. 4 joints) showed high layer-wise projection accuracy. This suggests that RL agents develop consistent internal "languages" for similar physical tasks. However, mapping a simpler network (2-joint) to a more complex one (4-joint) was less successful, as the simpler network lacked the necessary complexity to represent the additional degrees of freedom.

Conclusion and Future Directions

This work demonstrates that RL networks for robotic control possess inherent hierarchical structures that can be exploited for pruning and cross-robot knowledge transfer. While the study was limited to 2D vertical plane movements and homogeneous robot designs, it lays the groundwork for understanding RL "brains." Future research will explore 3D tasks with obstacles and investigate whether these internal mappings hold true across heterogeneous robot types, such as grippers versus legged robots.

Summary updated

BibTeX
@article{feldotto2022network,
  author = {Benedikt Feldotto and Heiko Lengenfelder and Florian Röhrbein and Alois C. Knoll},
  title = {{Network Layer Analysis for a RL-Based Robotic Reaching Task}},
  journal = {Frontiers Robotics AI},
  year = {2022},
  doi = {10.3389/frobt.2022.799644},
}
Figure