Analyzing Neural Jacobian Methods in Applications of Visual Servoing and Kinematic Control

Michael Przystupa, Masood Dehghan, Martin Jägersand, A. Rupam Mahmood

Abstract

Designing adaptable control laws that can transfer between different robots is a challenge because of kinematic and dynamic differences, as well as in scenarios where external sensors are used. In this work, we empirically investigate a neural networks ability to approximate the Jacobian matrix for an application in Cartesian control schemes. Specifically, we are interested in approximating the kinematic Jacobian, which arises from kinematic equations mapping a manipulator's joint angles to the end-effector's location. We propose two different approaches to learn the kinematic Jacobian. The first method arises from visual servoing where we learn the kinematic Jacobian as an approximate linear system of equations from the k-nearest neighbors for a desired joint configuration. The second, motivated by forward models in machine learning, learns the kinematic behavior directly and calculates the Jacobian by differentiating the learned neural kinematics model. Simulation experimental results show that both methods achieve better performance than alternative data-driven methods for control, provide closer approximations to the proper kinematics Jacobian matrix, and on average produce better-conditioned Jacobian matrices. Real-world experiments were conducted on a Kinova Gen-3 lightweight robotic manipulator, which includes an uncalibrated visual servoing experiment, a practical application of our methods, as well as a 7-DOF point-to-point task highlighting that our methods are applicable on real robotic manipulators.

Introduction

In classical robotics, controlling a manipulator requires a precise mathematical model called the kinematic Jacobian. This matrix serves as the mathematical bridge between a robot's joint movements and its end-effector's motion, defining how a small change in joint angles ($\Delta q$) translates to a change in the end-effector's position ($\Delta x$).

However, calculating an analytical Jacobian is difficult in "uncontrolled" settings—such as when using uncalibrated external cameras (Visual Servoing) or dealing with complex, hard-to-model mechanics. While "ab initio" (from scratch) learning methods exist, they often struggle with local limitations (finite differences have "no memory") or scalability issues (non-parametric methods like k-Nearest Neighbors). This research investigates whether neural networks can serve as efficient, global, parametric approximators for the Jacobian to enable smoother, more reliable robotic control.

Methodology

The authors propose and compare two distinct neural architectures for approximating the kinematic Jacobian:

  1. Neural Jacobian (NJ): The network is trained to output the $m \times n$ Jacobian matrix directly. It uses a loss function based on the Secant condition ($\Delta x \approx J \Delta q$), essentially learning the relationship between observed motion and joint changes. A variant, the Bi-directional Neural Jacobian (Bi-NJ), adds a loss term for the inverse Jacobian to ensure the resulting matrix is "well-conditioned" and mathematically stable for inversion.
  2. Neural Kinematics (NK): Instead of learning the derivative directly, the network learns the forward kinematics function ($f(q) = x$). Once the model can predict the robot's position accurately, the Jacobian is recovered by taking the derivative of the network's output with respect to its input via backpropagation (autodifferentiation).
Architectural comparison of the Neural Jacobian (NJ) and Neural Kinematics (NK) methods.

Experimental Setup

The methods were evaluated across three distinct environments:

  • Kinematic Simulation: A simulated Kinova Gen-3 robot performing reaching tasks. Data was collected using 110,000 to 200,000 points with random joint velocities.
  • 2-DOF Visual Servoing: A real-world task where a Kinova robot tracks a target using a standard webcam.
  • 7-DOF Point-to-Point: A real-world reaching task using the full degrees of freedom of the Kinova Gen-3 manipulator.

Key Results and Insights

Quantitative analysis demonstrated that neural methods provide more stable, better-conditioned control than traditional data-driven baselines like Broyden’s method or LL-KNN.

Convergence performance (Euclidean distance over time) across different trajectory lengths, comparing neural methods to traditional baselines.
  • The Power of Autodiff: The Neural Kinematics (NK) approach proved surprisingly robust. By simply learning "where the arm is," the network's internal gradients provided a highly accurate Jacobian. This suggests that learning the underlying function is often easier than learning its derivative directly. Tanh activation functions generally outperformed ReLU here, likely because Tanh is continuously differentiable.
  • Conditioning and Stability: In robotics, inverting the Jacobian is necessary to find required joint speeds. If the matrix is "ill-conditioned," movements become erratic. The Bi-NJ method successfully forced the network to produce more stable matrices, achieving a 15.6% performance gain in success rate over the best non-neural baseline in multi-point simulations.
  • Global Memory: Unlike local methods that fail when the robot moves far from its starting point, the neural networks built a global map of the workspace. On the physical 7-DOF Kinova robot, neural methods maintained high success rates (up to 91%) even for distant targets, whereas the k-NN baseline's performance dropped to 64%.

Limitations and Future Work

While effective, the Neural Kinematics (NK) method showed sensitivity to noise, particularly in visual servoing where tracker flicker can degrade performance. Additionally, the current models do not explicitly address "singularities"—configurations where the Jacobian becomes non-invertible. Future work will focus on incorporating robust loss functions to handle noisy outliers and developing strategies to manage kinematic singularities.

Summary updated

Abstract

Designing adaptable control laws that can transfer between different robots is a challenge because of kinematic and dynamic differences, as well as in scenarios where external sensors are used. In this work, we empirically investigate a neural networks ability to approximate the Jacobian matrix for an application in Cartesian control schemes. Specifically, we are interested in approximating the kinematic Jacobian, which arises from kinematic equations mapping a manipulator's joint angles to the end-effector's location. We propose two different approaches to learn the kinematic Jacobian. The first method arises from visual servoing where we learn the kinematic Jacobian as an approximate linear system of equations from the k-nearest neighbors for a desired joint configuration. The second, motivated by forward models in machine learning, learns the kinematic behavior directly and calculates the Jacobian by differentiating the learned neural kinematics model. Simulation experimental results show that both methods achieve better performance than alternative data-driven methods for control, provide closer approximations to the proper kinematics Jacobian matrix, and on average produce better-conditioned Jacobian matrices. Real-world experiments were conducted on a Kinova Gen-3 lightweight robotic manipulator, which includes an uncalibrated visual servoing experiment, a practical application of our methods, as well as a 7-DOF point-to-point task highlighting that our methods are applicable on real robotic manipulators.
BibTeX
@inproceedings{przystupa2021analyzing,
  author = {Michael Przystupa and Masood Dehghan and Martin Jägersand and A. Rupam Mahmood},
  title = {{Analyzing Neural Jacobian Methods in Applications of Visual Servoing and Kinematic Control}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=yYzOpsFmurT},
}
Figure