Vision-Guided Robot Control for 3D Object Recognition and Manipulation

Sun Q, E. Haemmerle, Yiwei Cheng, P. GamageInTech 2008 · DOI · OpenAlex

Abstract

Research into a fully automated vision-guided robot for identifying, visualising and manipulating 3D objects with complicated shapes is still undergoing major development world wide. The current trend is toward the development of more robust, intelligent and flexible vision-guided robot systems to operate in highly dynamic environments. The theoretical basis of image plane dynamics and robust image-based robot systems capable of manipulating moving objects still need further research. Research carried out in our group has been focused on the development of more robust image processing methods and adaptive control of robot. Our research has been focused on manufacturing automation and medical surgery (Graham, Xie et al. 2007), with the main issues being the visual feature tracking, object recognition and adaptive control of the robot that requires both position and force feedback. The developed vision-guided robot system is able to recognise general 2D and 3D objects using a CCD camera and the VisionPro software package. Object recognition algorithms were developed to improve the performance of the system and a user interface has been successfully developed to allow easy operation by users. Intensive testing was performed to verify the performance of the system and the results show that the system is able to recognise 3D objects and the restructured model is precise for controlling the robot. The kinematics model of the robot has also been validated together with the robot control algorithms for the control of the robot.

Introduction

In modern industrial and medical robotics, the transition from "blind" repetitive tasks to intelligent, adaptive interaction remains a significant challenge. Traditional robots often rely on pre-programmed paths, making them brittle in dynamic environments where objects might move, vary in appearance, or be subject to changing environmental conditions. This research addresses three specific challenges: the computational complexity of 3D recognition from varying angles, environmental sensitivity to lighting and texture, and the mathematical overhead of real-time inverse kinematics. By integrating advanced vision algorithms with intelligent control systems, the authors present a framework for robust robotic manipulation.

Methodology

The system utilizes an "eye-in-hand" configuration, where a CCD camera is mounted directly on the robot's end-effector. This setup provides a dynamic and proximal view of the workspace, feeding data into a multi-stage vision system that handles calibration, acquisition, and processing.

System architecture of the vision-guided robot control framework.

Key technical components include:

  • Active Contours (Snakes): To segment objects from the background, the system uses "snakes"—splines that minimize energy to "cling" to object edges. This provides a continuous estimation of the object's boundary even under slight deformations or noise.
  • Color-Invariant Descriptors: While standard descriptors like SIFT and SURF typically rely on grayscale information, the authors proposed an $m_1m_2m_3$ color model. This model is specifically designed to be invariant to illumination changes, shadows, and highlights. Principal Component Analysis (PCA) is then applied to reduce the dimensionality of the feature space, ensuring real-time processing speeds.
  • Sugeno Fuzzy Inference System (SFIS): For inverse kinematics—the process of determining joint angles for a desired end-effector position—traditional trigonometric solvers can be slow. The authors replaced these with an SFIS trained using a Genetic Algorithm (GA). The GA optimizes the fuzzy rules and membership functions to map Cartesian coordinates to robot joint angles with high precision.

Results and Discussion

Experimental validation was performed using an ASEA 5-DOF robot tasked with inspecting car wheels on a moving assembly line. The results highlighted the efficiency of the hybrid control approach. After tuning with Genetic Algorithms, the average sum of squared errors in joint variable prediction dropped from 2.339 to 0.000193 radians, a significant improvement in positioning accuracy.

Active contour (snake) iterations converging on a wheel rim.

The SFIS approach proved highly efficient, providing kinematic solutions in less than 0.8 milliseconds, which is well within the requirements for high-speed industrial operations. Furthermore, the vision system's robustness was demonstrated through stable feature identification under scale changes (up to a factor of two) and improved performance over standard SURF when color information was leveraged. A hierarchical recognition strategy further optimizes performance by moving the robot to capture additional views only when initial recognition scores are ambiguous, balancing accuracy with operational speed.

Conclusion

The developed vision-guided robot system successfully demonstrates the ability to recognize 3D objects and provides a precise, real-time control model. The integration of color-invariant features and fuzzy logic control offers a robust solution for dynamic environments. Future work will focus on automating threshold selection for recognition and integrating force feedback to handle tasks requiring physical contact, such as medical surgery or complex mechanical assembly.

Summary updated

Abstract

Research into a fully automated vision-guided robot for identifying, visualising and manipulating 3D objects with complicated shapes is still undergoing major development world wide. The current trend is toward the development of more robust, intelligent and flexible vision-guided robot systems to operate in highly dynamic environments. The theoretical basis of image plane dynamics and robust image-based robot systems capable of manipulating moving objects still need further research. Research carried out in our group has been focused on the development of more robust image processing methods and adaptive control of robot. Our research has been focused on manufacturing automation and medical surgery (Graham, Xie et al. 2007), with the main issues being the visual feature tracking, object recognition and adaptive control of the robot that requires both position and force feedback. The developed vision-guided robot system is able to recognise general 2D and 3D objects using a CCD camera and the VisionPro software package. Object recognition algorithms were developed to improve the performance of the system and a user interface has been successfully developed to allow easy operation by users. Intensive testing was performed to verify the performance of the system and the results show that the system is able to recognise 3D objects and the restructured model is precise for controlling the robot. The kinematics model of the robot has also been validated together with the robot control algorithms for the control of the robot.
BibTeX
@inproceedings{q2008vision,
  author = {Sun Q and E. Haemmerle and Yiwei Cheng and P. Gamage},
  title = {{Vision-Guided Robot Control for 3D Object Recognition and Manipulation}},
  booktitle = {InTech},
  year = {2008},
  doi = {10.5772/6223},
}
Figure