TINA: The Sheffeild AIVRU Vision System

This paper introduces TINA, a 3D vision system designed to enable robots to perceive and interact with their environment. TINA achieves this by recovering a detailed 3D geometric description of a scene from stereo images, which is then used for object recognition and guiding a robot arm in tasks like pick-and-place.

TL;DR

The Sheffield AIVRU vision system (TINA) uses passive stereo vision to build detailed 3D geometric models of scenes, enabling robust object recognition and precise robot arm control for tasks like pick-and-place in cluttered environments. This work demonstrates a significant step towards more autonomous and capable robotic systems by integrating sophisticated geometric reasoning with real-world robotic manipulation.

What Problem Does This Paper Address?

Robots operating in unstructured environments need to understand their surroundings in three dimensions to perform tasks effectively. Traditional vision systems often struggle with:

  • Depth Perception: Accurately determining the distance to objects and the shape of surfaces from 2D images.
  • Object Recognition in Clutter: Identifying specific objects when they are partially occluded or surrounded by other objects.
  • Precise Localization: Knowing the exact position and orientation of an object in 3D space for manipulation.

This paper addresses these challenges by developing a vision system that can extract rich 3D geometric information from stereo images, build comprehensive visual models, and use this information to guide a robot arm for precise manipulation tasks.

Methods

TINA is a modular system comprising several key components:

  • Passive Stereo Triangulation (PMF): This module recovers a sparse depth map from a pair of stereo images. It employs a Canny edge detector with sub-pixel accuracy and a disparity gradient constraint to solve the stereo correspondence problem. This constraint helps disambiguate matches and ensures a Lipschitz continuous disparity map.
  • Geometric Descriptive Base (GDB): This module takes the depth map and segments it into geometric primitives, specifically straight lines and circular arcs. It uses orthogonal regression and curve fitting techniques, recursively refining descriptions and handling issues like partial occlusion and unreliable data by predicting 2D data from 3D descriptions.
  • Scene and Model Matcher (SMM): This is the core recognition module. It matches 3D wireframe models of objects to the 3D scene descriptions generated by GDB. The matching process involves selecting focus features, identifying potential matches, and then finding consistent cliques of features that define a transformation (position and orientation) between the model and the scene. It leverages pairwise geometrical relationships and quaternion representations for efficient matching.
  • Integration of Edge Descriptions (TIED): This module integrates geometric information from multiple stereo views to create a more complete and robust object model. It uses statistical estimation theory to enforce consistency between views, improving depth location and refining the object's pose.
  • Regions, Edges, Vertices Graph (REVgraph): This is an environment that provides easy access between different representations of the scene (image, edge map, depth map, geometric description). It's built to support higher-level reasoning processes and includes a consistency maintenance system (CMS) for debugging and efficient reasoning.

Datasets

The paper describes experiments using:

  • CAD graphics generated images: These were used for evaluating the geometrical accuracy of the system, providing a controlled environment with known ground truth.
  • Natural scene images: The system was also tested on images acquired from natural scenes, demonstrating its applicability to real-world scenarios.
  • A specific object for pick-and-place: The demonstration involved a particular object that was pre-compiled into the system's visual model for the pick-and-place task.

The paper does not explicitly list or introduce new benchmark datasets for general use.

Key Insights and Ideas

  • Disparity Gradient Constraint: The use of a disparity gradient limit (e.g., 0.5) is crucial for robust stereo matching. It significantly reduces false matches by ensuring that the rate of change of disparity is limited, which is a more general continuity assumption than traditional methods.
  • Predicting 2D from 3D: A key innovation is using reliable 3D geometric descriptions to predict and refine unreliable or missing 2D image data. This allows for more complete scene descriptions even with noisy or incomplete stereo information.
  • Hierarchical Geometric Description: The system builds a rich, hierarchical geometric description of the scene, starting from edges and progressing to lines, arcs, and ultimately 3D models. This multi-level representation is essential for robust recognition.
  • Integration of Multiple Views: Combining information from multiple stereo views significantly improves the accuracy and completeness of object models and their localization, especially in depth.
  • Model-Based Recognition with Geometric Constraints: The SMM module effectively uses geometric relationships between features in a model to constrain the search space and find reliable matches in the scene, even in cluttered environments.

Key Results

  • The system successfully guided a UMI robot arm in a pick-and-place task, demonstrating its practical application.
  • The geometric descriptions recovered from stereo were found to have typical errors of less than a degree for line orientations and less than 3 degrees for normals of circular arcs in CAD images, with slightly higher errors for natural scenes.
  • Positional accuracy of features and curvature segmentation points were typically on the order of a few millimeters.
  • The SMM matcher, when matching a model with 41 features to a scene with 423 features, found a largest clique of 14 matched lines, indicating a successful partial match.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the time taken by the visual processing stages is not discussed, implying it might be a significant factor.
  • Model Dependency: The SMM matcher's search pruning heuristics are noted as being "obviously model dependent."
  • Future Work:
    • Augmenting the scene description with surface geometry and topological information.
  • Exploring predictive feed-forward to speed up the stereo algorithm.
  • Extending the matcher with a richer semantics of features and their pairwise geometrical relationships.
  • Exploiting negative or incompatible information to reduce false positives.
  • Iterating over camera parameter estimates using the recovered geometrical descriptions.

Summary updated

BibTeX
@inproceedings{1987tina,
  title = {{TINA: The Sheffeild AIVRU Vision System}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1987},
}
Figure