Abstract
This paper presents a framework for integrating verification vision into the RAPT (Robot Apt) object-level programming language. By utilizing symbolic reasoning and transformation matrices termed "modifying factors," the system bridges the gap between idealized off-line plans and real-world environmental uncertainties. The approach allows for runtime updates to a robot's world model based on sensory feedback without compromising the underlying logical structure of the spatial reasoning.
Introduction
Industrial robotics often faces a "reality gap" where off-line programs fail due to minor discrepancies between the programmed model and the physical environment. While high-level languages like RAPT excel at spatial reasoning—defining how objects should relate—they traditionally lack robust mechanisms to incorporate real-time sensory data. This work introduces a method to allow vision data to modify planned positions dynamically, ensuring that high-level commands like "place the bolt in the hole" can be executed accurately despite positioning errors.
Verification Vision and Language Extensions
The core of the proposed system is "verification vision." Unlike general-purpose computer vision, which attempts to identify unknown objects, verification vision leverages prior knowledge of the scene. It focuses on locating specific features (e.g., edges or vertices) of known objects to refine their coordinates. To support this, several new commands were integrated into RAPT:
- LOOK: Directs the vision system to identify a specific feature.
- INVIOLATE: Specifies constraints that must be maintained (e.g., an object's base remaining flush with a surface).
- TOLERANCE: Defines the acceptable error margins.
- COMBINE: Aggregates these instructions into a coherent vision task.
Symbolic Reasoning and Modifying Factors
Because vision data is only available at runtime, the system employs symbolic expressions as placeholders for unknown positions during compilation. The discrepancy between the nominal (planned) position and the sensed (actual) position is captured as a "modifying factor"—a transformation matrix.
These factors are propagated through the system using "linking rules." For instance, if a robot verifies the position of a block it is holding, the system automatically updates the estimated position of the robot's gripper. This propagation ensures that a single measurement can refine the entire chain of related objects in the assembly task.
Unified Sensing Framework
A key insight of this work is the treatment of sensors as providers of spatial constraints. By representing relationships like "AGAINST" (detected via touch or vision) as mathematical constraints, the framework provides a unified approach to multi-modal sensing. This allows the system to handle vision, touch, and proximity data under the same symbolic reasoning umbrella.
Implementation and Results
The framework was implemented within the RAPT environment and validated through simulations. The results demonstrated that the system could successfully calculate modifying factors from simulated vision data and adjust robot trajectories in real-time. By focusing on the difference between plan and reality, the system maintains computational efficiency, avoiding the need for full re-simulation during execution.
Limitations and Future Directions
The author identifies the "correspondence problem"—the difficulty of accurately matching model features to image pixels—as a primary challenge for real-world deployment. Additionally, while translational errors are well-handled, the treatment of rotational tolerances requires further development. Future work will focus on addressing these issues and testing the system in noisy, real-world environments.
Summary updated