IRV: Learning to Integrate Visual Information Across Camera Movements

Peter N. Prokopowicz, Paul R. Cooper

Abstract

This paper introduces IRV (Integrated Robot Vision), a connectionist system that learns to synthesize a stable, comprehensive visual representation by observing image shifts during stochastic camera movements. Unlike traditional computer vision systems that rely on explicit geometric models and external calibration, IRV utilizes a self-supervised, model-free approach to compensate for complex lens distortions and non-uniform sampling. Through a developmental phase of "hypothesize-and-test," the system learns its own visual-motor geometry from scratch, demonstrating biological plausibility and robustness to extreme input distortions.

Introduction

Biological and high-performance robotic vision systems often employ foveal sensors, where the center of the field of view provides high resolution while the periphery remains blurry. To perceive a complete scene, these sensors must perform rapid movements, known as saccades, resulting in a sequence of disjointed and overlapping snapshots. The fundamental challenge is integration: combining these disparate views into a coherent internal representation.

Traditional solutions require precise analytic models—mathematical descriptions of focal length, lens distortion, and mechanical alignment. However, these models are fragile; any physical change to the camera or lens requires recalibration. IRV addresses this by learning visual-motor geometry directly from experience, eliminating the need for prior hardware knowledge.

Methodology

The IRV architecture is built upon a connectionist memory that replaces static equations with a dynamic Visual Motor Calibration Map.

The Visual Motor Calibration Map

This map stores the relationship between camera movement vectors and the resulting apparent motion of pixels. The system employs three-way connections that link the camera movement command, the pixel location before the movement, and the pixel location after the movement.

Evidence Accumulation and Voting

Because individual movements can be ambiguous—multiple pixels may appear similar—IRV uses a voting scheme for evidence accumulation. Each camera movement triggers potential pixel correspondences to "vote" for specific geometric relationships. Over thousands of iterations, true correspondences accumulate significant support, while noise and random correlations are filtered out.

Visualization of sparse pixel correspondences learned through the voting mechanism.

Stochastic Scaling

To manage the computational complexity of evaluating all possible pixel pairings, IRV utilizes stochastic scaling. The algorithm evaluates only a small, random subset of correspondences at any given time, allowing it to scale effectively to high-resolution sensors without a combinatorial explosion.

Experimental Setup

The system was evaluated using a motorized pan-tilt platform in a laboratory environment.

  • Natural Observations: The robot observed standard laboratory activities to gather visual data.
  • Self-Generated Movements: Training data was produced through random but precise movements of the camera platform.
  • Simulated Distortions: To test robustness, researchers applied arbitrary scrambling and imaging distortions to the input, forcing the system to learn the underlying geometry despite the lack of visual coherence.

Results and Discussion

The experimental results highlight the efficiency of the connectionist approach:

  • Learning Progression: Predictive capabilities began to emerge after only 100 repetitions of specific movements.
  • Refinement: Accuracy improved significantly after approximately 1,964 examples.
  • Full Calibration: High-quality calibration was achieved after roughly 40,000 movements, representing about three days of continuous learning.

Key Insights

A core contribution of this work is the concept of a motor-based metric. Space is defined not by absolute units like millimeters, but by the motor commands required to shift a pixel from one position to another. This approach allows the system to remain robust even when input pixels are randomly scrambled, provided the scrambling remains consistent. Furthermore, IRV demonstrates predictive vision; by accurately predicting the subsequent frame after a move, the system confirms its understanding of its own geometry.

Conclusion

IRV provides a biologically plausible model for how systems can develop visual stability and hand-eye coordination. While the system is currently limited by the resolution of the peripheral input and assumes a relatively static environment, it successfully demonstrates the ability to "visualize" details beyond the current field of view. Future work will explore the integration of information in dynamic environments where objects move independently of the camera.

Summary updated

BibTeX
@inproceedings{prokopowicz1995irv,
  author = {Peter N. Prokopowicz and Paul R. Cooper},
  title = {{IRV: Learning to Integrate Visual Information Across Camera Movements}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1995},
}
Figure