Deep Learning to See - Towards New Foundations of Computer Vision

Alessandro Betti, Marco Gori, Stefano Melacci2022 · DOI

Abstract This paper proposes a radical shift in computer vision: moving away from massive, human-labeled image datasets toward an unsupervised "natural" learning protocol based on motion. By treating video as a continuous stream rather than a collection of shuffled frames, the authors introduce a "Vision Field Theory" where object identity and function (affordance) emerge naturally from the principles of motion invariance.

1. Introduction

The current "reign of computer vision" is built on supervised learning—training deep neural networks on millions of static, human-labeled images like ImageNet. The authors argue that this approach is fundamentally "unnatural" and computationally more difficult than the problem Nature solved. In the standard paradigm, temporal coherence—the "thread" that sews video frames together—is discarded, forcing machines to rely solely on spatial regularities to recognize objects.

In contrast, biological systems (like eagles or human infants) learn to see by interacting with a moving world. They do not require millions of labels; they use motion as a "hidden supervisor." This paper provides the mathematical and theoretical foundations for a vision system that learns "in the wild" (en plein air) by exploiting the physics of motion.

2. Vision Field Theory

The authors introduce Vision Field Theory, which models visual features and their velocities as interconnected fields. The core of their approach rests on two new principles:

  • Material Point Invariance (MPI): This principle states that the identity of a "material point" on an object remains constant as it moves. By enforcing that visual features must be invariant along the trajectories of motion (optical flow), the system can discover object identities without external labels.
  • Coupled Motion Invariance (CMI): This extends invariance to "affordances"—the functional use of an object. For example, a chair's "sit-ability" is invariant relative to the movement of the person sitting on it. CMI allows a system to learn what an object is for by observing how it interacts with other moving entities.

3. Foveated Neural Networks and the Nocturnal Bottleneck

To implement these principles, the authors propose Foveated Neural Networks (FNNs). Unlike standard Convolutional Neural Networks (CNNs) that use "weight sharing" across the entire image, FNNs mimic the human eye's fovea. They have high resolution at a "Focus of Attention" (FOA) and lower resolution in the periphery.

This architecture addresses what the authors call the "Nocturnal Bottleneck." Referencing evolutionary biology, they explain why foveated vision is a superior computational strategy for diurnal animals. It allows them to handle high-resolution details without overwhelming the brain's processing power. This variable resolution, combined with a Principle of Least Cognitive Action, allows the network to learn features through a variational framework that minimizes "prediction error" over time.

4. Motion as the Protagonist

Motion is not just a task (like tracking); it is the primary source of information for learning what things are. Even when looking at a still image, our eyes make micro-saccades, effectively turning a static scene into a moving signal. The authors suggest that the standard CNN practice of weight sharing is biologically implausible and limits the ability of a network to learn complex, motion-dependent invariances.

Furthermore, the system utilizes Predictive Coding as Regularization. The system learns by trying to predict the next frame. If a feature is truly "motion invariant," it becomes a reliable tool for predicting how the scene will look a few milliseconds into the future.

5. FOA-based Supervision and Results

As a foundational theory paper, the results are primarily mathematical proofs and conceptual frameworks. One key result is the demonstration of FOA-based Supervision. The authors show theoretically that a single "point" of supervision (e.g., a label for one frame) can be propagated across all pixels and frames through motion. This provides a massive amount of "free" training data for the agent, far exceeding what is possible with static labeling.

The theory provides a single framework (Vision Field Theory) that accounts for both object recognition (the "what" / ventral stream) and spatial localization/action (the "where" / dorsal stream), effectively unifying perception and action.

6. Limitations and Future Work

The authors acknowledge several hurdles. Solving the variational equations for Vision Field Theory is computationally intensive and requires specialized "online" gradient-based learning algorithms. Additionally, in an online learning setting, an agent might suffer from the "forgetting" problem—losing previously acquired knowledge if the current visual input is uninformative. Finally, while virtual environments are the proposed training ground, ensuring that features learned in simulation transfer to the noisy real world remains a significant challenge.

Authors: Alessandro Betti, Marco Gori, Stefano Melacci (University of Siena, 2021)

Summary updated

BibTeX
@misc{betti2022deep,
  author = {Alessandro Betti and Marco Gori and Stefano Melacci and Alessandro Betti and Stefano Melacci},
  title = {{Deep Learning to See - Towards New Foundations of Computer Vision}},
  year = {2022},
  doi = {10.1007/978-3-030-90987-1},
}
Figure