Learning Visual Features Under Motion Invariance

Alessandro Betti, Marco Gori, Stefano Melacci

Abstract

Humans are continuously exposed to a stream of visual data with a natural temporal structure. However, most successful computer vision algorithms work at image level, completely discarding the precious information carried by motion. In this paper, we claim that processing visual streams naturally leads to formulate the motion invariance principle, which enables the construction of a new theory of learning that originates from variational principles, just like in physics. Such principled approach is well suited for a discussion on a number of interesting questions that arise in vision, and it offers a well-posed computational scheme for the discovery of convolutional filters over the retina. Differently from traditional convolutional networks, which need massive supervision, the proposed theory offers a truly new scenario for the unsupervised processing of video signals, where features are extracted in a multi-layer architecture with motion invariance. While the theory enables the implementation of novel computer vision systems, it also sheds light on the role of information-based principles to drive possible biological solutions.

Introduction

Modern computer vision has achieved remarkable success through "intensive supervision," relying on millions of manually labeled images to train deep neural networks. However, this paradigm stands in stark contrast to biological systems, which learn to recognize objects with minimal external labeling. The authors argue that the current AI paradigm fails because it treats vision as a collection of static, unrelated frames, effectively discarding the rich temporal structure inherent in natural visual streams.

In nature, visual data is continuous. If an object moves across a field of vision, its identity remains constant despite changes in pixel-level representation. This temporal continuity provides a massive amount of "free supervision." This paper proposes a shift toward a physics-inspired theory of vision based on motion invariance, deriving a set of "laws of visual features" that enable unsupervised learning and provide a mathematical justification for hierarchical, receptive-field-based architectures.

The Principle of Least Cognitive Action

To move away from standard discrete backpropagation, the authors look toward variational principles used in analytical mechanics. They define a "cognitive action"—a functional that the learning system seeks to minimize. This action balances three primary goals:

  1. Information Transfer: Maximizing the mutual information between the input video and the extracted features to ensure the features are representative of the environment.
  2. Parsimony: Ensuring that features change smoothly over space and time, acting as a form of regularization.
  3. Motion Invariance: The "adiabatic" condition, which requires that a feature's value remains constant along the trajectory of a moving pixel.

Minimizing this action leads to a system of temporal differential equations. Instead of updating weights in discrete steps, the filters in this model evolve continuously according to these laws.

Spatial Locality and Receptive Fields

One of the most significant theoretical contributions of the paper is the mathematical derivation of receptive fields. Initially, the learning equations are "non-local," making them computationally intractable. The authors prove that by introducing receptive fields—limiting a filter's view to a small, local area—the system becomes spatially local and computationally efficient. This provides a rigorous mathematical justification for why both biological brains and convolutional neural networks (CNNs) utilize local connectivity; it is a necessity for turning complex, global integro-differential equations into manageable local ones.

Experimental Results and Insights

The theory was validated through unsupervised feature discovery on several complex video sources, including the Hollywood Dataset (HOHA2), clips from The Matrix, and the BDD100K (Berkeley DeepDrive) dataset.

Semantic Labeling

On the BDD100K dataset, features learned via motion invariance (referred to as "cal" models) were used to predict semantic classes such as "road," "building," and "sky." The results demonstrated that:

  • Unsupervised Success: The system successfully discovered convolutional filters from raw video without any human-provided labels.
  • Performance: The motion-invariant features outperformed standard unsupervised methods like Sparse Autoencoders.
  • Depth Benefits: Adding layers to the motion-invariant architecture consistently improved representation quality. For instance, a 7-layer "cal-rgb" model achieved a Mean Intersection over Union (IoU) of 38.47, significantly outperforming the baseline RGB-only model's 34.85.

Biological Parallels

The model also offers insights into biological phenomena. The authors demonstrate a "blurring" strategy: starting with low spatial resolution (similar to the poor visual acuity of newborns) helps the differential equations converge more robustly. Furthermore, the model requires periodic "resets" to maintain stability, suggesting a possible mathematical explanation for NREM sleep phases, where the absence of visual input allows internal dynamics to stabilize.

Conclusion and Future Work

By treating motion invariance as the "master" invariance—from which scale, rotation, and translation invariance naturally emerge—the authors provide a robust framework for unsupervised visual learning. While the current model focuses on the "what" pathway (ventral stream) and assumes a known velocity field (optical flow), future work aims to integrate the discovery of optical flow directly into the cognitive action and scale the system to larger, more diverse datasets. This research bridges the gap between the principles of physics and the mechanisms of computer vision, offering a path toward more efficient, biologically plausible AI.

Summary updated

Abstract

Humans are continuously exposed to a stream of visual data with a natural temporal structure. However, most successful computer vision algorithms work at image level, completely discarding the precious information carried by motion. In this paper, we claim that processing visual streams naturally leads to formulate the motion invariance principle, which enables the construction of a new theory of learning that originates from variational principles, just like in physics. Such principled approach is well suited for a discussion on a number of interesting questions that arise in vision, and it offers a well-posed computational scheme for the discovery of convolutional filters over the retina. Differently from traditional convolutional networks, which need massive supervision, the proposed theory offers a truly new scenario for the unsupervised processing of video signals, where features are extracted in a multi-layer architecture with motion invariance. While the theory enables the implementation of novel computer vision systems, it also sheds light on the role of information-based principles to drive possible biological solutions.
BibTeX
@article{betti2020learning,
  author = {Alessandro Betti and Marco Gori and Stefano Melacci},
  title = {{Learning Visual Features Under Motion Invariance}},
  journal = {Neural Networks},
  year = {2020},
  doi = {10.1016/j.neunet.2020.03.013},
}
Figure