Abstract
Extracting informative representations from videos is fundamental for the effective learning of various downstream tasks. Inspired by classical works on saliency, we present a novel information-theoretic approach to discover meaningful representations from videos in an unsupervised fashion. We argue that local entropy of pixel neighborhoods and its evolution in a video stream is a valuable intrinsic supervisory signal for learning to attend to salient features. We, thus, abstract visual features into a concise representation of keypoints that serve as dynamic information transporters. We discover in an unsupervised fashion spatio-temporally consistent keypoint representations that carry the prominent information across video frames, thanks to two original information-theoretic losses. First, a loss that maximizes the information covered by the keypoints in a frame. Second, a loss that encourages optimized keypoint transportation over time, thus, imposing consistency of the information flow. We evaluate our keypoint-based representation compared to state-of-the-art baselines in different downstream tasks such as learning object dynamics. To evaluate the expressivity and consistency of the keypoints, we propose a new set of metrics. Our empirical results showcase the superior performance of our information-driven keypoints that resolve challenges like attendance to both static and dynamic objects, and to objects abruptly entering and leaving the scene.
Introduction
In computer vision, keypoints serve as sparse representations of a scene's structure or an object's state. While supervised methods for keypoint detection are highly effective, they rely on extensive manual annotation. Existing unsupervised approaches typically utilize a reconstruction-based bottleneck, attempting to "redraw" one frame from another. However, these methods often fail to capture static objects, struggle with objects entering or leaving the frame, and can have difficulty distinguishing objects from subtle backgrounds.
This paper introduces MINT (Maximum Information keypoiNTs), a framework that shifts the focus from pixel reconstruction to information flow. By treating keypoints as "information transmitters," the model learns to identify and track salient features based on their intrinsic informativeness.
Methodology
Image Spatial Entropy (ISE)
The foundation of MINT is Image Spatial Entropy (ISE), which quantifies the informativeness of pixels based on their local neighborhoods. High-entropy regions typically correspond to edges and textures characteristic of objects, while low-entropy regions represent flat, uninformative backgrounds.
MINT Architecture
The architecture utilizes an "hourglass" CNN that outputs $K$ feature maps. A spatial soft-argmax layer converts these maps into $(x, y)$ coordinates. A key innovation is the prediction of an activation status for each keypoint, allowing the model to dynamically allocate resources as objects appear or disappear from the scene.
Information-Theoretic Losses
The model is trained using several novel losses:
- Masked Entropy (ME) Loss: Guides keypoints toward high-entropy regions.
- Masked Conditional Entropy (MCE) Loss: Prioritizes moving objects when keypoint resources are constrained.
- Information Transportation (IT) Loss: Ensures temporal consistency by forcing keypoints to track the same information pattern over time.
- Regularizers: Includes an overlapping loss to maintain keypoint dispersion and an active status loss to optimize keypoint usage.
Experiments and Results
Datasets and Metrics
MINT was evaluated across four diverse environments: CLEVRER (synthetic dynamics), MIME & SIMITATE (real-world manipulations), and MAGICAL (imitation learning). The authors also introduced two new metrics: DOP (Percentage of Detected Objects) and TOP (Percentage of Tracked Objects).
Performance Analysis
MINT demonstrates significant improvements over reconstruction-based baselines like Transporter:
- Detection & Tracking: On CLEVRER, MINT achieved a DOP of 0.855, outperforming Transporter's 0.787.
- Dynamics Prediction: MINT reached an 84.4% success rate for 1-step future position prediction, compared to 74.6% for baselines.
- Robustness: MINT successfully tracks objects even when they stop moving (static objects) and handles "come-and-go" scenarios by deactivating keypoints when objects leave the frame.
Conclusion
MINT provides a robust, unsupervised alternative to reconstruction-based keypoint learning by leveraging information theory. While currently limited by 2D constraints and color sensitivity, the approach successfully addresses long-standing challenges in tracking both static and dynamic objects in complex, real-world scenes. Future work will explore extending these principles to 3D spaces and continuous latent representations.
Summary updated