An information-theoretic approach to unsupervised keypoint representation learning

Ali Younes, Simone Schaub-Meyer, Georgia Chalvatzaki

TL;DR

MINT (Maximum Information keypoiNTs) is an unsupervised framework that discovers spatio-temporally consistent visual keypoints from videos using information theory rather than image reconstruction. By treating keypoints as "information transporters" and maximizing the local image entropy they cover and transport across frames, MINT effectively tracks both static and moving objects while dynamically activating or deactivating keypoints as objects enter and leave the scene.

What Problem Does This Paper Address?

Extracting compact visual representations—like keypoints—from raw video without human supervision is crucial for downstream robotic control, visual reasoning, and dynamics prediction. Existing unsupervised keypoint discovery methods (e.g., Transporter, Video Structure) generally rely on autoencoding architectures trained to reconstruct future or difference frames.

However, reconstruction-based approaches suffer from notable shortcomings:

  • They heavily bias towards large moving regions, often ignoring salient static background elements.
  • They struggle to maintain consistency through occlusions.
  • They rely on a fixed budget of keypoints, meaning they fail when objects dynamically appear or disappear ("come-and-go" scenarios).

MINT replaces pixel-level reconstruction with an information-theoretic supervisory signal: local image entropy.

Methods

Rather than reconstructing RGB pixels, MINT tracks the flow of visual information:

Input Frame(s) ---> Hourglass CNN ---> Spatial Soft-Argmax ---> Keypoints (x, y) & Activation Status (s)
                           |
                           v
               Local Entropy Layer (KDE) ---> Information Losses (ME, MCE, IT)
  1. Differentiable Entropy Layer: A custom, CUDA-accelerated layer that uses Kernel Density Estimation (KDE) on local pixel patches (e.g., $3 \times 3$) to compute a per-pixel Shannon entropy map $E(I)$ without exploding memory usage.
  2. Keypoint Architecture: An hourglass convolutional network predicts $K$ feature maps converted into 2D coordinates via a spatial soft-argmax. Crucially, it also predicts a learned binary activation status $s_t^{(i)} \in \{0, 1\}$ for each keypoint. Keypoints are converted into thresholded Gaussian heatmaps $h_t^{(i)}$.
  3. Information-Theoretic Objectives (grounded by Fano's Inequality):
  4. Masked Entropy (ME) Loss: Maximizes the cumulative local entropy covered by active keypoints within a single frame.
  5. Masked Conditional Entropy (MCE) Loss: Uses conditional entropy between consecutive frames $E(I_t \mid I_{t-1})$ to guide keypoints toward moving entities when keypoint budgets are tight.
  6. Information Transportation (IT) Loss: Maximizes mutual information between the entropy reconstructed via keypoint transportation and the target frame entropy, ensuring keypoints track the same visual feature across time without training a decoder.
  7. Regularization Losses: An overlapping loss prevents keypoints from clumping onto the same feature, while an active status loss penalizes redundant active keypoints.

Datasets

The authors evaluated MINT across four video benchmarks:

  • CLEVRER: Synthetic videos of geometric objects colliding, used for object detection, tracking, and dynamics prediction.
  • MIME: Real-world robotic imitation dataset featuring close-up videos of human hands manipulating objects (come-and-go scenarios).
  • SIMITATE: Cluttered, wide-view videos of humans performing manipulation tasks in real environments.
  • MAGICAL: A 2D benchmark for imitation learning control across varying difficulty levels (MoveToRegion, MoveToCorner, MakeLine).

New Evaluation Metrics: Because benchmark metrics for unsupervised keypoints were lacking, the authors formulated four tracking metrics using CLEVRER object masks: DOP (Detected Object %), TOP (Tracked Object %), UAK (Unsuccessful Keypoint Assignment, penalizing keypoints on empty space), and RAK (Redundant Keypoint Assignment, penalizing over-allocation to single objects).

Key Insights and Ideas

  • Entropy as an intrinsic saliency prior: Texture changes, edges, and object silhouettes possess higher local entropy than uniform backgrounds. Optimizing keypoint positions to capture entropy provides a strong inductive bias toward meaningful object boundaries.
  • Dispensing with image decoders: By formulating "information transportation" directly in the entropy space, MINT enforces temporal consistency without training heavy generative decoders or appearance encoders.
  • Dynamic keypoint allocation: Jointly optimizing entropy coverage against an activation penalty allows the network to learn when to "spawn" or "retire" keypoints dynamically as hands or objects enter and exit the camera view.

Key Results

  • Detection & Tracking (CLEVRER): MINT achieved an 85.5% object detection rate (DOP) and 83.8% tracking rate (TOP), outperforming Transporter (78.7% DOP / 74.5% TOP) and Video Structure (56.7% DOP / 54.3% TOP). MINT reduced unsuccessful background keypoints (UAK) from 18.4 down to 0.89.
  • Dynamics Prediction: Feeding MINT keypoints into an Interaction Network (IN) yielded a 1-step dynamics prediction success rate of 84.4% (compared to 74.6% for Transporter).
  • Realistic Clutter & Movement (MIME & SIMITATE): In human demonstration videos, MINT successfully deactivated keypoints after a hand left the scene, whereas baseline models either fixed their keypoint positions or left them wandering the background.
  • Imitation Learning (MAGICAL): A frozen MINT representation matched or surpassed an end-to-end task-specific CNN agent in task completion scores (e.g., $0.80$ vs. $0.30$ on MoveToCorner test jitter).

Limitations and Future Work

  • Photometric edge cases: Because the entropy layer operates on color distributions, it struggles with transparent objects or foreground objects that share identical color profiles with the background.
  • 2D Limitation: MINT operates exclusively in image pixel space without depth or 3D reasoning.
  • Relational modeling complexity: In complex multi-object control tasks (such as MakeLine), simple interaction architectures struggled to coordinate relational policies from keypoint coordinates alone, pointing to the need for better structural pooling mechanisms.

Summary updated

Abstract

Extracting informative representations from videos is fundamental for the effective learning of various downstream tasks. Inspired by classical works on saliency, we present a novel information-theoretic approach to discover meaningful representations from videos in an unsupervised fashion. We argue that local entropy of pixel neighborhoods and its evolution in a video stream is a valuable intrinsic supervisory signal for learning to attend to salient features. We, thus, abstract visual features into a concise representation of keypoints that serve as dynamic information transporters. We discover in an unsupervised fashion spatio-temporally consistent keypoint representations that carry the prominent information across video frames, thanks to two original information-theoretic losses. First, a loss that maximizes the information covered by the keypoints in a frame. Second, a loss that encourages optimized keypoint transportation over time, thus, imposing consistency of the information flow. We evaluate our keypoint-based representation compared to state-of-the-art baselines in different downstream tasks such as learning object dynamics. To evaluate the expressivity and consistency of the keypoints, we propose a new set of metrics. Our empirical results showcase the superior performance of our information-driven keypoints that resolve challenges like attendance to both static and dynamic objects, and to objects abruptly entering and leaving the scene.
BibTeX
@inproceedings{younes2023information,
  author = {Ali Younes and Simone Schaub-Meyer and Georgia Chalvatzaki},
  title = {{An information-theoretic approach to unsupervised keypoint representation learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=oPnpibcro8},
}

Loading related content...

Figure