In Simultaneous Localization and Mapping (SLAM), a system must decide which frames of a video stream are important enough to be stored as "keyframes." Traditional systems often rely on simple heuristics, such as inserting a keyframe every few meters or every few seconds. However, these fixed intervals are often inefficient: they lead to "map bloat" in simple environments where little changes, and they cause tracking failures in complex or fast-moving scenarios where the system needs more data to stay localized.
To solve this, researchers have turned to information theory. By measuring the mathematical uncertainty of a robot's pose or the information gain of a new observation, systems can adaptively decide when to trigger a new keyframe.
The Problem with Heuristics
Heuristic-based selection fails because it ignores the visual complexity of the scene. As noted by Lim et al., distance-based triggers result in data density that does not scale with the actual difficulty of the tracking task. In a featureless hallway, a robot might not need a new keyframe for ten meters; in a cluttered office, it might need one every ten centimeters.
Kuo et al. argue that these manual heuristics are difficult to tune, especially when moving between different sensor configurations. A principled approach should instead look at the "information" the system currently possesses.
Measuring Uncertainty via Entropy
The most common information-theoretic metric used is entropy, which quantifies the uncertainty in the estimated camera pose. When a robot moves, its confidence in its exact position naturally degrades.
Wudenka et al. propose using the negative entropy of the robot pose as a trigger. Instead of waiting for a fixed distance, the system monitors how "blurry" its probability distribution for its location has become. When this uncertainty crosses a threshold, a new keyframe is spawned to anchor the trajectory. Similarly, Li et al. add keyframes only if their inclusion significantly reduces the mathematical uncertainty of the current pose estimate.
This approach is particularly useful for loop closure. Steinbrücker et al. use differential entropy to identify when a camera has returned to a previously mapped area. By measuring the reduction in pose uncertainty when a current frame matches an old one, the system can autonomously trigger global map corrections.
Fisher Information and Tracking Reliability
A more granular way to measure information is through the Fisher Information Matrix (FIM). The FIM describes how much information an observable variable (like a set of pixels) carries about an unknown parameter (like the camera's 6-DoF pose).
Lim et al. manage keyframe selection by monitoring the determinant of the FIM. A new keyframe is triggered only when this value indicates that tracking uncertainty has exceeded a specific threshold. This ensures that the map density is proportional to the scene's complexity. Lin et al. apply a similar logic to LiDAR-based SLAM, using Fisher information to detect an "information break"—a point where the registration between consecutive frames becomes unreliable, signaling the need for a new keyframe.
Maximizing Information Gain
While some methods focus on uncertainty, others focus on information gain: how much new information a frame adds to the map.
Yang et al. select keyframes based on the volume of previously unobserved space they reveal. This is especially effective for neural implicit representations, where the goal is to reconstruct a dense surface. If a frame sees a part of the room the system hasn't seen before, it is highly informative and should be a keyframe.
This concept can even be applied at the pixel level. Fontán et al. developed a system that selects only the ~24 most "informative" pixels to track by maximizing the expected entropy reduction of the camera pose. By prioritizing pixels that specifically constrain the camera's degrees of freedom where uncertainty is highest, the authors achieved a 10x speedup with almost no loss in accuracy.
Benefits of Information-Theoretic Selection
- Efficiency: By filtering out redundant spatial information, systems like those proposed by Islam et al. and Ye et al. prevent map bloat and reduce the computational burden of feature extraction.
- Robustness: Adaptive selection allows systems to survive challenging conditions. Wudenka et al. showed that entropy-based selection significantly reduces tracking failures in repetitive terrains, like those found in planetary missions.
- Generality: As Kuo et al. demonstrate, using pose uncertainty derived from the FIM allows for a sensor-agnostic process. The same logic can be applied whether a robot has two cameras or six, without needing to manually retune thresholds.
Key Foundational Work
Large-Scale Multi-resolution Surface Reconstruction from RGB-D Sequences (2013) — contributed 2 concept elements to this direction.
Real Time Localization and 3D Reconstruction (2006) — contributed 1 concept elements to this direction.