Traditional computer vision is largely passive. Whether performing object recognition or 3D reconstruction, models typically process a fixed set of images or a single static snapshot. This approach often fails when faced with heavy occlusions or visual ambiguities—problems that a human would solve simply by moving their head or walking around an object.
A growing body of research treats perception as an active, sequential decision-making process. By using reinforcement learning (RL), agents can learn to "glance" at a scene, identify what information is missing, and strategically select the next-best viewpoint to resolve uncertainty.
The Next Best View Problem
The core challenge in active perception is the "Next-Best View" (NBV) problem: given a partial understanding of a scene, which perspective will provide the most useful new information?
Instead of using exhaustive sampling or fixed geometric paths (like a circular drone flight), researchers like Xiaoguang Han and Bo Dong frame this as an RL task. In their work on 3D point scene completion, an agent observes a partial 3D model and selects virtual viewpoints to perform 2D inpainting. The agent is rewarded for discovering missing geometry, effectively learning to "hunt" for occluded regions rather than wasting computation on already-visible surfaces.
Efficiency through Selective Attention
Processing every available camera angle in a multi-view system is computationally expensive and often redundant. Yunzhong Hou, Stephen Gould, and Liang Zheng addressed this with MVSelect, a module that sequentially picks the most informative views from a fixed array.
The intuition here is that once a model has a "good enough" representation to make a confident prediction, it should stop looking. By treating view selection as a discrete decision process, MVSelect achieves near-peak accuracy in object classification and pedestrian detection using only two or three "glances," drastically reducing the overhead of multi-camera systems.
Resolving Ambiguity and Occlusion
Active perception is particularly powerful for tasks where depth and occlusion create inherent ambiguity, such as 3D human pose estimation. Erik Gärtner and Cristian Sminchisescu developed Pose-DRL, where an observer agent learns to move its viewpoint to minimize depth ambiguity.
Similarly, in Embodied Visual Recognition, agents are trained to navigate around obstacles to uncover hidden features of target objects. Rather than trying to "hallucinate" what is behind a wall, the agent learns a policy where the reward is tied directly to recognition accuracy. This forces the agent to prioritize viewpoints that provide complementary visual evidence.
Beyond Camera Angles: Light and Touch
The principles of active view selection extend beyond moving a camera. The same RL logic can be applied to any parameter that changes how an agent perceives the world:
- Illumination Planning: In the ReLeaPS framework, Boxin Shi and colleagues use RL to select the optimal light directions for photometric stereo. By choosing just 20 light positions instead of 100+, the system maintains high-quality 3D surface reconstruction while significantly speeding up the scanning process.
- Active Touch: Edward J. Smith and Adriana Romero demonstrated that agents can learn where to touch an object to best understand its geometry. Their active 3D shape reconstruction framework uses RL to guide a robotic finger toward visual "blind spots," combining tactile sensing with visual priors to fill in geometric gaps.
Why Reinforcement Learning?
The shift toward RL for view selection works because it allows the system to internalize the relationship between action and information gain. While traditional heuristics (like "move toward the largest hole") can work, they often fail in complex, cluttered environments.
RL agents, as seen in Embodied Question Answering by Abhishek Das and Georgia Gkioxari, develop sophisticated exploration strategies. They learn to ignore irrelevant areas and focus on perspectives that resolve the specific task at hand—whether that is answering a question, segmenting a human in a crowd (SeekNet), or finding the most aesthetically pleasing "snap angle" for a 360° panorama.