Processing every camera angle in a multi-view system is computationally wasteful. This paper introduces MVSelect, a reinforcement learning-based module that sequentially picks the most informative views, achieving near-peak accuracy using only 2 or 3 cameras instead of the full set.
The Multi-View Efficiency Problem
Multi-view computer vision—using multiple cameras to look at the same scene—is essential for overcoming occlusions and limited fields of view. However, it comes with a "multi-view tax": processing $N$ images requires $N$ times the computation. This is a significant bottleneck for edge devices with limited power budgets.
While "active vision" (deciding where to look next) exists, it usually involves a sequential bottleneck that prevents GPUs from utilizing their parallel processing power effectively. The challenge is to select the "next-best-view" (NBV) to save computation without destroying the system's throughput or accuracy.
Selective View Pipelining
The authors propose Selective View Pipelining, driven by a module called MVSelect. This approach bridges the gap between sequential decision-making and parallel hardware execution.
The MVSelect Module
MVSelect is a lightweight reinforcement learning (RL) agent trained via Q-learning. It starts with a random initial view, extracts features, and then decides which camera to "query" next. The agent's state representation includes a one-hot vector of previously selected cameras and the aggregated features from those views, allowing it to understand what it has seen and what information is still missing.
Maintaining Throughput
To keep the GPU busy, the system takes a cue from CPU instruction pipelining. While the system is deciding the 3rd view for "Object A," it can simultaneously process the 1st view for "Object C." This maintains high throughput despite the sequential nature of the selection process. Furthermore, MVSelect can be trained jointly with the task network (classification or detection), allowing the entire system to learn which visual features are most decisive.
Key Insights: The "Two Glances" Heuristic
A core finding of this work is that for many tasks, a second well-chosen glance is sufficient. For example, a side view of a guitar might look like a mandolin, but a quick look at the front resolves the ambiguity immediately. You don't need the other 10 angles.
By treating view selection as a discrete, non-differentiable action, RL allows the model to naturally discover redundancy. In experiments, the authors found they could shut off half the cameras in a 12-camera setup with minimal impact, providing a roadmap for optimizing physical camera placement.
Experimental Results
The approach was evaluated on 3D object classification (ModelNet40, ScanObjectNN) and multi-view pedestrian detection (Wildtrack, MultiviewX).
- High Efficiency: On ModelNet40, MVSelect achieved 94.3% accuracy using only 2 views, nearly matching the 94.5% achieved by using all 12 views.
- Detection Performance: In pedestrian detection (Wildtrack), it achieved 88.6% MODA with 3 views, compared to 90.0% with all 7 views.
- Throughput & FLOPs: The method achieved a 1.72x to 3.03x increase in throughput and reduced computational costs to roughly $T/N$ (where $T$ is the number of selected views and $N$ is the total available).
Limitations and Future Work
While effective, the agent sometimes struggles with "instance-awareness" in detection, as it cannot know if a camera outside its current view would reveal a hidden person. Current work also assumes fixed camera layouts. Future research may explore dynamic or moving cameras, such as those on drones or robots, where the relative pose between the object and camera is constantly changing.
Summary updated