Learning to Select Views for Efficient Multi-View Understanding

Yunzhong Hou, Stephen Gould, Liang ZhengCVPR 2024 · DOI

Processing every camera angle in a multi-view system is computationally wasteful. This paper introduces MVSelect, a reinforcement learning-based module that sequentially picks the most informative views, achieving near-peak accuracy using only 2 or 3 cameras instead of the full set.

The Multi-View Efficiency Problem

Multi-view computer vision—using multiple cameras to look at the same scene—is essential for overcoming occlusions and limited fields of view. However, it comes with a "multi-view tax": processing $N$ images requires $N$ times the computation. This is a significant bottleneck for edge devices with limited power budgets.

While "active vision" (deciding where to look next) exists, it usually involves a sequential bottleneck that prevents GPUs from utilizing their parallel processing power effectively. The challenge is to select the "next-best-view" (NBV) to save computation without destroying the system's throughput or accuracy.

Selective View Pipelining

The authors propose Selective View Pipelining, driven by a module called MVSelect. This approach bridges the gap between sequential decision-making and parallel hardware execution.

Comparison of CPU pipelining, standard GPU parallelism, and the proposed Selective View Pipelining.

The MVSelect Module

MVSelect is a lightweight reinforcement learning (RL) agent trained via Q-learning. It starts with a random initial view, extracts features, and then decides which camera to "query" next. The agent's state representation includes a one-hot vector of previously selected cameras and the aggregated features from those views, allowing it to understand what it has seen and what information is still missing.

Maintaining Throughput

To keep the GPU busy, the system takes a cue from CPU instruction pipelining. While the system is deciding the 3rd view for "Object A," it can simultaneously process the 1st view for "Object C." This maintains high throughput despite the sequential nature of the selection process. Furthermore, MVSelect can be trained jointly with the task network (classification or detection), allowing the entire system to learn which visual features are most decisive.

Key Insights: The "Two Glances" Heuristic

A core finding of this work is that for many tasks, a second well-chosen glance is sufficient. For example, a side view of a guitar might look like a mandolin, but a quick look at the front resolves the ambiguity immediately. You don't need the other 10 angles.

The MVSelect agent resolving ambiguity between a guitar and a mandolin by selecting an informative second view.

By treating view selection as a discrete, non-differentiable action, RL allows the model to naturally discover redundancy. In experiments, the authors found they could shut off half the cameras in a 12-camera setup with minimal impact, providing a roadmap for optimizing physical camera placement.

Experimental Results

The approach was evaluated on 3D object classification (ModelNet40, ScanObjectNN) and multi-view pedestrian detection (Wildtrack, MultiviewX).

  • High Efficiency: On ModelNet40, MVSelect achieved 94.3% accuracy using only 2 views, nearly matching the 94.5% achieved by using all 12 views.
  • Detection Performance: In pedestrian detection (Wildtrack), it achieved 88.6% MODA with 3 views, compared to 90.0% with all 7 views.
  • Throughput & FLOPs: The method achieved a 1.72x to 3.03x increase in throughput and reduced computational costs to roughly $T/N$ (where $T$ is the number of selected views and $N$ is the total available).

Limitations and Future Work

While effective, the agent sometimes struggles with "instance-awareness" in detection, as it cannot know if a camera outside its current view would reveal a hidden person. Current work also assumes fixed camera layouts. Future research may explore dynamic or moving cameras, such as those on drones or robots, where the relative pose between the object and camera is constantly changing.

Summary updated

BibTeX
@inproceedings{hou2024learning,
  author = {Yunzhong Hou and Stephen Gould and Liang Zheng},
  title = {{Learning to Select Views for Efficient Multi-View Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01903},
}
Figure