Practical Scheduling of Computer Vision Functions

Adrien Chan-Hon-Tong, Stéphane HerbinVISIGRAPP (6: VISAPP) 2017 · DOI

TL;DR

This paper introduces a novel scheduling framework for computer vision functions, specifically addressing the challenge of efficiently processing batches of images under strict time constraints. By treating various computer vision classifiers as "black boxes" and employing a look-ahead planning approach with virtual states, the authors demonstrate a practical solution that significantly outperforms traditional baseline methods in terms of precision while meeting performance targets.

What Problem Does This Paper Address?

Developing sophisticated robotic platforms and intelligent systems often requires integrating a diverse array of computer vision functions, such as image classification, object detection, and segmentation. While many specialized algorithms exist for each task, they often come with trade-offs between accuracy, speed, and computational resources. This creates a significant challenge: how to dynamically schedule these functions to process large volumes of data (e.g., batches of images) within strict time limits, maximizing overall performance. Traditional scheduling approaches struggle with the inherent variability and data-dependent nature of computer vision tasks, where the performance of a function can vary drastically based on the input data. This paper aims to bridge this gap by proposing a practical scheduling framework that can adapt to these complexities.

Methods

The core of the proposed solution lies in a Look-Ahead Planning (LAP) strategy. Instead of pre-computing all possible outcomes, the system uses LAP to explore potential action sequences and select the most promising one. To handle the computational burden of exploring all possibilities, the authors introduce two key innovations:

  • Greedy Pruning: This technique limits the search space by prioritizing more promising paths, effectively maintaining a low computational volume while still leveraging long-term information.
  • Virtual States: This is a crucial departure from traditional methods. Instead of performing actual computations to determine the outcome of an action, the system uses a "virtual representation" of the state (e.g., distributions of labels and confidences). This allows for a coarse estimation of the state after an action without incurring the significant computational cost of a real execution. This virtual state concept allows the scheduler to operate within the LAP framework, making the scheduling process itself computationally inexpensive.

The system encapsulates each computer vision classifier as an "elementary action," characterized by its expected quality, speed, and resource consumption. A set of these elementary actions that can be run simultaneously forms an "action." The scheduler then iteratively selects actions based on the LAP strategy, aiming to optimize a defined "score" (in this case, expected precision).

Datasets

The experiments were conducted using the 2015 IEEE GRSS Data Fusion Contest dataset (grss_dfc_2015). This dataset consists of large remote sensing images (10000x10000 pixels) with a resolution of approximately 5cm. For the experiments, the dataset was split into training, validation, and testing sets. The computer vision functions were calibrated on the validation set, and the entire system was tested on the test set.

Key Insights and Ideas

  • The "Black Box" Approach is Practical: Treating complex computer vision functions as black boxes with known (calibrated) performance characteristics simplifies the scheduling problem significantly, making it tractable.
  • Virtual States are Key to Real-Time Scheduling: The ability to estimate the outcome of an action without performing it is critical for achieving real-time scheduling in dynamic environments like computer vision. This avoids the computational bottleneck of full simulations.
  • Complementarity of Classifiers: The experiments revealed that the scheduler effectively exploited the complementarity of different classifiers, leading to better overall performance than individual classifiers or simple combinations. This suggests that a scheduler can uncover synergistic benefits that might not be apparent from individual classifier evaluations.
  • Beyond Prototyping: While initially conceived to aid in prototyping and integration, the proposed scheduler achieved state-of-the-art results, demonstrating its potential for high-performance applications.

Key Results

The paper reports significant improvements in F₁ measure (a metric combining precision and recall) compared to various baseline methods. The proposed scheduler achieved an F₁ score of 50%, while baselines ranged from 2% to 9%. Notably, the scheduler maintained a recall of around 65% (a slight decrease from baselines' 90%) but achieved a substantial increase in precision. All systems completed within the allowed time constraint (1 hour for classifying a city), with the scheduling process itself adding less than 5% overhead.

Limitations and Future Work

The authors acknowledge that the current framework is primarily focused on time-constrained batch classification. While the results are promising, they suggest several avenues for future work:

  • Semi-Automated Pipeline Design: Developing a system where users define a cost function, and the scheduler automatically constructs an optimal computer vision pipeline.
  • Feedback Loops: Integrating feedback mechanisms between computer vision functions and camera devices to dynamically adjust image acquisition parameters.
  • Mutual Calibration: Exploring methods for calibrating classifiers based on their interactions within a cascaded system, rather than individually. This could further improve performance but presents challenges in terms of computational cost and memory.

Summary updated

BibTeX
@inproceedings{chanhontong2017practical,
  author = {Adrien Chan-Hon-Tong and Stéphane Herbin},
  title = {{Practical Scheduling of Computer Vision Functions}},
  booktitle = {VISIGRAPP (6: VISAPP)},
  year = {2017},
  doi = {10.5220/0006086303470352},
}
Figure