Design and management of image processing pipelines within CPS: Acquired experience towards the end of the FitOptiVis ECSEL Project

Carlo Sau, Claudia Rinaldi, Luigi Pomante, Francesca Palumbo, Giacomo Valente, Tiziana Fanni, Marcos Jesús Iglesias Martínez, Frank van der Linden, Twan Basten, Marc Geilen, Geran Peeren, Jiří Kadlec, Pekka Jääskeläinen, Lubomír Bulej, Francisco Barranco, Jukka Saarinen, Tero Säntti, Maria Katiuscia Zedda, Víctor Sánchez Ribes, Shayan Tabatabaei Nikkhah, Dip Goswami, Guillermo Amat, Lukáš Maršík, Mark van Helvoort, Luis Medina, Zaid Al-Ars, Ad de BeerMicroprocessors 2021 · DOI

Abstract

The FitOptiVis project addresses the critical need for efficient image and video processing in Cyber-Physical Systems (CPS). By introducing a unified reference architecture and the Quality and Resource Management Language (QRML), the project facilitates the management of complex, distributed pipelines. This paper details the methodologies used to bridge design-time optimization with run-time adaptation, enabling systems to dynamically balance performance and energy consumption. The framework's effectiveness is demonstrated through ten diverse industrial use cases, ranging from autonomous drones to medical imaging.

1. Introduction

Modern Cyber-Physical Systems (CPS)—including autonomous vehicles, smart grids, and robotic surgical assistants—increasingly depend on sophisticated "visual awareness" for real-time decision-making. However, image processing pipelines are resource-intensive, demanding high computational power, low latency, and high throughput. These requirements often clash with the strict thermal and energy constraints of battery-powered edge devices.

The primary challenge lies in the heterogeneity and dynamism of these systems. A single pipeline may be distributed across smart cameras (edge), local gateways (fog), and remote servers (cloud), each utilizing different hardware architectures (FPGAs, GPUs, CPUs). Traditional design methods are often static and fail to adapt to fluctuating network bandwidth or decreasing battery levels. The FitOptiVis project seeks to provide a standardized framework for building self-optimizing systems that can adapt to these changing environments.

2. Methodology

The FitOptiVis framework employs a multi-layered approach to manage complexity and optimize performance:

  • Quality and Resource Management Language (QRML): This Domain-Specific Language (DSL) allows developers to characterize components based on their resource requirements (e.g., memory, bandwidth) and output qualities (e.g., latency, accuracy). Utilizing Pareto algebra, QRML identifies optimal trade-offs between competing objectives.
  • Reference Architecture and Virtualization: By utilizing "Virtual Execution Platforms," the project abstracts underlying hardware. This enables seamless task migration across different hardware types (e.g., from an edge FPGA to a cloud GPU) without requiring application logic modifications.
  • MAPE-k Adaptation Loop: Adopting principles from autonomic computing, the system implements a Monitor-Analyze-Plan-Execute loop. This mechanism continuously monitors performance and dynamically switches between pre-defined "working points" to maintain system constraints.
  • PoCL-R (Portable Computing Language - Remote): An extension of OpenCL, PoCL-R allows devices to offload heavy computations to remote servers transparently, treating remote resources as if they were local.

3. Industrial Validation

The framework was validated across ten diverse industrial Use Cases (UCs), demonstrating its versatility:

  1. Water Supply Monitoring: Utilizing drones and smart cameras for infrastructure protection.
  2. Virtual Reality: Enabling low-latency 3D video for mobile AR/VR applications.
  3. Habit Tracking: Implementing privacy-preserving monitoring for elderly care.
  4. 3D Industrial Inspection: Facilitating high-speed quality control in manufacturing.
  5. Road Traffic Surveillance: Performing edge-based license plate recognition.
  6. Multi-Source Medical Streaming: Compositing video in surgical environments.
  7. Sustainable MRI: Reducing energy consumption and thermal load in medical imaging.
  8. Robotic Calibration: Providing non-invasive, automated calibration for robotic arms.
  9. Smart-Grid Surveillance: Protecting energy infrastructure through deterministic networking.
  10. Autonomous Exploration: Enabling adaptive video processing for planetary rovers.

4. Discussion and Key Insights

The project yielded several critical insights into the design of distributed CPS:

  • Compositionality: Optimization must consider the system as a whole rather than in isolation. By requiring components to expose resource needs via QRML, the system can construct pipelines that are "optimal by construction."
  • Edge-Cloud Continuum: The distinction between local and remote processing should be transparent to the application. PoCL-R is a vital enabler for this seamless integration.
  • Integrated Design and Run-Time: Effective optimization requires a combination of design-time analysis and run-time management. FitOptiVis uses design-time tools to generate a "menu" of optimal configurations, which the run-time manager selects based on real-time conditions.

5. Results

The FitOptiVis project resulted in a library of 26 mature components (hardware, software, and communication) with Technology Readiness Levels (TRL) reaching 6 or 7. Notable performance achievements include:

  • Computational Speedup: The DTRiMC tool achieved a 5x speedup in floating-point matrix multiplication compared to standard ARM Cortex A53 processors.
  • Bandwidth Optimization: In the smart-grid use case, the system successfully reduced bandwidth by transmitting high-resolution video only upon detecting an intruder.
  • Standardization: The successful adoption of QRML across all ten use cases proves its broad applicability.

6. Conclusion and Future Work

While the FitOptiVis framework provides a robust foundation for optimizing CPS image pipelines, future work is needed to expand tool coverage and transition prototypes into commercial products. Additionally, while the current focus is on performance and energy, future iterations of the QRML framework should explicitly incorporate reliability and security as primary optimization objectives.

Summary updated

BibTeX
@inproceedings{sau2021design,
  author = {Carlo Sau and Claudia Rinaldi and Luigi Pomante and Francesca Palumbo and Giacomo Valente and Tiziana Fanni and Marcos Jesús Iglesias Martínez and Frank van der Linden and Twan Basten and Marc Geilen and Geran Peeren and Jiří Kadlec and Pekka Jääskeläinen and Lubomír Bulej and Francisco Barranco and Jukka Saarinen and Tero Säntti and Maria Katiuscia Zedda and Víctor Sánchez Ribes and Shayan Tabatabaei Nikkhah and Dip Goswami and Guillermo Amat and Lukáš Maršík and Mark van Helvoort and Luis Medina and Zaid Al-Ars and Ad de Beer},
  title = {{Design and management of image processing pipelines within CPS: Acquired experience towards the end of the FitOptiVis ECSEL Project}},
  booktitle = {Microprocessors},
  year = {2021},
  doi = {10.1016/j.micpro.2021.104350},
}
Figure