Software-Defined Imaging: A Survey

Suren Jayasuriya, Odrika Iqbal, Venkatesh Kodukula, Victor Isaac Torres Muro, Robert LiKamWa, Andreas SpaniasProc. IEEE 2023 · OpenReview

Abstract

Huge advancements have been made over the years in terms of modern image-sensing hardware and visual computing algorithms (e.g., computer vision, image processing, and computational photography). However, to this day, there still exists a current gap between the hardware and software design in an imaging system, which silos one research domain from another. Bridging this gap is the key to unlocking new visual computing capabilities for end applications in commercial photography, industrial inspection, and robotics. In this survey, we explore existing works in the literature that can be leveraged to replace conventional hardware components in an imaging system with software for enhanced reconfigurability. As a result, the user can program the image sensor in a way best suited to the end application. We refer to this as software-defined imaging (SDI), where image sensor behavior can be altered by the system software depending on the user’s needs. The scope of our survey covers imaging systems for single-image capture, multi-image, and burst photography, as well as video. We review works related to the sensor primitives, image signal processor (ISP) pipeline, computer architecture, and operating system elements of the SDI stack. Finally, we outline the infrastructure and resources for SDI systems, and we also discuss possible future research directions for the field.

Introduction

Traditional imaging systems are typically designed as "black boxes." Hardware engineers build sensors with fixed behaviors, and software engineers develop algorithms that treat the sensor as a static source of pixels. This lack of co-design creates a massive efficiency gap. For instance, a computer vision algorithm might only need to track a small object, yet the sensor continues to read out millions of irrelevant pixels, wasting energy and bandwidth. Furthermore, switching sensor modes currently suffers from high latency due to bottlenecks in the operating system and driver stack.

Software-Defined Imaging (SDI) is an emerging paradigm that breaks down these silos. By making every layer of the imaging stack—from physical sensor primitives to the operating system—programmable and reconfigurable, SDI enables imaging systems to be dynamically optimized for specific tasks like autonomous navigation or high-speed industrial inspection.

Programmable Sensor Mechanisms and Methods

The SDI stack introduces reconfigurability at several levels. At the sensor level, programmable mechanisms allow for selective readout, such as defining a Region of Interest (ROI) to focus bandwidth on salient parts of a scene.

Figure 1: Programmable sensor mechanisms including windowing, subsampling (binning and skipping), and ROI characterization.

Key areas of research reviewed in the SDI stack include:

  • Programmable Sensor Primitives: Techniques for selective readout, coded exposure (using "fluttered" shutters), and adaptive quantization.
  • ISP Pipelines: Research into unified optimization frameworks like FlexISP and "ISP-free" pipelines that allow machine learning models to process RAW data directly, avoiding the energy cost of traditional demosaicing.
  • Software Abstractions: Domain-specific languages (DSLs) like Halide and Darkroom that allow developers to write high-performance image processing code portable across CPUs, GPUs, and FPGAs.

Key Insights: The "Pretty Image" Tax

A core insight of SDI is that traditional ISPs are designed to make images look good to the human eye. However, for machine vision, these steps are often unnecessary and can destroy useful information. Skipping the ISP can save up to 75% of a system's energy.

Additionally, SDI addresses motion blur through "Coded Exposure." Instead of trying to fix blur with software after the fact, SDI uses software to control the sensor's shutter pattern during exposure, making the blur mathematically easier to reverse.

Figure 4: Comparison of shutter patterns and resulting images. Coded exposure (d) preserves high-frequency details compared to traditional long exposure (b).

Results and Future Directions

Synthesis of recent literature shows that SDI principles lead to significant gains:

  • Energy Efficiency: Moving computations into the analog/mixed-signal domain of the sensor drastically reduces the power footprint of "always-on" devices.
  • Latency: Frameworks like Banner have demonstrated that modifying the OS stack can reduce resolution-switching latency from hundreds of milliseconds to near-zero.
  • Task Performance: Deep learning models trained on RAW data frequently outperform those trained on processed RGB images, especially in challenging low-light conditions.

Future work in SDI must overcome proprietary barriers that lock sensor "knobs" and integrate with emerging technologies like event-based cameras and Quanta image sensors, which offer massive potential for speed and dynamic range but require entirely new software abstractions.

Summary updated

Abstract

Huge advancements have been made over the years in terms of modern image-sensing hardware and visual computing algorithms (e.g., computer vision, image processing, and computational photography). However, to this day, there still exists a current gap between the hardware and software design in an imaging system, which silos one research domain from another. Bridging this gap is the key to unlocking new visual computing capabilities for end applications in commercial photography, industrial inspection, and robotics. In this survey, we explore existing works in the literature that can be leveraged to replace conventional hardware components in an imaging system with software for enhanced reconfigurability. As a result, the user can program the image sensor in a way best suited to the end application. We refer to this as software-defined imaging (SDI), where image sensor behavior can be altered by the system software depending on the user’s needs. The scope of our survey covers imaging systems for single-image capture, multi-image, and burst photography, as well as video. We review works related to the sensor primitives, image signal processor (ISP) pipeline, computer architecture, and operating system elements of the SDI stack. Finally, we outline the infrastructure and resources for SDI systems, and we also discuss possible future research directions for the field.
BibTeX
@inproceedings{jayasuriya2023software,
  author = {Suren Jayasuriya and Odrika Iqbal and Venkatesh Kodukula and Victor Isaac Torres Muro and Robert LiKamWa and Andreas Spanias},
  title = {{Software-Defined Imaging: A Survey}},
  booktitle = {Proc. IEEE},
  year = {2023},
  url = {https://openreview.net/forum?id=3AqQsvy1lZ},
}
Figure