TL;DR
This paper presents a novel FPGA-based processor that mimics the human retina to perform high-speed and energy-efficient tone mapping for High Dynamic Range (HDR) images. By employing several hardware optimization techniques, the processor achieves a throughput of 189 frames per second for 1024x768 images while consuming only 819 mW, outperforming existing state-of-the-art solutions.
What Problem Does This Paper Address?
High Dynamic Range (HDR) imaging captures scenes with a much wider range of light intensities than standard displays can reproduce. Tone mapping is a crucial post-processing step that compresses this wide dynamic range into a displayable range, preserving details in both dark and bright areas. However, existing tone mapping algorithms, often implemented in software on PCs, are computationally intensive, leading to slow processing speeds and high power consumption. This makes them unsuitable for real-time applications on embedded systems, such as autonomous vehicles or drone monitoring, which require both high throughput and low power usage. Furthermore, while some biological retina-inspired algorithms offer more natural image enhancement, their hardware implementation challenges remain.
Methods
The authors propose an FPGA-based processor that implements a biological retina-inspired tone mapping algorithm. The core of their approach lies in several hardware design techniques aimed at maximizing throughput and energy efficiency:
- Data Partition-Based Parallel Processing with S-Shape Sliding: To accelerate the 15x15 and 7x7 convolutions, the input image is partitioned, and pixels are buffered in multiple small BRAMs. This allows for simultaneous reading of pixels for dot multiplication, significantly reducing read latency. An "S-shape sliding" window strategy is employed to maximize data reuse between adjacent rows, further minimizing BRAM accesses.
- Adjacent Frame Feature Sharing: For video processing, adjacent frames often share similar statistical properties (mean and standard deviation). This technique leverages this by calculating these features for the current frame based on the previously computed values of the preceding frame, reducing the need for repeated calculations and data reads.
- Multi-Layer Convolution Pipelining: Instead of processing the 15x15 and 7x7 convolutions sequentially with intermediate buffering, this technique pipelines them. This allows the 7x7 convolution to begin processing data as soon as it's available from the 15x15 convolution, reducing power consumption associated with writing and reading intermediate data.
- Convolution Filter Compression with Zero Skipping Convolution: The authors observe that convolution filters are often symmetric and contain many zeros. They compress these filters by exploiting symmetry (reducing storage to 1/8) and using run-length encoding for consecutive identical values. Additionally, a zero-detection module skips multiplication operations when encountering zero values in the filter, further reducing computation and power.
Datasets
The paper evaluates the proposed processor using images from Mark Fairchild's HDR Photographic Survey. These images are used to compare the hardware processing results against software processing and to calculate performance metrics. No new datasets were introduced.
Key Insights and Ideas
- Biomimicry for Efficiency: The paper demonstrates that emulating biological visual processing mechanisms (like the retina's horizontal and bipolar cells) can lead to more natural image enhancement, and crucially, that these principles can be effectively translated into efficient hardware designs.
- Hardware-Centric Optimization is Key: The significant performance gains (throughput and energy efficiency) are not solely due to the algorithm but heavily rely on the clever hardware design techniques. This highlights the importance of co-designing algorithms and hardware for specialized applications.
- Data Reuse and Parallelism are Paramount: Techniques like data partitioning, S-shape sliding, and pipelining are central to achieving high throughput by maximizing data reuse and exploiting parallel processing capabilities of FPGAs.
- Exploiting Filter Properties: The observation that convolution filters are often symmetric and sparse is a valuable insight that enables significant compression and computational savings.
Key Results
- High Throughput: The proposed processor achieves a throughput of 189 frames per second for 1024x768 RGB images.
- Low Power Consumption: The design consumes 819 mW for processing 1024x768 RGB images.
- High Energy Efficiency: The energy efficiency is reported as 544,453 pixels/mW/s.
- Excellent Image Quality: The hardware processing results are comparable to software processing, with average PSNR of 82.0661 dB and SSIM of 0.9998, outperforming other compared tone mapping processors in these metrics.
- FPGA Implementation: The design is successfully implemented on a Xilinx Virtex 7 FPGA.
Limitations and Future Work
- Specific FPGA Target: The current implementation is on a Xilinx Virtex 7 FPGA. While this demonstrates feasibility, further work could explore optimizations for different FPGA architectures or even ASIC implementation for even greater efficiency.
- Algorithm Specificity: The hardware design is tailored to the specific biological retina-inspired algorithm used. Adapting these techniques to other tone mapping algorithms might require modifications.
- Real-world Testing: While the paper shows experimental setup and image results, further validation in real-world embedded applications (e.g., autonomous driving systems) would be beneficial.
The authors suggest that this processor is well-suited for high-speed and energy-constrained applications like autonomous vehicles and drone monitoring, implying that future work could focus on integrating this processor into such systems.
Summary updated