TL;DR
This paper introduces a novel "non-separable spatiotemporal" (non-SPAM) filter inspired by the human retina's efficient image processing. This filter decomposes images into time-varying components, enabling a progressive reconstruction method that improves image quality as more data is utilized, mimicking how our visual system processes information over time.
What Problem Does This Paper Address?
The paper tackles the ever-increasing demand for efficient image and video compression. While current standards like H.265/HEVC have improved, their progress is slow compared to the explosion of data. This necessitates groundbreaking approaches beyond mere parameterization. The authors propose that the human retina, with its sophisticated mechanism for capturing, transforming, and compressing visual information, offers a compelling model for developing more efficient compression techniques. Specifically, they aim to mimic the retina's ability to extract essential information for dynamic encoding/decoding systems, even from static images presented over time.
Methods
The core of this work is the non-separable spatiotemporal (non-SPAM) filter. This filter is inspired by retinal cells (photo-receptors and horizontal cells) that act as edge and motion detectors. Unlike traditional separable filters, the non-SPAM filter's spatial behavior varies with time, allowing it to detect temporal luminance variations even in uniform regions.
The filter's construction is based on a time-varying Difference of Gaussians (DoG) pyramid. The filter's output, termed "activation degree," is a spatiotemporal convolution of the input image with the non-SPAM filter. For a static image presented over time, this convolution simplifies to a spatial convolution with a time-varying DoG filter. This time-varying nature is achieved by multiplying spatial Gaussian filters with temporal filters (Re(t) and Rs(t)).
For practical implementation, the continuous filter is discretized. The paper then establishes that this discretized non-SPAM filter forms a frame. This frame property is crucial for the progressive reconstruction of the input image. The reconstruction process involves using a limited subset of the computed coefficients, with the image quality improving as more coefficients are utilized. This progressive reconstruction is achieved using a gradient descent algorithm to solve a least squares problem, inspired by the Rank-Order-Coding (ROC) model.
Datasets
Not discussed in the paper. The authors use a single still image for demonstration purposes.
Key Insights and Ideas
- Retinal Inspiration for Compression: The human retina's layered structure and selective information transmission provide a powerful biological blueprint for developing efficient image compression algorithms.
- Time-Varying Spatial Filters: The non-SPAM filter's novelty lies in its ability to adapt its spatial characteristics over time, allowing for richer information extraction than static filters.
- Frame Theory for Reconstruction: Establishing the non-SPAM filter as a frame is key to guaranteeing perfect reconstruction and enabling progressive decoding.
- Progressive Reconstruction Mimics Visual Perception: The method of reconstructing images by progressively adding more coefficients mirrors how our visual system builds a coherent perception of a scene over time.
Key Results
- The non-SPAM filter successfully decomposes a still image into time-varying components, visualized as low-pass and band-pass filtered versions over time.
- The progressive reconstruction method demonstrates that image quality significantly improves as the percentage of utilized coefficients increases. For instance, using only 40% of coefficients allows for recognition of the image's basic structures, while 100% yields near-perfect reconstruction (MSE of 0.3884 for a 64x64 image).
- The theoretical framework establishes the non-SPAM filter as a frame, ensuring its invertibility for reconstruction.
Limitations and Future Work
- Limited to Still Images: The current work focuses on static images presented over time. A significant future direction is to extend this filtering and coding approach to video streams.
- Parameter Tuning: While parameters are tuned based on neuroscientific results, further optimization for specific compression tasks might be beneficial.
- Computational Complexity: While the paper mentions avoiding direct computation of the inverse matrix operator for reconstruction, the gradient descent approach for minimization might still have computational implications for real-time applications.
The authors explicitly state their aim to extend this filtering result to video streams, suggesting this is their primary future research direction.
Summary updated