Abstract
High-resolution image processing tasks, such as 4K video matting and complex photo filtering, present significant computational challenges. Traditional guided upsampling methods, like Joint Bilateral Upsampling (JBU), often suffer from blurring or bleeding artifacts. This paper presents Guided Linear Upsampling (GLU), a simple and efficient alternative that represents high-resolution pixels as linear interpolations of just two optimized low-resolution pixels. By jointly optimizing the downsampling and upsampling processes, GLU preserves fine structural details and achieves massive speedups, enabling real-time 4K video processing.
1. Introduction
Processing high-resolution imagery—particularly 4K video—is a resource-intensive endeavor. Even algorithms with linear time complexity can become bottlenecks when handling millions of pixels per frame, especially on mobile or edge hardware. To mitigate this, researchers often employ "guided upsampling": the algorithm processes a downsampled version of the image, and the result is then upscaled using the original high-resolution image as a structural guide.
While effective in theory, classic approaches like Joint Bilateral Upsampling (JBU) and Bilateral Guided Upsampling (BGU) have notable drawbacks. JBU frequently over-blurs edges, while BGU can lead to "color bleeding" where attributes from one object leak into adjacent ones. Furthermore, standard downsampling often discards thin, isolated structures (e.g., hair or whiskers) before processing begins, making them impossible to recover during the upsampling phase.
2. Guided Linear Upsampling (GLU)
The core innovation of GLU is the realization that complex pixel neighborhoods are not required to reconstruct a high-resolution point. Instead, GLU leverages the "color line model," which posits that colors within a small local patch typically fall along a line in RGB space. Consequently, any pixel in that patch can be represented using just two endpoints.
2.1 Linear Representation and Optimization
GLU represents every high-resolution pixel as a linear interpolation between two pixels selected from a $3 \times 3$ low-resolution neighborhood. The algorithm identifies the optimal two pixels and their corresponding interpolation weights by attempting to reconstruct the original source image. Because most image operators (such as dehazing or matting) are designed to preserve local structures, the parameters optimized for the source image remain highly effective for the processed target image. This "target-free" optimization allows the upsampling parameters to be precomputed once for a given source image.
2.2 Joint Downsampling Optimization
A critical component of GLU is the joint optimization of the downsampling process. Rather than using a fixed grid, GLU employs an iterative approach. If a specific high-resolution pixel proves difficult to reconstruct (indicated by high error), the algorithm replaces a low-resolution pixel in the proxy image with that high-resolution outlier. This ensures that fine, high-frequency structures—which would normally be lost during downsampling—are preserved in the low-resolution representation.
2.3 Computational Efficiency
To ensure real-time performance, the authors reduced the optimization complexity from quadratic to linear by fixing one pixel in the interpolation pair and searching only for the second. This optimization allows the method to handle high-resolution inputs with minimal latency.
3. Experimental Results
The authors evaluated GLU across a variety of tasks, including alpha matting, colorization, smoothing, and dehazing, using benchmarks such as the Rhemann et al. dataset, DIV2K, and NTIRE-19.
3.1 Quantitative and Qualitative Performance
GLU consistently outperformed JBU and BGU in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). Qualitatively, GLU demonstrated a superior ability to maintain sharp edges and thin structures. Notably, while competing methods fail at high scaling factors, GLU produces sharp, usable results even at 128x downsampling.
3.2 Speed and Real-Time Applications
On a consumer-grade GPU (GTX 1650), the "GLU-" variant processed 2K images in approximately 5.6ms and 4K images in 14.3ms. This performance enables 4K video processing at over 60 FPS. The authors also demonstrated an interactive matting system where users can apply scribbles to a 10-megapixel image and receive instantaneous feedback, despite the complexity of the underlying matting algorithm.
4. Limitations and Conclusion
GLU is highly effective when the processed output maintains the structural characteristics of the source image. However, it struggles with "new edges"—sharp structures introduced by an operator (such as certain style transfers or CycleGAN effects) that do not exist in the original guidance map. In these instances, the lack of guidance information can lead to unsmooth artifacts.
In conclusion, Guided Linear Upsampling offers a robust, high-speed solution for scaling image processing tasks. By rethinking the relationship between downsampling and upsampling through the lens of the color line model, GLU provides a practical path forward for real-time high-resolution applications.
Summary updated