Comparing upscaling algorithms from HD to Ultra HD by evaluating preference of experience

Jing Li, Yao Koudota, Marcus Barkowsky, Helene Primon, Patrick Le Callet2014 · DOI

TL;DR

This paper investigates whether current upscaling algorithms can effectively convert HD content to Ultra HD (UHD) without sacrificing visual quality. Through a subjective user study, the researchers found that simpler algorithms like Lanczos-3 often outperformed complex super-resolution methods, suggesting that UHD-specific adaptations are needed for optimal visual experience.

What Problem Does This Paper Address?

The advent of Ultra High Definition Television (UHDTV) promises a significantly enhanced viewing experience, but a major hurdle is the scarcity of native UHD content. To bridge this gap, the industry needs to upscale existing High Definition (HD) and Full HD (FHD) video to UHD resolutions. This raises a critical question: can current upscaling algorithms deliver a high-quality UHD experience, or will they introduce noticeable artifacts and degrade the perceived quality? This paper directly addresses this by evaluating the subjective preference of viewers for various upscaled UHD videos compared to native UHD content.

Methods

The core of this research is a subjective experiment employing the Paired Comparison methodology. This involves presenting viewers with two video sequences side-by-side and asking them to choose which one they prefer. The researchers upscaled 720p and 1080p video sequences to UHD resolution using a selection of upscaling algorithms. These algorithms included common ones like bilinear, bicubic, and Lanczos, as well as more advanced super-resolution techniques such as Pocs (Projections onto convex sets) and RSR (Robust Super Resolution). A "Hypothetical Reference Circuit" (HRC) condition, which combined Lanczos-3 with additional sharpening, was also included.

To analyze the results, the Bradley-Terry (BT) model was used to convert the paired comparison data into psychophysical scale values, representing the observers' preference. Statistical significance of the preferences was tested using Barnard's exact test.

Datasets

The study utilized eight source video sequences, originally in 4K resolution, which were then downscaled to 1080p and 720p for the upscaling experiments. These sequences were chosen to represent a variety of visual characteristics, including high frequency components, luminance contrast, color contrast, and motion. The sequences included: "Fountain," "Runner," "Tall Building," "Table Car," "Birthday," "Wiener," and two clips from "Tears of Steel" (ToS2 and ToS3). The paper does not introduce a new dataset but uses existing video content for evaluation.

Key Insights and Ideas

  • Simplicity can be superior: Counterintuitively, computationally less expensive algorithms like Lanczos-3 often performed better or comparably to complex super-resolution methods when upscaling to UHD. This suggests that the "state-of-the-art" in super-resolution might not directly translate to optimal perceptual quality for UHD.
  • Content matters: The effectiveness of upscaling algorithms is highly dependent on the video content. For sequences with fast motion, viewer preferences were less distinct, while for content with high frequency components and slow motion, the benefit of 4K resolution was more apparent.
  • UHD viewing is different: The paper highlights that human visual perception at UHD resolutions and viewing distances differs from HD. Factors like peripheral vision, center bias, and the increased immersive feeling can influence how viewers perceive artifacts and overall quality.
  • Sharpening can help: The combination of Lanczos-3 with additional sharpening (HRC condition) showed promising results, indicating that post-processing steps can play a significant role in enhancing perceived quality.

Key Results

  • In general conditions, the original 4K sequences were preferred over all upscaled versions.
  • Upscaled versions from 1080p were generally preferred over those upscaled from 720p.
  • Lanczos-3 and Lanczos-3 with sharpening algorithms performed best among the tested upscaling methods.
  • More complex super-resolution algorithms like Pocs and RSR performed the worst in terms of subjective preference.
  • For fast-motion content, viewer preferences between different upscaled sequences were not significantly different.
  • Significance tests revealed that observers' perceptual sensitivity to upscaled sequences was higher for 720p than for 1080p.

Limitations and Future Work

The authors acknowledge that the performance of upscaling algorithms is content-dependent and that human visual perception at UHD resolutions has unique characteristics. They suggest that future work should focus on developing UHDTV-oriented upscaling algorithms that are specifically adapted to the mechanisms of the human visual system, considering factors like visual field, center bias, and peripheral vision. This could lead to algorithms that better leverage the benefits of UHD while mitigating potential artifacts.

Summary updated

Abstract

As the next generation of TV, Ultra High Definition Television (UHDTV) is attracting more and more people's attention as it provides a new viewing experience. Considering content delivery, due to the lack of Ultra HD resources, a direct question for the industry is that whether the state-of-the-art upscaling algorithms can be utilized to upscale the current HD or Full HD resources to UHD, gaining benefit from the higher resolution but without losing the high quality viewing experience. To investigate this, in this study, we upscaled 720p and 1080p sequences to UHD resolution by different upscaling algorithms. Paired Comparison methodology was used in the subjective experiment to evaluate their performances. The results showed that for the case of fast motion content, viewers' preference on different upscaled video sequences is not significantly different. In general conditions, the low complexity upscaling algorithms (e.g., lanczos-3) performed better than the high complexity algorithms (e.g., Robust Super Resolution algorithm). A novel upscaling algorithm adapted to UHD is recommended to be developed based on the mechanisms of human visual system.
BibTeX
@misc{li2014comparing,
  author = {Jing Li and Yao Koudota and Marcus Barkowsky and Helene Primon and Patrick Le Callet},
  title = {{Comparing upscaling algorithms from HD to Ultra HD by evaluating preference of experience}},
  year = {2014},
  doi = {10.1109/qomex.2014.6982320},
}

Loading related content...

Figure