Development of an Image Quality Improvement Device for Electrical Flip Board using AR Technology

Hiroyuki Ando, Yuki I. Kawamura, Kenichi HayashiThe Journal of The Institute of Image Information and Television Engineers 2014 · DOI

Abstract

In modern television production, thin-panel displays integrated into studio sets—often referred to as "re-filming monitors"—are increasingly utilized to present dynamic information, reduce costs, and enhance visual variety. However, capturing these displays with studio cameras typically introduces significant image degradation, including moiré patterns, reflections, and color inaccuracies. While traditional operational adjustments offer limited relief, they often fail to meet broadcast quality standards. This paper presents a novel system that leverages Augmented Reality (AR) technology to recognize screen areas in real-time and digitally overlay high-quality source footage, effectively eliminating these artifacts.

Introduction

"Re-filming" occurs when a studio camera captures a physical monitor or tablet. This process is essential for interactive segments, weather reports, and news broadcasts. Despite its prevalence, it introduces four primary visual defects:

  1. Brightness Mismatch: Displays often appear washed out or excessively dark relative to studio lighting.
  2. Color Inaccuracy: The conversion from digital signal to light and back to a camera sensor distorts the original color profile.
  3. Reflections: Studio lighting frequently creates glare or "white-out" on screen surfaces.
  4. Moiré Patterns: Interference between the display's pixel grid and the camera's sensor grid results in distracting wavy lines.

Traditional solutions, such as anti-glare filters or iris adjustments, often compromise the overall shot quality. Our approach seeks to maintain the spatial presence of the physical monitor while delivering the visual fidelity of a direct digital feed.

System Architecture and Methods

The core innovation of this system is the replacement of the degraded filmed pixels with the original digital source material using an AR-based pipeline. Rather than attempting to filter out noise, the system treats the monitor as an AR target.

The system pipeline illustrates the flow from re-filmed and source footage to the final composite image, including lighting estimation and occlusion detection.

Markerless Tracking

To ensure the system is non-intrusive for studio sets, we implemented markerless tracking. Using the FAST (Features from Accelerated Segment Test) detector, the system identifies natural features on the screen to calculate its position and orientation (homography) without the need for physical QR codes.

Visualization of the feature matching process, where green points on the tablet are mapped to the original source image to determine spatial orientation.

Real-Time Processing Pipeline

The system operates on a high-end GPGPU-equipped PC to handle HD-SDI signals with minimal latency. The pipeline includes:

  • Lighting Estimation: The system analyzes the brightness of the physical screen to apply matching shading to the digital overlay, ensuring a natural look.
  • Occlusion Handling: A critical component that detects when a presenter or object moves in front of the screen, creating a mask to prevent the digital overlay from covering them.
  • Alpha Blending: To prevent jarring visual transitions, the overlay is phased in over several frames using temporal fading.

Results and Discussion

The prototype was evaluated using professional 60-inch and 100-inch studio displays, as well as 10-inch tablets used as "electronic flip boards."

Key performance metrics include:

  • Latency: The system maintains a fixed delay of only 2 frames, making it viable for live broadcast environments.
  • Stability: Tracking remains robust even when up to 50% of the screen is occluded or when the device is moved rapidly by a presenter.
  • Visual Fidelity: The system successfully eliminates all moiré and reflection artifacts, providing crisp text and vibrant colors that match the original source material.

Conclusion and Future Work

This AR-based image improvement device offers a significant leap in broadcast quality for re-filmed content. Future development will focus on reducing the initial recognition time (currently ~1 second), improving occlusion accuracy for presenters wearing colors similar to the screen content, and expanding support for multi-camera studio environments. Beyond quality correction, this technology holds potential for virtual displays and real-time visual effects on film sets.

Summary updated

Abstract

番組制作のスタジオセットの一部として設置され,情報を提示する薄型ディスプレイを「再撮モニタ」と呼ぶ.番組制作における演出の多様化や省資源化,低コスト化などへの要求に応じ,再撮モニタの活用機会が増加している.しかし,ディスプレイの画面をカメラで撮影した映像信号と,ディスプレイに入力している元の映像信号の間には画質劣化が存在する.この画質劣化に対する運用レベルの対策には限界があり,オンエア映像で画質劣化を目にすることも少なくない.今回,AR技術を用いることで再撮モニタを撮影した映像から画面領域を認識し,画質改善を行う装置を開発した.
BibTeX
@article{ando2014development,
  author = {Hiroyuki Ando and Yuki I. Kawamura and Kenichi Hayashi},
  title = {{Development of an Image Quality Improvement Device for Electrical Flip Board using AR Technology}},
  journal = {The Journal of The Institute of Image Information and Television Engineers},
  year = {2014},
  doi = {10.3169/itej.69.j6},
}
Figure