CompenHR: Efficient Full Compensation for High-resolution Projector

Yuxi Wang, Haibin Ling, Bingyao Huang

Introduction

Projecting images onto non-planar, textured, or colored surfaces—such as curved walls or patterned backgrounds—inevitably introduces geometric and photometric distortions. Geometric distortion warps the intended image shape, while photometric distortion alters color and brightness based on the surface's properties. While deep learning has emerged as a powerful tool for "full compensation" (addressing both distortions simultaneously), existing end-to-end models face significant scalability issues.

Current state-of-the-art (SOTA) models typically struggle with high-resolution setups (e.g., 1024×1024). Naively scaling these models leads to a "memory wall," where computational costs and VRAM requirements exceed the capacity of standard workstation GPUs. CompenHR addresses this by introducing an efficient framework that maintains high-quality compensation while drastically reducing the computational footprint.

Methodology

CompenHR decomposes the compensation task into two specialized, attention-aware sub-networks: GANet and PANet.

GANet: Attention-based Geometry Correction

Geometric correction is often too complex for a single-step transformation. GANet employs a coarse-to-fine architecture. It first utilizes a combination of affine transformations and Thin Plate Spline (TPS) grid generators to establish a "coarse" alignment. This initial grid is then refined by a dedicated grid refinement network. By integrating attention modules, GANet can focus on critical structural features, ensuring precise alignment even on complex surfaces.

PANet: Attention-based Photometric Compensation

The primary challenge in high-resolution photometric compensation is processing large images without losing high-frequency detail. Standard downsampling techniques (like pooling or strided convolutions) discard spatial information. PANet circumvents this by using a pixel unshuffle/shuffle sampling scheme.

"Unshuffling" reshapes a high-resolution image into multiple lower-resolution channels. This allows the network to process a smaller spatial footprint—saving significant memory—while preserving every pixel from the original input. After processing, a "shuffle" operation restores the image to its original resolution. Like GANet, PANet incorporates attention blocks to prioritize key features such as edges and specific color textures.

Sampling-based Reformulation

To ensure the system is end-to-end trainable, the authors mathematically reformulated the compensation process. This integration allows the sampling scheme to be part of the global optimization objective, ensuring that the unshuffle/shuffle operations work harmoniously with the learning process.

Datasets and Evaluation

To evaluate CompenHR, the authors introduced a new high-resolution benchmark, moving beyond the 256×256 limitations of previous datasets:

  • Real-world Dataset: Comprising 24 different projector-camera setups with diverse surfaces, including five challenging specular/shiny materials. It includes 700 colorful high-resolution images (500 for training, 200 for testing).
  • Synthetic Dataset: 100 setups rendered in Blender with varying poses, materials, and lighting conditions, primarily used to pre-train the photometric module.

Results

CompenHR demonstrates significant improvements over existing methods in both quality and efficiency:

  • Compensation Quality: CompenHR achieved a PSNR of 20.95, outperforming the high-resolution baseline CmpSt(HR) (20.55) and providing much sharper visual results than low-resolution alternatives.
  • Computational Efficiency: The model is approximately 4.4x faster to train than the high-resolution baseline.
  • Resource Management: It requires roughly half the memory (4.9GB vs 10GB) and 4.5x fewer FLOPS (364G vs 1645G) compared to CmpSt(HR). This efficiency allows CompenHR to be trained on a single mid-range GPU (e.g., Nvidia 1080), which was previously impossible for high-resolution end-to-end models.

Discussion and Limitations

While CompenHR represents a major step forward, several challenges remain. The system is currently designed for static setups; dynamic environments where the projector or surface moves would require online learning or faster adaptation. Additionally, while the model handles mild specularity better than its predecessors, it still struggles with highly reflective, mirror-like surfaces. Sharp geometric discontinuities (e.g., occlusions or very sharp corners) also remain a hurdle for the GANet component.

Conclusion

CompenHR provides a scalable solution for high-resolution projector compensation. By leveraging pixel unshuffling and coarse-to-fine refinement, it breaks the "memory wall" associated with deep learning in this domain, offering a practical path for high-fidelity projection on complex real-world surfaces.

Summary updated

BibTeX
@inproceedings{wang2023compenhr,
  author = {Yuxi Wang and Haibin Ling and Bingyao Huang},
  title = {{CompenHR: Efficient Full Compensation for High-resolution Projector}},
  booktitle = {VR},
  year = {2023},
  doi = {10.1109/vr55154.2023.00029},
}
Figure