Graphics hardware accelerated panorama builder for mobile phones

Miguel Bordallo López, Jari Hannuksela, Olli Sílven, Markku VehviläinenProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2009 · DOI

TL;DR

This paper presents a novel interactive document scanning application for mobile phones that builds high-resolution panoramas from low-resolution video frames. By leveraging the mobile device's GPU for accelerated image processing, it significantly improves usability and speed, addressing the common frustrations of ad-hoc document capture.

What Problem Does This Paper Address?

Capturing clear, high-resolution images of documents using mobile phone cameras is surprisingly difficult. Users often struggle with blurry images due to motion or focus issues, and the delay in processing and reviewing results makes the process frustrating and inefficient. This is particularly true for common business tasks like scanning documents, whiteboards, or business cards. The core problem lies in the mismatch between the processing power of mobile devices, camera capabilities, and the demand for an interactive, immediate user experience. Traditional methods of stitching images require offloading to a desktop, which defeats the purpose of a mobile solution.

Methods

The authors propose an interactive approach that guides the user through capturing a sequence of VGA resolution (640x480) video frames. Key to their method is:

  • Interactive User Guidance: The application provides real-time feedback on image quality and motion, guiding the user to move the device optimally. This transforms the traditional "take multiple shots and hope for the best" approach into a more controlled, single-take video experience.
  • On-line Motion Estimation: The system estimates camera motion between frames to facilitate registration and guide the user.
  • Quality Evaluation and Frame Selection: Frames are automatically assessed for blur and motion, and only the best quality frames are selected for stitching, conserving memory and improving the final output.
  • GPU Acceleration: The core of their innovation is the use of the mobile device's Graphics Processing Unit (GPU) to accelerate computationally intensive tasks like image warping and interpolation. They utilize the OpenGL ES 1.1 API for this purpose, mapping these operations onto the GPU's parallel processing capabilities.

Datasets

Not discussed in the paper. The authors implemented their application on a Nokia N95 mobile phone and demonstrated its capabilities using real-world document images.

Key Insights and Ideas

  • Video as a Solution: Shifting from single-shot still images to low-resolution video frames is a crucial insight. This allows for continuous capture and real-time feedback, making the process more interactive and forgiving of minor user errors.
  • GPU for Vision on Mobile: The paper strongly advocates for leveraging the GPU, not just for graphics, but as a general-purpose compute engine for vision tasks on mobile devices. This is a significant departure from relying solely on the CPU for such operations.
  • Interactivity is Key to Usability: The success of mobile imaging applications hinges on providing immediate feedback and guidance. The interactive interface is not just a feature but a fundamental requirement for a good user experience in this context.
  • Trade-offs in Mobile Vision: The authors highlight the inherent trade-offs between computational cost, interactivity, and image quality on resource-constrained mobile platforms.

Key Results

  • The implemented interactive document scanning application on a Nokia N95 phone demonstrated the viability of their concept.
  • GPU acceleration for image warping and interpolation achieved processing times of less than 100ms per VGA frame, a significant speedup compared to software implementations (which were about four times slower).
  • The system provides real-time feedback on motion and quality, enabling users to capture good quality images for stitching.
  • A final mosaic image (1397x1099 pixels) was constructed from eight VGA images, showcasing the practical capabilities of the approach.

Limitations and Future Work

  • Memory Constraints: The authors acknowledge that memory needs are a significant bottleneck for fine-registration with current mobile devices, limiting the size of final mosaics and the number of input frames.
  • OpenGL ES 1.1 Limitations: The fixed-function pipeline of OpenGL ES 1.1 has limitations, such as requiring single-buffer mode for readback and overheads from copying images as textures.
  • Future Directions: The paper points towards future developments with OpenGL ES 2.0 and emerging standards like OpenCL. These offer more flexibility with programmable shaders and a unified approach to parallel computing across CPUs and GPUs, which could further enhance performance and enable more complex vision algorithms on mobile devices. They suggest that OpenCL could be particularly beneficial for distributing tasks across available processing units.

Summary updated

Abstract

Modern mobile communication devices frequently contain built-in cameras allowing users to capture highresolution still images, but at the same time the imaging applications are facing both usability and throughput bottlenecks. The difficulties in taking ad hoc pictures of printed paper documents with multi-megapixel cellular phone cameras on a common business use case, illustrate these problems for anyone. The result can be examined only after several seconds, and is often blurry, so a new picture is needed, although the view-finder image had looked good. The process can be a frustrating one with waits and the user not being able to predict the quality beforehand. The problems can be traced to the processor speed and camera resolution mismatch, and application interactivity demands. In this context we analyze building mosaic images of printed documents from frames selected from VGA resolution (640x480 pixel) video. High interactivity is achieved by providing real-time feedback on the quality, while simultaneously guiding the user actions. The graphics processing unit of the mobile device can be used to speed up the reconstruction computations. To demonstrate the viability of the concept, we present an interactive document scanning application implemented on a Nokia N95 mobile phone.
BibTeX
@inproceedings{lopez2009graphics,
  author = {Miguel Bordallo López and Jari Hannuksela and Olli Sílven and Markku Vehviläinen},
  title = {{Graphics hardware accelerated panorama builder for mobile phones}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2009},
  doi = {10.1117/12.816511},
}
Figure