Study of Detecting Method of Broadcasting Camera's Movement using MEMS Sensor and Image Processing Technology

Daiichiro Kato, Kazutoshi Muto, Hideki Mitsumine, Hiroyuki Okamoto, Alessandro Moro, Toshikazu Seki, Shintaro MizukamiJournal of the Robotics Society of Japan 2014 · DOI

Abstract

The virtual studio, which enables the real-time compositing of live-action footage with computer graphics (CG), is an essential tool for modern broadcasting. However, utilizing handheld cameras in these environments remains a significant challenge due to the requirement for highly precise, real-time camera movement data. This paper presents a novel hybrid sensor system designed to measure the 3D position and orientation of a camera using Micro Electro Mechanical Systems (MEMS) and Particle Image Velocimetry (PIV) technology. By implementing advanced noise control for MEMS and a specialized image processing algorithm, the system achieves an angular precision of approximately 0.2° and a positional error within 3% of the moving distance at a standard video rate of 59.94 Hz.

Introduction: The Challenge of Handheld Tracking

In virtual studio productions, the virtual camera in the CG engine must perfectly synchronize with the physical camera's movements to maintain the illusion of a shared space. While traditional studio setups rely on heavy pedestals with integrated rotary encoders, these systems are bulky and limit movement to flat, prepared surfaces.

Broadcasting camera movement axes: Pan, Tilt, Roll, Zoom, and Focus.

There is a growing demand for handheld camera work to create dynamic and immersive content. However, tracking a handheld camera in a featureless green-screen environment is difficult. Existing solutions, such as infrared ceiling markers or magnetic sensors, often suffer from complex calibration requirements, sensitivity to studio lighting, or electromagnetic interference. To address this, a self-contained tracking solution is required that can accurately capture all degrees of freedom (Figure 1) without external infrastructure.

Methodology: The Hybrid Sensor Approach

The researchers developed a hybrid sensor unit that attaches directly to a standard broadcast camera. The system bifurcates the tracking task into orientation and translation domains:

  1. Orientation (Attitude Tracking): The system employs a 6-axis MEMS module containing three gyroscopes and three accelerometers. To mitigate the drift inherent in gyroscopes and the high-frequency noise of accelerometers, a complementary filtering approach was used. Accelerometers provide a stable gravity reference for low-frequency tilt and roll, while gyroscopes capture high-frequency rotational changes.
  2. Position (Translation Tracking): To track 3D movement (x, y, z), the unit features a downward-facing "sensing camera" paired with a laser rangefinder. The system applies Particle Image Velocimetry (PIV)—a technique borrowed from fluid dynamics—to track the "flow" of the floor texture.
  3. Mechanical Stabilization: A dedicated motor keeps the sensing camera pointing vertically at the floor regardless of the main camera's tilt or pan. This ensures the floor plane remains parallel to the sensor, significantly simplifying the image processing calculations.
Overview of the hybrid sensor system and its internal components, including MEMS modules and the PIV sensing camera.

To handle the uniform, low-contrast floors typical of virtual studios, the system utilizes Contrast Limited Adaptive Histogram Equalization (CLAHE). This enhances microscopic surface details, such as dust or minor scratches, into trackable features for the Lucas-Kanade optical flow algorithm.

Evaluation and Key Insights

The prototype was evaluated against a high-precision robot camera system, which served as the ground truth with an accuracy of 0.0018° for rotation and ±5mm for position.

Several key insights emerged from the study:

  • The Floor as a Reference: By focusing exclusively on the floor, the sensor avoids the visual noise of moving actors and changing props in the main frame.
  • Feature Extraction: The use of CLAHE proved that even "featureless" studio floors contain enough microscopic texture for reliable PIV tracking.
  • Drift Management: While gravity provides an absolute reference for tilt and roll, pan (azimuth) lacks such a reference. The authors implemented a 0.1°/s cutoff filter to suppress micro-drifts when the camera is stationary.
  • Coordinate Transformation: Using Quaternions allowed the system to accurately map the sensor's data to the camera's principal lens point while avoiding the mathematical pitfalls of gimbal lock.

Results and Performance

The hybrid sensor demonstrated performance suitable for professional broadcast environments:

  • Angular Precision: Achieved approximately 0.2°. Tilt errors remained within 0.3° even during significant 30° inclines.
  • Positional Accuracy: The PIV process maintained an error rate within 3% of the total distance traveled (e.g., 15 cm error over a 5-meter move).
  • Real-time Processing: The system operates at 59.94 Hz with a latency of less than three frames, meeting the strict requirements for live television production to ensure audio-visual synchronization.

Conclusion and Future Work

While the system provides a robust solution for handheld tracking, pan drift remains a primary limitation due to the lack of an absolute horizontal reference. Future research will focus on optimizing PIV algorithms and improving MEMS performance to meet the pixel-perfect requirements of high-end drama production. Additionally, the team plans to conduct subjective evaluations with professional operators to ensure the tracked movement feels natural in a live CG environment.

Reference: Kato, D., et al. (2014). Study of Detecting Method of Broadcasting Camera's Movement using MEMS Sensor and Image Processing Technology. Journal of the Robotics Society of Japan, 32(7), 576-585.

Summary updated

Abstract

The virtual studio which can make composite real image and computer graphics is indispensable equipments for broadcasting stations. However, it's difficult to use handy type cameras in such kind of studios, because it requires the precise camera movement data. So, we devised the new sensor which could measure angle and the 3 dimensional position of the camera, using Micro Electro Mechanical Systems (MEMS) and Particle Image Velocimetry (PIV) technology. We clarified the method of MEMS's noise control, and the development of a new image processing algorithm, we can get precision around 0.2[deg] by the new MEMS sensor, and within 3% to moving distance in a rate of 59.94[Hz] (video rate) by PIV process.
BibTeX
@article{kato2014study,
  author = {Daiichiro Kato and Kazutoshi Muto and Hideki Mitsumine and Hiroyuki Okamoto and Alessandro Moro and Toshikazu Seki and Shintaro Mizukami},
  title = {{Study of Detecting Method of Broadcasting Camera's Movement using MEMS Sensor and Image Processing Technology}},
  journal = {Journal of the Robotics Society of Japan},
  year = {2014},
  doi = {10.7210/jrsj.32.576},
}
Figure