3D Pose Estimation in Projection Space via Pixel-level Visible Light Communication

Michikuni Eguchi, Takefumi HirakiarXiv 2026 · arXiv

This paper, recently submitted to IEEE, presents a novel approach to 3D pose estimation for devices within a projector-based Augmented Reality (AR) environment. By leveraging Pixel-level Visible Light Communication (PVLC), the researchers have developed a way to track devices in 3D space with high precision and low latency, all without the need for external cameras or complex infrastructure.

TL;DR

The authors extend Pixel-level Visible Light Communication (PVLC) from 2D to 3D by formulating pose estimation as a Perspective-n-Point (PnP) problem. Using a projector as an "inverse camera" and a small array of photosensors, the system achieves centimeter-level 3D tracking with just 8ms of latency, outperforming traditional camera-based markers at high movement speeds.

What Problem Does This Paper Address?

Projector-based AR, or Spatial Augmented Reality (SAR), overlays digital information directly onto the physical world. For this to feel seamless, the system must know exactly where a device (like a handheld tool or a robot) is relative to the projector.

Traditionally, this requires external sensing like motion capture systems or depth cameras, which are expensive, bulky, and require tedious geometric calibration. PVLC was proposed as a solution: it embeds invisible spatial codes into every projected pixel using high-speed brightness modulation. However, until now, PVLC was limited to 2D tracking—fine for a robot moving on a flat table, but useless for aerial drones or complex 3D interactions. This paper bridges that gap, moving PVLC from the 2D plane into 3D space.

Methods

The core innovation lies in treating the projector as an "inverse camera."

  1. PVLC Encoding: The system uses a Digital Micromirror Device (DMD) projector to modulate the brightness of every pixel at a rate invisible to the human eye. Each pixel transmits its own 2D coordinate $(u, v)$ using Gray codes.
  2. PnP Formulation: The researchers mounted four photosensors in a known 3D layout (e.g., an 80mm square) on a device. Each sensor decodes the $(u, v)$ coordinate of the projector pixel illuminating it.
  3. Solving for Pose: By combining these 2D "image points" (decoded coordinates) with the known 3D "object points" (the sensor layout) and the projector's intrinsic parameters, the system solves the Perspective-n-Point (PnP) problem to calculate the device's 3D position and orientation.
FIGURE 1. Conceptual overview of the proposed 3D pose estimation system using Pixel-level Visible Light Communication (PVLC). The projector embeds invisible position codes into each pixel. Multiple photosensors on the device decode their 2D pixel coordinates, which are combined with the projector’s intrinsic parameters and the known 3D sensor geometry, and fed into a PnP solver to estimate the device’s 3D pose.

The team compared several PnP solvers, including Homography Decomposition (HD), IPPE, P3P, and EPnP, finding that solvers assuming coplanar points (like HD and IPPE) provided the most stable results for their flat sensor board. They also implemented a Kalman Filter to smooth the trajectory and handle noise.

Datasets

The authors did not use a standard public dataset; instead, they built a custom experimental rig to generate ground-truth data.

  • Hardware: A DLP LightCrafter 4500 projector and a custom sensor board with four photodiodes.
  • Ground Truth: The sensor device was mounted on a robotic manipulator (xArm5) with a positioning repeatability of $\pm0.1$ mm.
  • Baseline: They compared their results against standard ArUco markers tracked by an Intel RealSense D415 camera.

Key Insights and Ideas

  • The "Inverse Camera" Paradigm: The most elegant shift is realizing that the math used for camera-based pose estimation works perfectly in reverse for projectors. In this setup, the projector's "pixels" are the sensors' inputs.
  • Whole-Volume Tracking: Unlike other light-based tracking methods that require dedicated "tracking regions" in the corner of a projection, PVLC embeds codes in every pixel. This means the entire projection volume is a tracking zone, and the device never "leaves" the trackable area as long as it's illuminated.
  • Robustness to Motion Blur: Traditional cameras suffer from motion blur at high speeds, which ruins marker detection. Because PVLC sensors decode high-speed pulses rather than integrating light over an exposure time, the system is significantly more robust to fast movement.
FIGURE 3. Comparison of PnP-based 3D pose estimation in (a) a camera system and (b) the proposed PVLC system. In PVLC, (1) 3D marker points become photosensor positions, (2) the camera image plane becomes the projector pixel plane, and (3) 2D projected points become decoded pixel coordinates.

Key Results

  • Latency: The PVLC system achieved a latency of approximately 8 ms, significantly faster than the 40.2 ms measured for the camera-based AR marker baseline.
  • Accuracy: With an 80mm sensor layout and Kalman filtering, the system achieved a 3D position RMSE of 19.2 mm.
  • Speed Performance: At high speeds (400 mm/s), the proposed method maintained stable accuracy, while the camera-based baseline's error increased by a factor of three to four due to motion blur.
  • Layout Size: Increasing the sensor layout from 80mm to 160mm improved 3D position accuracy by approximately 40%.
FIGURE 4. Prototype sensor device. (a) Overview of the sensor device receiving projected light from the projector. (b) Photosensor board with four photodiodes arranged in an 80 mm square on the same plane. (c) Signal processing circuit consisting of a current-to-voltage conversion circuit, amplification circuit, comparator circuit, and microcontroller.

Limitations and Future Work

  • Miniaturization Trade-off: The math shows that depth and angular errors increase as the sensor layout gets smaller. Tracking very small devices will require higher-resolution projectors or more sophisticated sensor fusion.
  • Projection Volume: The system only works within the "frustum" of the projector. If the device moves into a shadow or outside the light beam, tracking is lost.
  • Future Directions: The authors suggest fusing PVLC data with Inertial Measurement Units (IMUs) via an Extended Kalman Filter to improve stability during extremely agile maneuvers (like drone flight) and investigating higher-resolution projectors to push accuracy into the sub-centimeter range.

Summary updated

Abstract

Projector-based augmented reality enables natural interaction between users and devices by overlaying digital information onto the physical world through projected imagery. A key technical requirement of such systems is accurate positional alignment between the projector and the devices. Pixel-level Visible Light Communication (PVLC) achieves this alignment without external sensing infrastructure by embedding invisible spatial codes into each projected pixel through high-speed brightness modulation that preserves image quality. However, the information that PVLC transmits is inherently planar. PVLC has therefore been limited to devices moving on a flat surface and cannot estimate the 3D pose of devices in three-dimensional space, such as aerial robots. To extend PVLC-based pose estimation to three dimensions, we formulate it as a Perspective-n-Point (PnP) problem. The 2D pixel positions that four coplanar photosensors on the device receive via PVLC serve as the image points, and their known 3D layout provides the object points. Experiments show that the proposed system estimates 3D pose with a latency of approximately 8~ms and a 3D position RMSE of 19.2~mm after Kalman filtering. These results confirm the first realization of real-time 3D pose estimation coordinated with high-quality projected image content without external sensing infrastructure.
BibTeX
@misc{eguchi2026d,
  author = {Michikuni Eguchi and Takefumi Hiraki},
  title = {{3D Pose Estimation in Projection Space via Pixel-level Visible Light Communication}},
  year = {2026},
  eprint = {2611.06121},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06121},
}
Figure