TL;DR
This paper introduces "Prakash," a novel motion capture system that uses inexpensive photosensing tags and multiplexed light emitters to track 3D motion, orientation, and incident illumination. This approach overcomes limitations of traditional systems by working in natural lighting, supporting an unlimited number of tags, and enabling real-time capture of photometric scene properties for enhanced virtual set integration and special effects.
What Problem Does This Paper Address?
Traditional optical motion capture systems, widely used in film and gaming for special effects and user interfaces, typically rely on high-speed cameras and specialized, often expensive, markers. These systems often require controlled environments to ensure high contrast between markers and their background, limiting their use in natural settings. Furthermore, they primarily focus on geometric information (position and orientation) and struggle to capture photometric properties like incident illumination and surface reflectance, which are crucial for realistic integration of virtual elements into real-world scenes. The authors aim to address these limitations by developing a system that is more scalable, cost-effective, and capable of capturing both geometric and photometric data in unconstrained environments.
Methods
The core of the "Prakash" system is a unique spatio-temporal modulation technique. Instead of high-speed cameras, Prakash uses a set of inexpensive "beamers" (LEDs with passive binary masks) that project coded light patterns. These patterns are designed to spatio-temporally encode the 3D space. Photosensing "tags," embedded in objects or attire, receive these coded signals.
Key methodological innovations include:
- Spatio-Temporal Modulation: The system uses a cluster of beamers, each projecting a unique bit of a binary code. By strategically arranging these beamers and exploiting epipolar geometry, the system can uniquely label points in 3D space. This is a departure from traditional structured light approaches that often use a single projector.
- Photosensing Tags: These tags are simple, low-cost devices that act as receivers. They can decode the optical signals to determine their location and orientation. Crucially, they also measure incident illumination.
- Intensity-Based Orientation: Unlike many marker systems, Prakash can compute the orientation of the tags by analyzing the falloff in received signal strength from multiple light sources, which follows a natural cosine law based on the angle between the light source and the tag's surface normal.
- Photometric Capture: The system directly measures incident illumination at each tag. By combining this with external camera readings and the tag's estimated reflectance, it can compute the intrinsic reflectance of scene points.
- Multiplexed Illuminators: The system uses time-division multiplexing (TDM) for the LEDs, allowing for a high update rate that is independent of the number of tags. Future work suggests exploring amplitude, frequency, or code-division multiplexing for "always-on" operation.
Datasets
Not discussed in the paper. The paper focuses on the system's methodology and validation using controlled experiments and demonstrations rather than large-scale datasets.
Key Insights and Ideas
- Reversing the Paradigm: Instead of using cameras to see markers, Prakash uses light emitters to label space and simple photosensors to decode it. This fundamental shift allows for operation in natural lighting and reduces system complexity and cost.
- Illumination as Data: The system's ability to capture incident illumination and surface reflectance alongside geometric data is a significant advancement. This opens doors for much more realistic integration of virtual objects and effects, as the virtual elements can be rendered to match the real-world lighting conditions.
- Scalability and Cost-Effectiveness: The use of inexpensive, off-the-shelf components (LEDs, photosensors, PIC microcontrollers) and a simple tag design makes the system highly scalable and cost-effective compared to traditional high-speed camera systems.
- Imperceptible Tags: The ability to embed tags imperceptibly into clothing or objects is a major advantage for applications like on-set motion capture or augmented reality, where aesthetics and comfort are important.
- "Zero-D" Camera: The authors describe their tag system as a "very high speed zero-D (point photo-sampling) camera," highlighting its ability to capture data at extremely high temporal resolutions.
Key Results
- High Update Rates: The system can achieve update rates approaching 1 MHz for certain measurements, with demonstrated location tracking at 500 Hz and potential for over 20,000 Hz with IrDA-based communication.
- Accurate Location and Orientation: The system demonstrates accurate 1D resolution of approximately 5.8 mm at 3 meters and orientation estimation reliable to within 1 degree.
- Robustness: The system shows robustness in bright outdoor conditions and even when tags are placed behind specular surfaces or IR-transparent materials.
- Photometric Capture: Demonstrations show the ability to realistically overlay textures and virtual objects that match captured incident illumination and surface properties.
- Low Cost: The tags and projectors are estimated to cost in the tens of US dollars.
Limitations and Future Work
Limitations:
- Line-of-Sight Requirement: Tags must be in the line of sight of the projectors.
- Ambient Lighting Sensitivity: Performance can degrade in very strong ambient lighting or due to multi-path distortions from secondary scattering.
- Dynamic Range: The system's dynamic range is limited, affecting reflectance estimation in certain scenarios.
- Orientation Ambiguity: The system cannot estimate rotation around the optical axis of the photosensor.
- Time Skewing: The time-division multiplexing for location estimation can lead to inaccuracies if the photosensor is displaced during the finite time it takes to acquire all bits.
Future Work:
- Advanced Multiplexing: Exploring optical amplitude, frequency, or code-division multiplexing for "always-on" operation without temporal synchronization.
- Photosensing Surfaces: Integrating photosensing tags into smart fabrics for continuous sampling rather than discrete points.
- BRDF Capture: Leveraging the system to estimate Bidirectional Reflectance Distribution Functions (BRDFs) by moving surfaces and capturing irradiance/radiance.
- Participating Media: Estimating properties of fog or liquids by measuring attenuation between multiple sensors.
- Illumination Manipulation: Dynamically adjusting illuminators based on real-time feedback from tags.
- Camera Adaptation: Using tag data to optimize camera parameters and perform reflectance-illumination factorization.
- Spatial Encoding: Developing more sophisticated spatial encoding schemes and calibration algorithms for arbitrary projector arrangements.
- Sensor Upgrade: Moving from point-sampling photosensors to 1D/2D optical sensors or low-resolution cameras for richer angular information.
Summary updated