Introduction
This paper presents a system for capturing nearly synchronous multi-view video and camera poses using commodity AR-enabled smartphones. Traditional "Free-Viewpoint Video" (FVV) or 4D reconstruction systems typically require expensive, high-speed camera rigs synchronized via hardware. Moving this capability to handheld devices in uncontrolled environments introduces significant challenges in temporal synchronization, spatial calibration, and data throughput.
Methods
The proposed system utilizes a decentralized architecture where a "host" mobile coordinates multiple "client" mobiles.
Spatial Calibration and Pose Estimation
Each device leverages on-board SLAM (ARCore) for self-tracking. To establish a global coordinate system, the system uses Cloud Anchors, allowing devices to find a common reference point in the environment. This approach eliminates the need for manual "checkerboard" calibration sessions before recording.
Temporal Synchronization
To overcome the lack of hardware triggers, the system employs a delay-compensation strategy. The host measures the Round Trip Time (RTT) to each client and calculates an artificial delay for each device, ensuring frames are captured simultaneously despite network jitter.
Communication Architecture
The system uses a two-tier communication model to handle high throughput (~160KB-per-frame):
- Relay Server: Utilizes UDP for low-latency synchronization triggers.
- Data Manager: Employs a custom socket-based TCP architecture to stream JPEG-encoded frames and metadata, avoiding the overhead and timeouts associated with standard HTTP/REST architectures.
The 4DM Dataset
The authors introduced the 4DM (4DMobile) dataset to validate the system. It features six people recording an actor performing sports (table tennis) outdoors. The dataset includes three complexity levels: Easy (static recorders), Medium (three moving recorders), and Hard (all recorders moving with occluders). Unlike traditional datasets, 4DM captures the instability of consumer-grade handheld hardware in natural lighting.
Key Results and Insights
The system successfully managed data from up to four mobiles at 10fps on a standard 2.4GHz WiFi network before hitting bandwidth congestion.
Reconstruction Performance
The 3D skeleton reconstruction, validated using OpenPose and global Bundle Adjustment (BA), achieved an average re-projection error of 2.86 to 3.11 pixels across the 4DM dataset. The authors demonstrated that initializing BA with the "center of mass" of triangulated points allowed for a global optimization three times faster than standard incremental approaches.
Synchronization Accuracy
The trigger-based synchronization method achieved an error of approximately 21ms ± 17ms, comparable to high-end NTP-based software solutions while offering greater flexibility for online frequency adjustment.
Limitations and Future Work
Current limitations include sensitivity to user positioning (baseline issues) and SLAM drift in feature-poor environments. Future work aims to implement AR-based UI guidance for users and more adaptive latency compensation to handle network spikes. While 5G may alleviate bandwidth constraints, current WiFi remains a bottleneck for scaling beyond 4-6 devices at high frame rates.
Summary updated