TL;DR
This paper introduces a novel method to synchronize multiple Intel RealSense RGB-D cameras, overcoming limitations of existing solutions that struggle with scaling to many cameras or synchronizing both RGB and depth streams. Their approach uses a master-slave architecture to synchronize host system clocks and then leverages these synchronized clocks to achieve accurate, frame-rate-consistent capture across an arbitrary number of cameras, significantly reducing artifacts in 3D reconstructions.
What Problem Does This Paper Address?
Accurate 3D reconstruction of objects often requires data from multiple cameras to capture complete geometric information and avoid artifacts. However, synchronizing these cameras to capture frames simultaneously is a significant challenge. Existing solutions, like Intel's RealSense cameras, offer built-in synchronization modes (GenLock, Slave, Full Slave), but these have limitations: GenLock halves the frame rate and only works with depth cameras, while Slave modes are constrained by USB bandwidth, limiting the number of cameras per host and struggling with synchronizing cameras across different hosts due to differing system clocks. This paper aims to solve the problem of synchronizing a large number of RealSense RGB-D cameras across multiple hosts to enable high-quality, artifact-free 3D reconstruction.
Methods
The core of the proposed method is a multi-host synchronization strategy that builds upon the concept of global timestamps. First, they establish a master-slave architecture where one host acts as the master, and others are slaves. The key innovation lies in accurately synchronizing the system clocks of these multiple hosts. This is achieved by:
- Estimating Delays: They meticulously estimate two types of delays:
- Time Synchronization Delay ($t_s$): The time taken for a slave host to receive the master's system clock and set its own. This is calculated by averaging the time taken for
get-timeandset-timeOS functions. - Network Synchronization Delay ($t_l$): The time taken for data packets to travel between the master and slave hosts. This is measured by sending multiple data packets and calculating the average round-trip time.
- Master-Slave Clock Synchronization: Using the estimated delays, slave hosts adjust their system clocks to match the master host's clock, effectively creating a unified system clock across all hosts.
- Global Timestamp Regression: Once hosts are synchronized, they leverage the concept of "global timestamps." This involves regressing a linear model between the camera's internal sensor timestamps and the host's time-of-arrival timestamps. This regression accounts for any minor discrepancies and provides a unified timestamp for each frame, regardless of which host it arrived on.
- Frame Gathering: Using these synchronized global timestamps, frames from different cameras (even across different hosts) that share a global timestamp within a small threshold are considered simultaneously captured and can be reliably gathered for 3D reconstruction.
This approach extends the single-host synchronization capabilities to a multi-host setup, allowing for a scalable number of cameras.
Datasets
The paper does not introduce new datasets. Instead, it utilizes data captured from a custom experimental setup involving 18 Intel RealSense D455 RGB-D cameras. This setup is described in detail, including the camera arrangement (360° coverage, multiple heights) and the use of external triggers for initial synchronization.
Key Insights and Ideas
- Multi-host synchronization is crucial for scaling: The inherent USB bandwidth limitations of a single host necessitate a multi-host approach for systems with many cameras. However, synchronizing across hosts is significantly harder than within a single host.
- Accurate delay estimation is the linchpin: The core challenge in multi-host synchronization is the difference in system clocks. The paper's breakthrough is a robust method for estimating and compensating for both network latency and time synchronization delays between hosts.
- Global timestamps bridge the gap: By synchronizing host clocks and then using a regression model to derive global timestamps, the system can effectively treat cameras connected to different hosts as if they were on a single, perfectly synchronized system.
- RealSense limitations are overcome: The proposed method circumvents the RealSense's native limitations, such as GenLock's frame rate reduction and depth-only sync, and the Slave mode's per-host camera count.
- Scalability is achieved: The architecture is designed to be scalable by simply adding more hosts, allowing for an "arbitrary number" of cameras to be synchronized.
Key Results
- Quantitative Evaluation:
- The proposed method significantly reduced synchronization errors. For 15 FPS, the Average Maximum Delay (AMD) dropped from 547.5033 ms (None) to 5.8539 ms (Both delays), and Root Mean Delay Variation (RMDV) from 194.8521 ms to 3.8868 ms.
- Similar improvements were observed at 45 FPS.
- The best performance was achieved when both time and network synchronization delays were considered.
- Qualitative Evaluation:
- Visual results using a digital clock displayed on a tablet showed that the proposed method synchronized the cameras to display identical times, indicating accurate frame capture.
- In contrast, NTP synchronization (without specific delay compensation) and GenLock mode (which has its own limitations) showed discrepancies in displayed times, indicating less accurate synchronization.
- 3D reconstruction results showed a significant reduction in artifacts when using the proposed synchronization method compared to standard NTP synchronization. Reprojection errors were reduced from a mean of 9.7030 cm to 2.3942 cm.
Limitations and Future Work
- Limitations:
- The paper notes that if a host's USB interface could support a wider bandwidth (e.g., with extension cards), a single host might suffice for synchronization without the proposed multi-host method, up to hardware limitations.
- GenLock mode, while not the focus, is mentioned as capable of synchronizing multiple depth cameras without frame rate loss if external triggers are used and all cameras start simultaneously.
- Future Work:
- The paper suggests that the method will be helpful in various applications of computer vision related to 3D reconstruction.
- While not explicitly stated as future work, the scalability aspect implies that further testing with even larger numbers of cameras and hosts would be a natural progression.
Summary updated