When building 3D maps, tracking robots, or reconstructing dynamic scenes, we rely on streams of data from multiple sensors—such as cameras, LiDARs, IMUs, and microphones. To fuse this data accurately, engineers face two fundamental hurdles: determining where the sensors are relative to each other (spatial calibration) and aligning their internal clocks (temporal synchronization).
Traditionally, these problems were treated in isolation. Engineers would first compute the static physical offsets (extrinsics) using a calibration rig, and subsequently attempt to align the data timelines. However, in dynamic environments or during aggressive motion, this sequential approach fails. A tiny millisecond error in timing translates directly into a spatial misalignment, causing optimization algorithms to diverge or propagate errors.
The core insight of joint spatiotemporal calibration is that space and time are fundamentally coupled through motion. By treating temporal offsets not as fixed constants, but as latent variables or state variables within a unified geometric optimization framework, systems can leverage spatial constraints to resolve timing, and timing constraints to resolve geometry.
The Core Intuition: Motion Bridges Space and Time
Why can we solve for time by looking at geometry? The answer lies in kinematics.
Imagine a sensor platform moving through space at a certain velocity $v$. If there is an unknown temporal delay $\Delta t$ between two sensors (for example, a camera and a LiDAR), the data recorded by the second sensor will appear to be physically shifted relative to the first. Under motion, a temporal offset manifests precisely as a spatial displacement:
$$\Delta x = v \cdot \Delta t$$
By explicitly modeling this relationship, a synchronization error can be transformed into a geometric alignment problem. As demonstrated by Kodaira et al., incorporating the platform's estimated velocity allows temporal delays to be optimized alongside six-degree-of-freedom (6DoF) spatial parameters within a single geometric framework.
Neglecting this coupling is highly destructive to calibration accuracy. For instance, Li et al. show that failing to account for the time delay between a high-speed event camera and a robot's odometer drastically inflates rotation estimation errors. By leveraging the kinematic correlation of instantaneous linear velocities rather than drifting trajectories, the authors demonstrate that spatial alignment is highly sensitive to, and informative of, temporal synchronization.
This interdependence creates a classic "chicken-and-egg" dilemma. As noted by Ferranti et al. in the context of sensor network self-calibration, you cannot easily map out receiver coordinates without knowing signal transmission times, nor can you determine transmission times without knowing the coordinates. Attempting to resolve them sequentially introduces severe error propagation, as shown by Vo et al.. The solution is to optimize both simultaneously.
Sequential Calibration (Error-Prone):
[Temporal Sync] ──(Uncalibrated Space)──> [Spatial Calibration] ──> Accumulated Error
Joint Spatiotemporal Calibration (Robust):
┌──────────────────────┐
▼ │
[Spatial Parameters] ◄───► [Temporal Offsets]
▲ │
└──────────────────────┘
│
▼
Unified Geometric Framework
Unified State Estimation Frameworks
To exploit the physical coupling between space and time, modern frameworks integrate temporal offsets directly into the core estimation math. This is achieved primarily through two paradigms: batch optimization and online filtering.
Batch Optimization and Graph SLAM
In offline or windowed reconstruction, batch optimization minimizes residuals across all time steps and sensors at once to achieve global consistency.
- Minimal Solvers: For unsynchronized camera pairs, Albl et al. demonstrate that the temporal offset can be integrated directly into the two-view geometry model as a latent variable. This allows minimal algebraic solvers to recover both relative pose and timing simultaneously without requiring an initial guess.
- Spatiotemporal Bundle Adjustment: To scale this up to multiple sensors, frameworks must optimize global consistency. Tang et al. show that integrating temporal synchronization with intrinsic and extrinsic camera calibration within a single cascaded framework addresses the limitations of isolating these tasks. In acoustic networks, Wang et al. employ batch optimization to simultaneously refine 3D spatial coordinates and temporal clock parameters—including initial offsets and sampling rate drifts—by minimizing residuals across the entire network.
Online Filtering
For real-time robotics, waiting for a massive batch optimization is impractical. Instead, systems use tightly-coupled filters to update spatial and temporal parameters on the fly.
- Sliding-Window Filters: In multi-modal odometry, assuming static and perfect factory synchronization leads to tracking failures during aggressive maneuvers. To counter this, Geneva et al. (LIC-Fusion) and Yang et al. (LIC-Fusion 2.0) introduce sliding-window filters that track LiDAR, camera, and IMU data concurrently. By performing online spatiotemporal calibration within the filter, the system dynamically refines physical mounting offsets and time delays.
- Error-State Filtering: Similarly, Goudar et al. utilize an Error State Kalman Filter (ESKF) to achieve online spatiotemporal calibration for Ultra-Wideband (UWB) aided inertial localization, demonstrating that real-time software refinement can achieve millimeter-level spatial and millisecond-level temporal accuracy.
Overcoming Unobservability and Local Minima
Simultaneously estimating space and time introduces a major mathematical challenge: unobservability. If a sensor platform moves at a constant velocity in a straight line, it is mathematically impossible to distinguish between a physical spatial offset along that axis and a temporal delay. The parameters become degenerate, and optimization algorithms can get trapped in local minima.
To make the system observable, the sensor rig must undergo sufficient excitation—meaning it must experience acceleration or rotational motion. He et al. address this by conducting rigorous observability analyses using the Fisher Information Matrix. By incorporating physical measurements like Time Difference of Arrival (TDOA) and Direction of Arrival (DOA) with hardware-specific variables like clock drift and 3D Euler rotations into a unified observation model, the authors establish the exact geometric conditions and minimum data movements required for a successful joint calibration.
Additionally, providing a reliable initialization is critical. Ferranti et al. tackle the initialization problem in TDOA sensor networks by introducing a novel symbolic factorization of the measurement matrix. This algebraic breakthrough provides exact solutions for "minimal" network configurations, bypassing the limitations of sequential estimation and preventing non-linear optimization algorithms from getting stuck in local minima.
Broad Applicability Across Modalities
The beauty of joint spatiotemporal optimization is that its underlying physics applies regardless of the sensor modality.
- Visual-Inertial Fusion: On consumer hardware like mobile phones, factory calibrations are often imperfect. Tanskanen et al. demonstrate that rigorous online spatiotemporal calibration between a monocular camera and an IMU is what unlocks the ability to recover absolute metric scale in real-time, turning arbitrary units into centimeters.
- Distributed Multi-Robot Systems: When multiple independent robots interact, their internal clocks inevitably drift. Frameworks like D-SLATS, introduced by Hespanha et al., and the mutual localization methods by Wen et al., treat time synchronization as an additional geometric state variable. By modeling time differences as geometric shifts, distributed agents can utilize the physical coupling between signal travel times and clock biases to localize and synchronize themselves without a central server.
- Heterogeneous Cameras: The paradigm even extends to mixing entirely different sensor architectures. Wang et al. (EF-Calib) apply joint spatiotemporal optimization to systems combining standard frame-based cameras with high-speed event cameras. By representing the camera's motion as a continuous-time B-spline trajectory and optimizing time delay as an internal variable, they achieve millisecond-level synchronization accuracy that matches the performance of expensive, hardware-synchronized trigger systems.
Ultimately, joint spatiotemporal calibration works because it stops fighting the interplay between space and time, and instead uses motion to let them calibrate each other.
Key Foundational Work
Live Metric 3D Reconstruction on Mobile Phones (2013) — contributed 1 concept elements to this direction.
Spatiotemporal Bundle Adjustment for Dynamic 3D Reconstruction (2016) — contributed 1 concept elements to this direction.