TL;DR
This paper introduces a novel linear method for reconstructing the 3D trajectory of a moving point from a series of 2D images. It addresses the limitations of traditional triangulation by proposing a "reconstructibility" metric to quantify reconstruction accuracy and a practical algorithm that handles missing data and camera motion.
What Problem Does This Paper Address?
Reconstructing the 3D structure of a scene from 2D images is a fundamental problem in computer vision. For static points, triangulation is a well-established technique. However, when points are moving between image captures, traditional triangulation fails because the rays from the camera centers to the point no longer intersect at a single point. This scenario is common in real-world applications where cameras are often monocular and scenes contain dynamic elements. Existing methods for dynamic scenes often rely on factorization and can struggle with missing data or orthographic camera assumptions. This paper aims to provide a more robust and general linear solution for reconstructing moving point trajectories, even with non-coincidental image captures and potential occlusions.
Methods
The core contribution is a linear reconstruction algorithm inspired by the Direct Linear Transform (DLT) method. The key idea is to represent the 3D trajectory of a moving point as a compact linear combination of a trajectory basis (specifically, Discrete Cosine Transform or DCT bases are used). This allows for a compact representation of the trajectory, enabling an overconstrained linear least squares system to be formulated.
The paper also introduces a crucial concept: reconstructibility. This metric quantifies the degree of interference between the point's trajectory and the camera's trajectory. It's defined as the ratio of the null space component of the camera trajectory to the trajectory basis matrix. A higher reconstructibility value indicates better potential for accurate point trajectory reconstruction. The authors derive a theorem stating that reconstruction is impossible if the point and camera trajectories are perfectly correlated (corr(X, C) = ±1).
The algorithm involves:
- Linearizing the projection model: Representing the 3D point projection into 2D images using camera projection matrices.
- Trajectory basis representation: Expressing the 3D point trajectory as a linear combination of basis vectors (e.g., DCT).
- Formulating a linear least squares system: Combining the linearized projection and trajectory basis to create an overdetermined system.
- Solving for trajectory coefficients: Using the least squares solution to estimate the coefficients of the trajectory basis.
A "specialized DCT basis" is also proposed, which is the projection of the original DCT basis onto the null space of the camera trajectory. This aims to improve reconstructibility by reducing the basis's ability to reconstruct the camera trajectory.
Datasets
The paper evaluates its method on both synthetic data generated from motion capture data and real-world data. The real-world data includes scenarios like:
- Rock climbing
- Handshake
- Public speech
- Greeting
These real-world sequences were collected from multiple photographers, leading to asynchronous images and effectively a random camera trajectory, which aligns with the theoretical conditions for good reconstruction.
Key Insights and Ideas
- Reconstructibility is Key: The paper highlights that the accuracy of 3D trajectory reconstruction is fundamentally limited by the correlation between the point's motion and the camera's motion. This is a crucial insight for understanding the inherent difficulties in dynamic scene reconstruction.
- Linear Trajectory Basis Simplifies the Problem: Representing trajectories with a compact linear basis (like DCT) transforms a complex non-linear problem into a solvable linear least squares problem.
- Specialized Basis for Better Performance: Projecting the trajectory basis onto the null space of the camera trajectory (the "specialized DCT basis") can significantly improve reconstruction accuracy, especially when camera motion is smooth and predictable.
- Handling Missing Data is Natural: The linear least squares formulation inherently handles missing data points (due to occlusion, etc.) without requiring explicit imputation.
Key Results
- The proposed linear algorithm demonstrates accurate 3D trajectory reconstruction on both synthetic and real-world data.
- The "reconstructibility" metric is shown to be inversely proportional to the 3D reconstruction error, providing a useful diagnostic tool.
- The specialized DCT basis consistently outperforms the original DCT basis in reconstruction accuracy, particularly for smooth camera trajectories and when fewer basis functions are used.
- The method shows robustness to occlusion and varying frame rates.
- Qualitative results show that the reconstructed trajectories closely match the ground truth, even in challenging perspective projection scenarios where other methods struggle.
Limitations and Future Work
- Predefined Basis: The method relies on a predefined trajectory basis. While DCT is shown to be effective, choosing the optimal basis for arbitrary motions might require further investigation.
- Camera Pose Estimation: The algorithm assumes that camera poses are known. While the paper mentions using robust structure-from-motion for static points to estimate relative poses, accurate camera pose estimation remains a prerequisite.
- "Perfect" Reconstructibility is Ideal: The theoretical condition for perfect reconstructibility (infinite η) is difficult to achieve in practice, as the true point trajectory is unknown.
Future work could explore adaptive basis selection, more robust camera pose estimation in dynamic scenes, and extending the method to reconstruct non-rigid structures.
Summary updated