Introduction
In computer vision and robotics, evaluating the accuracy of estimated camera trajectories against ground truth is essential for advancing Simultaneous Localization and Mapping (SLAM) and Structure-from-Motion (SfM) systems. Historically, the Absolute Trajectory Error (ATE) has been the standard metric. However, ATE is notoriously sensitive to outliers; a single erroneous pose can disproportionately inflate the error, obscuring the overall quality of the trajectory. While alternatives like Mean Average Accuracy (mAA) address outlier sensitivity, they often fail during collinear motion (e.g., a vehicle moving in a straight line) and can be biased by trajectory length. This paper proposes a new suite of metrics—Translation Alignment Score (TAS), Rotation Alignment Score (RAS), and Pose Alignment Score (PAS)—designed to be robust, scale-invariant, and consistent across diverse scenarios.
Methodology
The authors advocate for a decoupled evaluation strategy, assessing translation and rotation independently before integration. This avoids the numerical instability of combining disparate units (meters and degrees) prematurely.
Translation Alignment Score (TAS)
TAS employs a data-driven thresholding approach. Rather than using a fixed distance, it defines a reference distance $d$ as the upper quartile of closest-pair distances between ground-truth cameras. This ensures the metric adapts to the specific scale and density of any dataset. The estimated trajectory is aligned to the ground truth using a robust registration method (a modified PCR-99) capable of handling up to 99% outliers. Accuracy is then calculated as the area under the cumulative frequency curve across 100 thresholds ranging from $0.01d$ to $d$.
Rotation and Pose Scores
RAS follows a similar logic to TAS but focuses on angular errors, utilizing robust single-rotation averaging for alignment. PAS is the arithmetic mean of TAS and RAS, providing a comprehensive 6-DOF accuracy score.
Experimental Evaluation
To rigorously "stress test" these metrics, the authors utilized Monte Carlo simulations. Unlike real-world datasets with fixed noise profiles, simulations allowed for precise control over variables such as Gaussian noise levels ($\sigma_t$), outlier percentages, and trajectory geometry (e.g., collinear motion).
Key Insights
- Discerning Power: A robust metric must distinguish between different noise levels even in the presence of outliers. The authors demonstrate that TAS maintains significantly higher "discerning power" than ATE or mAA when outliers are introduced.
- Stability in Collinear Motion: Many geometric metrics become degenerate when motion is perfectly linear. TAS remains stable in these common automotive scenarios because it relies on distance errors after robust registration rather than complex geometric constraints.
Results and Discussion
Experimental results highlight the superiority of the proposed scores in challenging conditions. In tests with 50% outliers, the discerning power of mAA dropped by 74%, whereas TAS only saw a 51% reduction, remaining sensitive to the underlying quality of inlier poses.
Furthermore, TAS and PAS were shown to be largely insensitive to the number of views in a sequence. This makes them fairer for comparing performance across datasets of varying lengths, unlike mAA which tends to reward larger datasets regardless of per-pose accuracy.
Conclusion
The TAS, RAS, and PAS metrics offer a robust alternative for pose evaluation, particularly in environments prone to outliers or linear motion. While their "gentle" penalty for failure might be less suited for safety-critical systems where any failure is catastrophic, their ability to provide a granular, scale-invariant view of trajectory quality makes them a powerful tool for general research and development in spatial AI.
Summary updated