Alignment Scores: Robust Metrics for Multiview Pose Accuracy Evaluation

Javier Civera, Seong Hun Lee, Javier Civera 0001arXiv 2024 · DOI · arXiv

Introduction

In computer vision and robotics, evaluating the accuracy of estimated camera trajectories against ground truth is essential for advancing Simultaneous Localization and Mapping (SLAM) and Structure-from-Motion (SfM) systems. Historically, the Absolute Trajectory Error (ATE) has been the standard metric. However, ATE is notoriously sensitive to outliers; a single erroneous pose can disproportionately inflate the error, obscuring the overall quality of the trajectory. While alternatives like Mean Average Accuracy (mAA) address outlier sensitivity, they often fail during collinear motion (e.g., a vehicle moving in a straight line) and can be biased by trajectory length. This paper proposes a new suite of metrics—Translation Alignment Score (TAS), Rotation Alignment Score (RAS), and Pose Alignment Score (PAS)—designed to be robust, scale-invariant, and consistent across diverse scenarios.

Methodology

The authors advocate for a decoupled evaluation strategy, assessing translation and rotation independently before integration. This avoids the numerical instability of combining disparate units (meters and degrees) prematurely.

Translation Alignment Score (TAS)

TAS employs a data-driven thresholding approach. Rather than using a fixed distance, it defines a reference distance $d$ as the upper quartile of closest-pair distances between ground-truth cameras. This ensures the metric adapts to the specific scale and density of any dataset. The estimated trajectory is aligned to the ground truth using a robust registration method (a modified PCR-99) capable of handling up to 99% outliers. Accuracy is then calculated as the area under the cumulative frequency curve across 100 thresholds ranging from $0.01d$ to $d$.

Conceptual illustration of the cumulative frequency calculation for distance and angular errors used to derive TAS and RAS.

Rotation and Pose Scores

RAS follows a similar logic to TAS but focuses on angular errors, utilizing robust single-rotation averaging for alignment. PAS is the arithmetic mean of TAS and RAS, providing a comprehensive 6-DOF accuracy score.

Experimental Evaluation

To rigorously "stress test" these metrics, the authors utilized Monte Carlo simulations. Unlike real-world datasets with fixed noise profiles, simulations allowed for precise control over variables such as Gaussian noise levels ($\sigma_t$), outlier percentages, and trajectory geometry (e.g., collinear motion).

Key Insights

  • Discerning Power: A robust metric must distinguish between different noise levels even in the presence of outliers. The authors demonstrate that TAS maintains significantly higher "discerning power" than ATE or mAA when outliers are introduced.
  • Stability in Collinear Motion: Many geometric metrics become degenerate when motion is perfectly linear. TAS remains stable in these common automotive scenarios because it relies on distance errors after robust registration rather than complex geometric constraints.

Results and Discussion

Experimental results highlight the superiority of the proposed scores in challenging conditions. In tests with 50% outliers, the discerning power of mAA dropped by 74%, whereas TAS only saw a 51% reduction, remaining sensitive to the underlying quality of inlier poses.

Performance comparison under collinear motion. TAS (bottom right) maintains clear separation between noise levels (colored boxplots) even as outliers increase, whereas ATE and DTE (top row) fail to provide useful differentiation.

Furthermore, TAS and PAS were shown to be largely insensitive to the number of views in a sequence. This makes them fairer for comparing performance across datasets of varying lengths, unlike mAA which tends to reward larger datasets regardless of per-pose accuracy.

Conclusion

The TAS, RAS, and PAS metrics offer a robust alternative for pose evaluation, particularly in environments prone to outliers or linear motion. While their "gentle" penalty for failure might be less suited for safety-critical systems where any failure is catastrophic, their ability to provide a granular, scale-invariant view of trajectory quality makes them a powerful tool for general research and development in spatial AI.

Summary updated

Abstract

We propose three novel metrics for evaluating the accuracy of a set of estimated camera poses given the ground truth: Translation Alignment Score (TAS), Rotation Alignment Score (RAS), and Pose Alignment Score (PAS). The TAS evaluates the translation accuracy independently of the rotations, and the RAS evaluates the rotation accuracy independently of the translations. The PAS is the average of the two scores, evaluating the combined accuracy of both translations and rotations. The TAS is computed in four steps: (1) Find the upper quartile of the closest-pair-distances, $d$. (2) Align the estimated trajectory to the ground truth using a robust registration method. (3) Collect all distance errors and obtain the cumulative frequencies for multiple thresholds ranging from $0.01d$ to $d$ with a resolution $0.01d$. (4) Add up these cumulative frequencies and normalize them such that the theoretical maximum is 1. The TAS has practical advantages over the existing metrics in that (1) it is robust to outliers and collinear motion, and (2) there is no need to adjust parameters on different datasets. The RAS is computed in a similar manner to the TAS and is also shown to be more robust against outliers than the existing rotation metrics. We verify our claims through extensive simulations and provide in-depth discussion of the strengths and weaknesses of the proposed metrics.
BibTeX
@misc{civera2024alignment,
  author = {Javier Civera and Seong Hun Lee and Javier Civera},
  title = {{Alignment Scores: Robust Metrics for Multiview Pose Accuracy Evaluation}},
  year = {2024},
  eprint = {2407.20391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20391},
}
Figure