Optimal-Transport-Based Metric For SMLM

Quentin Denoyelle, Thanh-An Pham, Pol del Aguila Pla, Daniel Sage, Michael Unser

Introduction

Single-Molecule Localization Microscopy (SMLM) has revolutionized biological imaging by enabling the visualization of cellular structures at the nanoscale, effectively bypassing the diffraction limit of light. However, the quantitative evaluation of SMLM reconstruction algorithms remains a significant challenge. Current standard practices typically separate performance into two distinct categories: detection metrics (e.g., Jaccard index, precision, and recall) and localization metrics (e.g., Root Mean Square Error, RMSE).

This bifurcation is problematic because detection and localization are intrinsically linked. Calculating RMSE requires an initial assignment of detected points to ground-truth points—a process that is often arbitrary and becomes increasingly unstable in the presence of false positives or missing detections. While empirical "efficiency" formulas have been developed to bridge this gap, they often lack a rigorous mathematical foundation and can fail in edge cases.

The Flat Metric Framework

To address these limitations, researchers have proposed the Flat Metric (also known as the Kantor-Rubinstein norm) as a unified evaluation tool. Rooted in the theory of Unbalanced Optimal Transport, the Flat Metric treats SMLM data as a collection of Dirac masses in space.

The metric calculates the minimum "cost" to transform the ground-truth point set into the detected point set. This optimization problem allows for three distinct operations:

  1. Transport: Moving a mass from one location to another, representing localization error.
  2. Destruction: Removing a mass that exists in the detection set but not in the ground truth (false positive).
  3. Creation: Adding a mass where one is missing in the detection set (false negative).
Visualization of the Flat Metric's implicit pairing mechanism. Lines represent transport (localization error), while crosses indicate points that are created or destroyed (detection errors).

A critical component of this framework is the parameter $\lambda$, which serves as a tolerance radius. If the distance between a detection and a ground-truth point exceeds $2\lambda$, the optimization determines it is more "cost-effective" to destroy the error and create a new point rather than transport it. This mechanism allows the Flat Metric to resolve point pairings implicitly, eliminating the need for manual assignment rules.

Experimental Validation

The Flat Metric was validated using both synthetic simulations and real-world data from the SMLM 2016 Challenge. In synthetic tests, 100 ground-truth points were subjected to varying levels of recall (0% to 100%) and localization jitter (0 to 250 nm). For the 2016 Challenge data, the metric was applied to four 2D datasets (Endoplasmic Reticulum and Microtubules) to benchmark 31 different participant algorithms.

Key Insights

  • Physical Interpretability: The Flat Metric is "homogeneous to nanometers," meaning its final score can be interpreted as an "equivalent localization accuracy." This provides researchers with an intuitive physical scale for error.
  • Unified Continuum: The metric offers a smooth transition between pure localization and pure detection. At large $\lambda$ values, it approximates the 1-Wasserstein distance; at small $\lambda$, it behaves like the Total Variation norm.
  • Robustness: Unlike empirical efficiency metrics, the Flat Metric remains well-defined and stable even in scenarios with 0% recall, making it a reliable tool for evaluating poorly performing algorithms.

Results and Discussion

The Flat Metric demonstrated a high degree of correlation with the "efficiency" metric used in the SMLM 2016 Challenge, providing a formal mathematical justification for the empirical success of such scoring systems. Furthermore, the Flat Metric proved superior at identifying reconstruction methods that struggled with complex, high-density datasets like microtubules.

Analysis of the Flat Metric (a) compared to ad-hoc Efficiency (b) across recall and radius parameters. Scatter plots (c) show the strong correlation between the two metrics across various SMLM datasets.

Computationally, the authors showed that the Flat Metric can be efficiently solved as a finite-dimensional linear program. This ensures that the metric is practical for integration into standard SMLM evaluation pipelines.

Conclusion

The introduction of the Flat Metric provides a mathematically rigorous and physically intuitive framework for assessing SMLM performance. By unifying detection and localization errors through optimal transport, it offers a more robust and consistent benchmarking tool than previous ad-hoc methods. Future work may extend this framework to 3D localization and explore entropic-regularized solvers to handle extremely high-density datasets more efficiently.

Summary updated

Abstract

We propose the use of Flat Metric to assess the performance of reconstruction methods for single-molecule localization microscopy (SMLM) in scenarios where the ground-truth is available. Flat Metric is intimately related to the concept of optimal transport between measures of different mass, providing solid mathematical foundations for SMLM evaluation and integrating both localization and detection performance. In this paper, we provide the foundations of Flat Metric and validate this measure by applying it to controlled synthetic examples and to data from the SMLM 2016 Challenge.
BibTeX
@inproceedings{denoyelle2020optimal,
  author = {Quentin Denoyelle and Thanh-An Pham and Pol del Aguila Pla and Daniel Sage and Michael Unser},
  title = {{Optimal-Transport-Based Metric For SMLM}},
  booktitle = {ISBI},
  year = {2020},
  doi = {10.1109/isbi48211.2021.9433874},
}
Figure