Comment on mr-2021-59

Yulia Pustovalova, Frank Delaglio, D. Levi Craft, Haribabu Arthanari, Ad Bax, Martin Billeter, Mark J. Bostock, Hesam Dashti, D. Flemming Hansen, Sven G. Hyberts, Bruce A. Johnson, Krzysztof Kazimierczuk, Hengfa Lu, Mark W. Maciejewski, Tomas M. Miljenović, Mehdi Mobli, Daniel Nietlispach, Vladislav Orekhov, Robert Powers, Xiaobo Qu, Scott A. Robson, David Rovnyak, Gerhard Wagner, Jinfa Ying, Matthew Zambrello, Jeffrey C. Hoch, David L. Donoho, Adam D. SchuylerOpenalex Unknown 2021 · DOI · OpenAlex

Abstract. Although the concepts of nonuniform sampling (NUS) and non-Fourier spectral reconstruction in multidimensional NMR began to emerge 4 decades ago, it is only relatively recently that NUS has become more commonplace. Advantages of NUS include the ability to tailor experiments to reduce data collection time and to improve spectral quality, whether through detection of closely spaced peaks (i.e., “resolution”) or peaks of weak intensity (i.e., “sensitivity”). Wider adoption of these methods is the result of improvements in computational performance, a growing abundance and flexibility of software, support from NMR spectrometer vendors, and the increased data sampling demands imposed by higher magnetic fields. However, the identification of best practices still remains a significant and unmet challenge. Unlike the discrete Fourier transform, non-Fourier methods used to reconstruct spectra from NUS data are nonlinear, depend on the complexity and nature of the signals, and lack quantitative or formal theory describing their performance. Seemingly subtle algorithmic differences may lead to significant variabilities in spectral qualities and artifacts. A community-based critical assessment of NUS challenge problems has been initiated, called the “Nonuniform Sampling Contest” (NUScon), with the objective of determining best practices for processing and analyzing NUS experiments. We address this objective by constructing challenges from NMR experiments that we inject with synthetic signals, and we process these challenges using workflows submitted by the community. In the initial rounds of NUScon our aim is to establish objective criteria for evaluating the quality of spectral reconstructions. We present here a software package for performing the quantitative analyses, and we present the results from the first two rounds of NUScon. We discuss the challenges that remain and present a roadmap for continued community-driven development with the ultimate aim of providing best practices in this rapidly evolving field. The NUScon software package and all data from evaluating the challenge problems are hosted on the NMRbox platform.

The Challenge of Nonuniform Sampling

Multidimensional NMR spectroscopy is a vital tool for structural biology, but it is traditionally slow because it requires "uniform sampling"—collecting data points at regular intervals to satisfy the Nyquist theorem. Nonuniform sampling (NUS) allows researchers to skip most of these points, potentially saving weeks of instrument time or enabling much higher resolution.

The catch is that reconstructing a spectrum from incomplete data requires complex, nonlinear algorithms. Unlike the standard Fourier Transform, these methods lack a solid theoretical framework to predict their performance. For years, the field relied on heuristics and anecdotal evidence, making it nearly impossible to determine which software or sampling schedule was actually "best." NUScon was created to replace these subjective assessments with rigorous, standardized metrics.

Methodology: Synthetic Signal Injection

The authors developed a modular, automated workflow hosted on the NMRbox platform. The core innovation is the synthetic signal injection procedure, which establishes a "ground truth" to evaluate reconstruction fidelity.

The NUScon workflow: from experimental baseline to quantitative scoring via synthetic signal injection.

As illustrated in the workflow above, the process involves:

  1. Baseline Data: Starting with high-quality, uniformly sampled experimental data.
  2. Injection: Identifying "empty" regions in the spectrum and injecting synthetic peaks with precisely known frequencies, amplitudes, and phases.
  3. Subsampling: Masking the data using various NUS schedules (like exponentially biased sampling) to create challenge datasets.
  4. Reconstruction: Contestants use their preferred algorithms (e.g., Iterative Soft Thresholding, Maximum Entropy, or Low Rank) to reconstruct the full spectrum.
  5. Quantitative Scoring: Comparing reconstructed peaks against the known "truth" using five specific metrics (M1-M5) covering frequency accuracy, linearity of intensity, and true/false positive rates.

Experimental Datasets

The paper utilizes experimental data from three distinct protein targets to ensure diversity in signal complexity:

  • Protein A: An 89-residue protein containing both a well-folded domain and an intrinsically disordered region.
  • Protein B: A 147-residue protein (16 kDa) known to form dimers.
  • Protein C: A 107-residue, well-behaved 12 kDa protein.

The challenges utilized common triple-resonance experiments (HNCA, HNCACB) and NOESY-HSQC experiments. The resulting NUScon archive serves as a public benchmark for the NMR community.

Key Insights and Results

The 2018 and 2019 challenge rounds involved 12 different software packages and over 600 spectral reconstructions, requiring over 19,000 CPU hours.

Visual representation of synthetic peaks (green) injected into an experimental NMR spectrum (blue) for evaluation.

Key findings include:

  • The Necessity of Synthetic Truth: Because NUS is often used to achieve resolution beyond what a standard Fourier Transform can provide, synthetic peaks are the only reliable way to measure absolute error.
  • Rankings over Raw Scores: To prevent a single high-performing category from skewing results, the authors used percentile rankings to aggregate scores across different metrics.
  • Algorithm Specificity: There is no "one size fits all" algorithm. Some methods excel at sensitivity (detecting weak peaks), while others are superior at maintaining linearity (the mathematical relationship between peak heights).
  • Validation: Visual inspections confirmed that the quantitative metrics (M1-M5) accurately reflected the qualitative "look" of the spectra, proving the metrics are reliable proxies for human expert judgment.

Future Directions

While current synthetic signals are "ideal" exponential decays, future rounds aim to include more realistic artifacts like phase noise and non-exponential decay. A major planned upgrade is the transition to In-situ Receiver Operator Characteristic (IROC) analysis to remove human bias from detection scores by evaluating across a continuous range of thresholds. The project also plans to expand into 4D NMR and small molecule applications.

Summary updated

Abstract

<strong class="journal-contentHeaderColor">Abstract.</strong> Although the concepts of nonuniform sampling (NUS​​​​​​​) and non-Fourier spectral reconstruction in multidimensional NMR began to emerge 4 decades ago <span class="cit" id="xref_paren.1">(<a href="#bib1.bibx4">Bodenhausen and Ernst</a>, <a href="#bib1.bibx4">1981</a>; <a href="#bib1.bibx2">Barna and Laue</a>, <a href="#bib1.bibx2">1987</a>)</span>, it is only relatively recently that NUS has become more commonplace. Advantages of NUS include the ability to tailor experiments to reduce data collection time and to improve spectral quality, whether through detection<span id="page844"/> of closely spaced peaks (i.e., “resolution”) or peaks of weak intensity (i.e., “sensitivity”). Wider adoption of these methods is the result of improvements in computational performance, a growing abundance and flexibility of software, support from NMR spectrometer vendors, and the increased data sampling demands imposed by higher magnetic fields. However, the identification of best practices still remains a significant and unmet challenge. Unlike the discrete Fourier transform, non-Fourier methods used to reconstruct spectra from NUS data are nonlinear, depend on the complexity and nature of the signals, and lack quantitative or formal theory describing their performance. Seemingly subtle algorithmic differences may lead to significant variabilities in spectral qualities and artifacts. A community-based critical assessment of NUS challenge problems has been initiated, called the “Nonuniform Sampling Contest” (NUScon), with the objective of determining best practices for processing and analyzing NUS experiments. We address this objective by constructing challenges from NMR experiments that we inject with synthetic signals, and we process these challenges using workflows submitted by the community. In the initial rounds of NUScon our aim is to establish objective criteria for evaluating the quality of spectral reconstructions. We present here a software package for performing the quantitative analyses, and we present the results from the first two rounds of NUScon. We discuss the challenges that remain and present a roadmap for continued community-driven development with the ultimate aim of providing best practices in this rapidly evolving field. The NUScon software package and all data from evaluating the challenge problems are hosted on the NMRbox platform.
BibTeX
@inproceedings{pustovalova2021comment,
  author = {Yulia Pustovalova and Frank Delaglio and D. Levi Craft and Haribabu Arthanari and Ad Bax and Martin Billeter and Mark J. Bostock and Hesam Dashti and D. Flemming Hansen and Sven G. Hyberts and Bruce A. Johnson and Krzysztof Kazimierczuk and Hengfa Lu and Mark W. Maciejewski and Tomas M. Miljenović and Mehdi Mobli and Daniel Nietlispach and Vladislav Orekhov and Robert Powers and Xiaobo Qu and Scott A. Robson and David Rovnyak and Gerhard Wagner and Jinfa Ying and Matthew Zambrello and Jeffrey C. Hoch and David L. Donoho and Adam D. Schuyler},
  title = {{Comment on mr-2021-59}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.5194/mr-2021-59-rc2},
}
Figure