REPIC — an ensemble learning methodology for cryo-EM particle picking

Christopher J. F. Cameron, Sebastian Jh Seager, Fred J. Sigworth, Hemant D. Tagare, Mark GersteinbioRxiv 2023 · DOI

TL;DR

This paper introduces REPIC, a novel ensemble learning method for cryo-EM particle picking that combines outputs from multiple "picker" algorithms to identify a consensus set of high-quality particles. REPIC significantly reduces the need for manual intervention and expert knowledge, enabling high-resolution reconstructions even for challenging datasets.

What Problem Does This Paper Address?

Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the 3D structure of biological molecules. A critical early step in this process is "particle picking," where individual 2D projections of molecules (particles) are identified within noisy electron micrographs. This is challenging due to low signal-to-noise ratios and the absence of a definitive "ground truth" for particle locations. Current automated picking tools, known as "pickers," often produce different sets of particles, making it difficult for researchers to select the best picker or combine their outputs effectively. This often leads to a significant manual effort in filtering and selecting particles, which is time-consuming, subjective, and requires considerable expertise. The paper aims to address this bottleneck by providing a robust, automated method for generating high-quality particle sets.

Methods

REPIC employs an ensemble learning approach to reconcile the outputs of multiple particle pickers. The core of REPIC involves three main steps:

  1. Graph Building: Each identified particle from different pickers is represented as a vertex in a computational graph. An edge is created between two vertices if their corresponding particle bounding boxes significantly overlap (measured by the Jaccard Index). Crucially, edges are only formed between particles from different pickers.
  2. Clique Finding: The algorithm identifies "cliques" of size k (where k is the number of pickers used) in this graph. A clique represents a group of k particles, each from a different picker, that all significantly overlap with each other. These cliques are potential candidates for consensus particles.
  3. Clique Optimization: To select the optimal set of consensus particles, REPIC frames this as an integer linear programming (ILP) problem. This optimization aims to find a disjoint set of cliques that maximizes the overall particle overlap and the confidence scores provided by the individual pickers, ensuring that each particle is assigned to only one consensus particle.

REPIC offers two modes: "one-shot" for immediate consensus picking and "iterative" mode, which uses the consensus particles from one iteration to retrain the pickers for subsequent, improved iterations. This iterative approach is particularly useful for "difficult-to-pick" particles or when starting with minimal manual input.

Datasets

The study evaluated REPIC on four publicly available cryo-EM datasets from the EMPIAR resource:

  • TRPV1 (EMPIAR-10005)
  • β-galactosidase (β-gal - EMPIAR-10017)
  • T20S proteasome (EMPIAR-10057)
  • Fatty acid synthase (FAS - EMPIAR-10454)

These datasets were chosen because they contain high-quality particles that have been used to generate published reconstructions, serving as a "normative" benchmark. The authors also note that for the FAS dataset (EMPIAR-10454), a subset of micrographs was used due to its large size.

Key Insights and Ideas

  • Ensemble Power: Combining multiple, imperfect pickers can yield a more reliable and higher-quality particle set than any single picker alone, especially when the best picker for a given dataset is unknown.
  • Graph-Based Consensus: Framing particle picking as a graph problem with clique optimization provides a principled, non-greedy way to reconcile diverse picker outputs.
  • Iterative Improvement: The iterative retraining loop is a powerful mechanism for bootstrapping pickers, allowing them to learn from consensus particles and progressively improve their performance, even from minimal initial training data.
  • Reduced Manual Effort: REPIC significantly alleviates the burden of manual particle selection and filtering, democratizing cryo-EM analysis by reducing the reliance on expert knowledge.
  • Transfer Learning Potential: The method shows promise for transfer learning, where pickers trained on one dataset can be used to initiate picking on a related, but different, dataset.

Key Results

  • High-Quality Reconstructions: REPIC consistently produced high-resolution reconstructions comparable to, or even exceeding, those obtained from normative particle sets. These reconstructions were achieved without the need for downstream 2D or 3D classification, which are often subjective and non-reproducible.
  • Improved Precision and Recall: In "one-shot" mode, REPIC achieved higher precision and recall compared to individual out-of-the-box pickers across most datasets.
  • Effective Iterative Mode: The iterative mode, even when initialized with a very small number of manually picked particles (as few as 1-3 per micrograph), successfully bootstrapped pickers to achieve high-resolution reconstructions. For example, in the T20S proteasome dataset, DeepPicker's reconstruction improved from a disconnected map to a high-resolution map (3.37 Å).
  • Robustness: REPIC's consensus particle sets remained stable across iterations, even when individual pickers showed instability, demonstrating the robustness of the ensemble approach.

Limitations and Future Work

  • Runtime: The authors acknowledge that the graph building step, particularly the exhaustive search for overlapping boxes, is the current bottleneck for REPIC's runtime. They suggest using k-d trees to optimize this process.
  • Parallelization: The iterative picker retraining loop is currently sequential. Future work could involve parallelizing picker applications or re-training only the worst-performing pickers to speed up iterations.
  • Dataset Dissimilarity: Transfer learning between highly dissimilar datasets can still fail, as observed with the B-gal to T20S proteasome transfer, highlighting the importance of dataset similarity.

Future work will focus on improving the computational efficiency of both one-shot and iterative modes and exploring further applications of ensemble learning in cryo-EM.

Summary updated

Abstract

Cryo-EM (cryogenic electron microscopy) particle identification from micrographs (i.e., picking) is challenging due to the low signal-to-noise ratio and lack of ground truth for particle locations. Moreover, current computational methods (“pickers”) identify different particle sets, complicating the selection of the best-suited picker for a protein of interest. Here, we present REPIC, an ensemble learning methodology that uses multiple pickers to find consensus particles. REPIC identifies consensus particles by framing its task as a graph problem and using integer linear programming to select particles. REPIC picks high-quality particles when the best picker is not known a priori and for known difficult-to-pick particles (e.g., TRPV1). Reconstructions using consensus particles achieve resolutions comparable to those from particles picked by experts, without the need for downstream particle filtering. Overall, our results show REPIC requires minimal (often no) manual picking and significantly reduces the burden on cryo-EM users for picker selection and particle picking. Availability https://github.com/ccameron/REPIC
BibTeX
@misc{cameron2023repic,
  author = {Christopher J. F. Cameron and Sebastian JH Seager and Fred J. Sigworth and Hemant D. Tagare and Mark Gerstein},
  title = {{REPIC — an ensemble learning methodology for cryo-EM particle picking}},
  year = {2023},
  doi = {10.1101/2023.05.13.540636},
  howpublished = {bioRxiv},
}
Figure