Learning smoothing models of copy number profiles using breakpoint annotations

Toby Dylan Hocking, Gudrun Schleiermacher, Isabelle Janoueix-Lerosey, Valentina Boeva, Julie Cappo, Olivier Delattre, Francis R. Bach, Jean-Philippe VertBMC Bioinformatics 2013 · OpenReview

TL;DR

This paper introduces a novel method for selecting the best copy number profile smoothing model and its parameters by using expert visual annotations as a "gold standard." They created a new benchmark dataset of 575 annotated neuroblastoma profiles and demonstrated that their annotation-guided approach is quantitative and more effective than previous methods for real-world data.

What Problem Does This Paper Address?

Detecting copy number alterations (CNAs) in DNA is crucial for understanding diseases like cancer. While many computational models exist to smooth noisy DNA copy number profiles and identify breakpoints (regions where copy number changes), choosing the right model and its "smoothness" parameter is a significant challenge. This parameter controls how many breakpoints the model detects, and current methods often rely on heuristics or simulations that may not reflect real-world data. Biologists often visually inspect these profiles, but this process is time-consuming and subjective. The paper aims to bridge this gap by providing a quantitative, annotation-driven method for model selection that aligns with expert visual interpretation.

Methods

The authors propose a three-pronged approach:

  1. Annotation-Guided Model Training: Instead of relying on simulated data or subjective heuristics, they use expert annotations of breakpoint regions as a ground truth. They developed a protocol to select the model and its smoothing parameter that best matches these visual annotations.
  2. Cross-Validation for Error Estimation: They employ cross-validation techniques to estimate the false positive and false negative rates of trained models on real data, providing a quantitative measure of their accuracy.
  3. Benchmarking and Comparison: They applied their methods to compare 17 different smoothing algorithms on a newly created dataset of 575 annotated neuroblastoma copy number profiles.

A key innovation is defining annotations not as precise breakpoint locations, but as regions with an expected number of breakpoints (e.g., 0, 1, or >1). This makes annotation more practical for human experts. They also distinguish between "global" models (trained on a database of annotations from multiple profiles) and "local" models (trained on annotations from a single profile).

Datasets

The primary dataset used is a new collection of 575 annotated neuroblastoma copy number profiles. This dataset was created by two experts, resulting in two annotation databases:

  • "Original" annotations: Created systematically by labeling 6 predefined regions per chromosome as containing 0 or >0 breakpoints.
  • "Detailed" annotations: Created using a GUI that allowed for more flexible region drawing and included 1-breakpoint annotations.

This dataset is made publicly available as a benchmark for future research.

Key Insights and Ideas

  • Expert annotations are a powerful "gold standard": Using visual annotations from experienced biologists as a ground truth provides a quantitative and practical way to evaluate and select copy number profile smoothing models.
  • Global models generalize better: Models trained on a collection of annotations from multiple profiles (global models) tend to perform better on unseen data than models trained on a single profile (local models), even though local models might fit the training annotations more closely.
  • Smoothness parameter tuning is critical: Default parameters for many algorithms perform poorly, highlighting the importance of finding the optimal smoothness parameter for accurate breakpoint detection.
  • Annotation-based training is feasible and efficient: Annotating even a small subset of profiles (around 10) was sufficient to train a global model that achieved near-optimal performance, suggesting that the time investment in annotation can be highly beneficial.

Key Results

  • Best performing models: The cghseg.k and pelt.n algorithms, when trained using the annotation-guided approach (global models), consistently showed the best breakpoint detection accuracy.
  • Error rates: The best global models achieved a test error of around 7.7% on the detailed annotation dataset.
  • Comparison to default models: Global models trained with annotations significantly outperformed default models, which often showed high false positive and false negative rates.
  • Computational efficiency: While cghseg.k and pelt.n were the most accurate, cghseg.k was found to be faster than pelt.n for global model training.
  • Generalization across annotators: Models trained on one annotation dataset generalized well to the other, indicating robustness to variations in expert interpretation.

Limitations and Future Work

  • Annotation time: While the authors argue that annotation is feasible and efficient, creating these datasets still requires expert time.
  • Non-zero error: Even the best models exhibit some training and testing error, suggesting room for improvement.
  • Future directions:
    • Applying annotation-based model training to other algorithms and data types.
  • Developing models that can directly incorporate annotations for joint breakpoint detection and copy number calling.
  • Investigating chromosome-specific parameter tuning for further error reduction.

Summary updated

Abstract

Background Many models have been proposed to detect copy number alterations in chromosomal copy number profiles, but it is usually not obvious to decide which is most effective for a given data set. Furthermore, most methods have a smoothing parameter that determines the number of breakpoints and must be chosen using various heuristics. Results We present three contributions for copy number profile smoothing model selection. First, we propose to select the model and degree of smoothness that maximizes agreement with visual breakpoint region annotations. Second, we develop cross-validation procedures to estimate the error of the trained models. Third, we apply these methods to compare 17 smoothing models on a new database of 575 annotated neuroblastoma copy number profiles, which we make available as a public benchmark for testing new algorithms. Conclusions Whereas previous studies have been qualitative or limited to simulated data, our annotation-guided approach is quantitative and suggests which algorithms are fastest and most accurate in practice on real data. In the neuroblastoma data, the equivalent pelt.n and cghseg.k methods were the best breakpoint detectors, and exhibited reasonable computation times.
BibTeX
@article{hocking2013learning,
  author = {Toby Dylan Hocking and Gudrun Schleiermacher and Isabelle Janoueix-Lerosey and Valentina Boeva and Julie Cappo and Olivier Delattre and Francis R. Bach and Jean-Philippe Vert},
  title = {{Learning smoothing models of copy number profiles using breakpoint annotations}},
  journal = {BMC Bioinformatics},
  year = {2013},
  url = {https://openreview.net/forum?id=6PhZ_tG5FZw},
}

Loading related content...

Figure