Assessing genomic reproducibility of read alignment tools

Pelin Icer Baykal, M. A. Simonov, Dhrithi Deshpande, Ful Belin Korukoglu, Jaden Moore, Karishma Chhugani, Chung-Shyan Liu, Varuni Sarwal, Neha Rajkumar, Mohammed Alser, Niko Beerenwinkel, Serghei MangulbioRxiv 2025 · DOI

TL;DR

This paper introduces a novel method using synthetic sequencing data to rigorously assess the reproducibility of DNA read alignment tools. They found that while most tools are robust to read order changes, some struggle with strand orientation shifts, and these inconsistencies can impact downstream variant calling.

What Problem Does This Paper Address?

Genomic research heavily relies on accurately mapping short DNA sequencing reads to a reference genome. This "read mapping" step is foundational for many analyses, like identifying genetic variations. However, the consistency of these mapping results across different experimental runs or processing pipelines (i.e., reproducibility) is often overlooked. While tools are benchmarked for accuracy and speed, their performance under subtle technical variations (like changes in read order or strand orientation) is less understood. This lack of understanding can lead to unreliable biological conclusions, especially when these mapping inconsistencies propagate to downstream analyses like variant calling.

Methods

The authors developed a methodology using synthetic replicates of sequencing data. They took original sequencing reads and created modified versions by:

  1. Shuffling: Randomly reordering the reads.
  2. Reverse Complementing: Flipping the base sequence and its quality scores.
  3. Combined Perturbation: Applying both shuffling and reverse complementing.

These synthetic replicates simulate common sources of technical variability in sequencing. They then aligned these synthetic reads using eight different popular read alignment tools and compared the results to the original alignments to quantify discrepancies. They evaluated reproducibility at both the global (mapping location) and local (edit distance) levels.

Datasets

The study used five paired-end Whole Genome Sequencing (WGS) Illumina datasets from the 1000 Genomes Project. These datasets are publicly available and considered high-quality, making them suitable for reproducibility benchmarking. No new benchmarks were introduced, but the synthetic replicates themselves serve as a novel evaluation dataset.

Key Insights and Ideas

  • Reproducibility is not a given: Even well-established alignment tools can exhibit significant differences in their output when subjected to minor changes in input data.
  • Strand orientation matters: Some tools are particularly sensitive to the directionality of reads (forward vs. reverse strand), leading to more inconsistencies when reads are reverse complemented.
  • High-quality filtering helps, but doesn't solve everything: Focusing on high-quality alignments improves consistency, but doesn't always restore perfect agreement, highlighting the importance of evaluating both mapping agreement and stability.
  • Alignment inconsistencies cascade downstream: The paper demonstrates that mapping errors can directly impact the accuracy of structural variant (SV) calls, underscoring the need for reproducible alignment as a prerequisite for reliable variant detection.

Key Results

  • Mapping rates are generally stable: Most tools maintained consistent mapping rates across synthetic replicates, with only minimap2 and SNAP showing noticeable changes under reverse complementing.
  • Variability in common reads: While most reads were commonly mapped (91.66% to 100%), the consistency of these mappings varied. Inconsistent mappings ranged from 0% to 13.53% for primary reads.
  • Tool-specific sensitivities:
    • Bowtie2 was fully reproducible under shuffling but showed reduced consistency with reverse complementing.
  • Subread was fully reproducible under reverse complementing but showed increased variability under shuffling.
  • SNAP and minimap2 showed the most significant variability under reverse complementing.
  • Downstream impact on SV calling: Tools that showed high alignment reproducibility (e.g., Bowtie2, HISAT2, minimap2) also maintained perfect SV concordance. Tools with lower alignment reproducibility (e.g., Subread) showed significantly lower SV concordance (as low as 87%).

Limitations and Future Work

  • Synthetic vs. Real-world variability: The authors acknowledge that synthetic replicates, while useful, don't fully capture the complexity of real-world sequencing variability (e.g., library preparation biases, sequencing chemistry artifacts).
  • Short reads only: The study focused on short paired-end Illumina reads. It's unclear if similar reproducibility patterns would hold for long-read sequencing technologies, which present different alignment challenges.
  • Future Work: The authors suggest extending this framework to benchmark multiple structural variant callers, incorporate more complex scenarios like multi-mapping and ambiguous alignments, and explore reproducibility in long-read sequencing. They advocate for adopting reproducibility as a fundamental benchmark in genomic studies and for tool developers to incorporate reproducibility metrics into their validation processes.

Summary updated

Abstract

Genomic research relies on accurate and reproducible computational analyses of DNA sequencing data to draw reliable biological conclusions. Read mapping, the process of aligning reads to a reference genome, is central to many applications, including variant detection and comparative genomics. While several tools have been developed for this task, genomic reproducibility <sup>1</sup> , defined as the consistency of results across replicates, remains underexplored. Here, we address this question by introducing a methodology based on synthetic replicates of sequencing data, generated by perturbing the original reads through shuffling, reverse complementing, or combined shuffling and reverse complementing. Our approach is able to simulate variability observed across sequencing runs due to differences in library preparation techniques. We evaluated the reproducibility of eight alignment tools (BWA-MEM2 <sup>2</sup> , Bowtie2 <sup>3</sup> , HISAT2 <sup>4</sup> , minimap2 <sup>5</sup> , NextGenMap <sup>6</sup> , SNAP <sup>7</sup> , SMALT <sup>7,8</sup> and Subread <sup>9</sup> ) under these perturbations using whole-genome sequencing (WGS) data. Synthetic replicates were aligned and compared to the original sample to quantify discrepancies. Mapping accuracy changes ranged from 0.0001% to 4.4% for primary reads, which are alignments not marked as secondary, supplementary, or duplicates, and up to 12.2% for high-quality primary reads. For primary reads, the percentage of reads commonly mapped in both the original and synthetic replicate ranged from 91.66% to 100% relative to the total number of mapped reads in the original dataset. High-quality filtering improved consistency, though some tools still failed to recover more than 70% of the original alignments. Within the set of common reads, the incidence of inconsistent mappings was as high as 13.53% for primary reads and 6.73% for high-quality primary reads. Bowtie 2 was fully reproducible under the shuffling replicate, Subread was fully reproducible under the reverse-complement replicate, whereas NextGenMap exhibited only minor inconsistencies. By contrast, SNAP and minimap2 displayed the most significant variability under reverse complementing. We further demonstrate that alignment inconsistencies propagate to the downstream task of calling structural genomic variants. Using Manta for structural-variant (SV) calling, we observed that Bowtie 2, HISAT2, and minimap2 maintained perfect SV concordance between original and replicate alignments, whereas other tools exhibited lower concordance, with Subread scoring only 87% concordance. In conclusion, our comprehensive evaluation demonstrates that synthetic perturbations reveal critical differences in how alignment tools handle technical variability and how these differences propagate to downstream variant analyses, underscoring the necessity of incorporating reproducibility benchmarks into the selection and validation of read mappers to ensure robust and reliable genomic interpretations.
BibTeX
@misc{baykal2025assessing,
  author = {Pelin Icer Baykal and M. A. Simonov and Dhrithi Deshpande and Ful Belin Korukoglu and Jaden Moore and Karishma Chhugani and Chung-Shyan Liu and Varuni Sarwal and Neha Rajkumar and Mohammed Alser and Niko Beerenwinkel and Serghei Mangul},
  title = {{Assessing genomic reproducibility of read alignment tools}},
  year = {2025},
  doi = {10.1101/2025.05.08.652934},
  howpublished = {bioRxiv},
}
Figure