TL;DR
The nGASP project benchmarked gene prediction software for nematode genomes, revealing that "combiner" algorithms, which integrate multiple prediction methods, achieve the highest accuracy. This work establishes a crucial baseline for nematode genome annotation and has guided the creation of new gene sets for several Caenorhabditis species and Brugia malayi.
What Problem Does This Paper Address?
Accurately identifying protein-coding genes within a genome is a fundamental step in biological research. While the C. elegans genome is well-annotated, many other nematode genomes are being sequenced, and there was a lack of knowledge regarding which gene prediction tools perform best on these genomes. This gap hinders efficient and reliable annotation of these new genomic resources. The nGASP project aimed to address this by objectively assessing the accuracy of various gene prediction software on C. elegans and using these findings to inform the annotation of other nematode genomes.
Methods
The nGASP project involved a competition where 17 research groups submitted gene predictions for 10 Mb of the C. elegans genome. The competition was structured into four categories:
- Category 1 (ab initio): Gene prediction based solely on genomic sequence.
- Category 2 (multi-genome alignments): Gene prediction utilizing alignments between multiple nematode genomes.
- Category 3 (expressed sequence alignments): Gene prediction incorporating alignments of ESTs, mRNAs, and proteins.
- Category 4 (combiners): Gene prediction systems that integrate gene models from other predictors and any of the data types used in categories 1-3.
Predictions were evaluated using two reference gene sets from WormBase: ref1 (for sensitivity) and ref2 (for specificity). Accuracy was measured at base, exon, isoform, and gene levels. The study also investigated specific gene features that correlated with prediction difficulty.
Datasets
- Genomic DNA: 10 Mb of the C. elegans genome was used for both training and testing. These regions were selected based on gene density and conservation.
- Auxiliary Training Data: Provided to participants, including:
- Repeat coordinates.
- Coordinates of coding exons, introns, and UTRs from confirmed and unconfirmed isoforms.
- DNA sequences for C. briggsae and C. remanei.
- Multi-genome alignments between C. elegans, C. briggsae, and C. remanei.
- Amino acid sequences of proteins with BLAST matches.
- Nucleotide sequences of C. elegans ESTs/cDNAs with BLAT matches.
- Coordinates of BLAST and BLAT matches.
- Reference Gene Sets: Two sets derived from WormBase (WS160):
ref1: Confirmed isoforms for sensitivity assessment.ref2: All isoforms (confirmed and unconfirmed) for specificity assessment.
Key Insights and Ideas
- Combiners Reign Supreme: The most striking finding is the superior performance of "combiner" algorithms. These systems, by integrating the strengths of multiple prediction methods and diverse data sources, significantly outperform individual predictors.
- Nematode Genome Specificity: While combiners in C. elegans showed slightly lower specificity than in human genomes (as reported by EGASP), their sensitivity was higher. This suggests that the C. elegans genome, or its annotation, presents unique challenges.
- Predicting the Difficult: The study identified specific gene features that make prediction challenging. These include unusual exon hexamer content, a high number of exons, very short exons, long introns, weak start signals, weak splice sites, poorly conserved orthologs, and genes with many isoforms. This provides a roadmap for improving future gene prediction tools.
- Ab Initio vs. Evidence-Based: Gene predictors relying on expressed sequence data (ESTs, proteins) generally performed better than ab initio predictors, highlighting the value of experimental evidence.
Key Results
- Overall Accuracy: The best submitted gene sets achieved base-level sensitivity >99% and specificity >93%.
- Gene Level Performance:
- Combiners achieved a median gene-level sensitivity of 78% and specificity of 42%.
- Gene-finders using expressed sequence alignments had a median sensitivity of 68% and specificity of 39%.
- Ab initio gene-finders had a median sensitivity of 54% and specificity of 32%.
- Isoform Prediction Challenges: Isoform-level prediction remains a significant challenge, with the best gene sets achieving only ~66% sensitivity and ~56% specificity.
- Difficult Genes: Genes with features like unusual hexamer content, many exons, short exons, long introns, weak signals, or poor ortholog conservation were significantly harder to predict.
- New Gene Sets: The project has led to the creation of new gene sets for C. briggsae, C. remanei, C. brenneri, C. japonica, and Brugia malayi using the best-performing gene-finders.
Limitations and Future Work
- Reference Set Bias: The authors acknowledge that differences in reference set definitions between nGASP and previous projects (like EGASP) might contribute to observed accuracy variations.
- Combiner Improvement: The accuracy of combiners could be further enhanced by improving the performance of their constituent gene prediction sets.
- Future Directions: The identified difficult gene features provide clear targets for future development of more accurate gene prediction algorithms. The newly generated gene sets will serve as a foundation for further annotation and comparative genomic studies in nematodes.
Summary updated