Representative transcript sets for evaluating a translational initiation sites predictor

Jia Zeng, Reda Alhajj, Douglas J. DemetrickBMC Bioinformatics 2009 · DOI

TL;DR

This paper introduces a novel algorithm for constructing high-quality, representative benchmark datasets of mRNA sequences for evaluating translational initiation site (TIS) prediction tools. By filtering sequences based on protein structural properties (molecular weight, isoelectric point, and hydrophobicity), they generated datasets that better reflect the "real world" protein population, leading to more accurate assessments of TIS predictors.

What Problem Does This Paper Address?

Predicting translational initiation sites (TISs) in mRNA sequences is a fundamental task in bioinformatics, crucial for understanding protein synthesis. To accurately assess the performance of new TIS prediction algorithms, reliable and representative benchmark datasets are essential. However, existing datasets often suffer from issues like sequencing errors, lack of representativeness of the actual expressed protein population, redundancy, or inaccessibility. This makes it difficult to rigorously compare different prediction methods and understand their true performance in a biological context. The authors aim to address this by developing a generalizable method to create high-quality, representative, and accessible benchmark datasets.

Methods

The core of the paper is a three-phase algorithm for constructing these representative transcript collections.

  • Phase I: Initial Retrieval and Filtering: The process begins by retrieving a large collection of mRNA sequences from the NCBI RefSeq database. Sequences are then filtered to remove those with non-standard amino acids, start codons other than AUG, or data format errors.
  • Phase II: Representative Selection: This phase focuses on selecting sequences whose corresponding protein products exhibit characteristics representative of the organism's general protein population. This is achieved by analyzing three key structural parameters: molecular weight (MW), isoelectric point (pI), and hydrophobicity index (HI) profile. The algorithm identifies sequences whose MW and pI fall within specific ranges observed in proteomic studies, and then further filters based on hydrophobicity profiles. This phase is further broken down into intermediate steps to manage computational complexity.
  • Phase III: Redundancy Elimination and Refinement: In this final phase, sequences are refined to ensure they are non-redundant and suitable for TIS prediction. This involves removing sequences with short 5' UTRs (less than 12 nucleotides, as many TIS predictors rely on this region) and eliminating redundant copies of homologous proteins using BLAST. This step aims to create a dataset where each protein family is represented by a single transcript.

Datasets

The authors generated four representative transcript collections for the following model organisms:

  • Homo sapiens (human)
  • Mus musculus (house mouse)
  • Caenorhabditis elegans (roundworm)
  • Drosophila melanogaster (fruit fly)

These datasets were constructed in three phases, with the number of sequences decreasing significantly at each stage (as shown in Table 1). The paper also references and compares their results against existing benchmark datasets, including those from Pedersen and Nielsen [1], Nadershahi et al. [2], Hatzigeorgiou [3], and Saeys et al. [4].

Key Insights and Ideas

  • Protein structural properties as a proxy for representativeness: The core idea is that by filtering mRNA sequences based on the molecular weight, isoelectric point, and hydrophobicity of their encoded proteins, one can create a dataset that mirrors the distribution of proteins actually found in cellular proteomic studies. This is a more robust approach than simply using raw genomic or EST data.
  • The "average" protein population is not uniformly distributed: The MW-pI plots (Figures 1, 2, 3) reveal distinct clusters and gaps, indicating that the expressed protein population is not evenly distributed across all possible MW and pI values. This highlights the importance of using representative datasets rather than arbitrary ones.
  • Bimodal distribution of isoelectric points: The analysis consistently showed a bimodal distribution in the isoelectric point (pI) of proteins, with peaks around neutral pH. This observation aligns with previous proteomic studies and suggests a potential functional significance.
  • Importance of non-redundancy for homologous proteins: While homology is used as a criterion for redundancy elimination, the authors acknowledge that similar proteins might have different TIS contexts, which could still affect predictor accuracy. However, for a general benchmark, this approach is deemed practical.
  • Limitations of existing benchmarks: The paper implicitly and explicitly criticizes existing datasets for issues like sequencing errors, limited scope, and inaccessibility, underscoring the need for their proposed method.

Key Results

  • The algorithm successfully generated four representative transcript collections for human, mouse, C. elegans, and D. melanogaster.
  • Statistical analysis of these datasets (Tables 2-8) confirmed that the protein products generally fall within ranges consistent with published proteomic data (e.g., MW 20-70 KDa, pI 5-9).
  • The generated datasets were used to evaluate six existing TIS predictors. The results (Table 11) showed that predictors like GENSCAN, TISHunter, and CUB performed well on these new datasets, with GENSCAN showing inconsistent performance on other benchmark datasets.
  • A comparative study with existing benchmark datasets (Table 12) highlighted that some predictors exhibit inconsistent performance across different datasets, further validating the need for the authors' proposed representative datasets.

Limitations and Future Work

  • Arbitrary parameter ranges: The specific ranges chosen for MW and pI in Phase II (e.g., 20-70 KDa for MW, 5-9 for pI) are acknowledged as somewhat arbitrary and based on human proteomic data. While the algorithm itself is general, applying it to a new organism might require careful selection or investigation of these parameters. The authors suggest a strategy for dealing with this by dividing ranges into subgroups.
  • Homology as a redundancy criterion: While effective, using homology alone to eliminate redundant proteins might overlook subtle differences in TIS context that could impact predictor accuracy.
  • Future work: The authors suggest that their method can be extended to create more selective protein datasets by integrating additional protein structural parameters, which could facilitate studies on specific protein structures. They also propose a strategy for constructing datasets when specific parameter ranges are not predefined.

Summary updated

BibTeX
@article{zeng2009representative,
  author = {Jia Zeng and Reda Alhajj and Douglas J. Demetrick},
  title = {{Representative transcript sets for evaluating a translational initiation sites predictor}},
  journal = {BMC Bioinformatics},
  year = {2009},
  doi = {10.1186/1471-2105-10-206},
}
Figure