TL;DR
This paper introduces "indirect two-sided relative ranking," a novel method for comparing gene expression profiles that is more robust to experimental variations (like batch effects) than existing direct comparison methods. By leveraging the similarity of each sample to the entire database, this approach significantly improves the ability to discover similar drugs and classify biological samples, leading to richer scientific discoveries.
What Problem Does This Paper Address?
Gene expression data, generated from microarrays, holds immense potential for understanding biological processes, diseases, and drug effects. However, this data is notoriously noisy and difficult to compare directly due to variations introduced by different experimental conditions (e.g., different labs, reagents, or processing batches). These "experimental barriers" often prevent researchers from accurately combining and analyzing data from diverse sources, hiding valuable insights. Existing methods, like the Connectivity Map (CMAP), attempt to address this by comparing gene expression profiles, but they can still be sensitive to these experimental variations. The authors aim to develop a more robust similarity measure that can effectively bridge these experimental gaps.
Methods
The core of the paper is the introduction of an "indirect two-sided relative ranking" method. Unlike direct methods that compare two gene expression profiles in isolation, this approach compares how each of the two profiles relates to all other profiles in the database.
- Direct Approach (CMAP-like): This method, exemplified by CMAP, uses a truncated Kolmogorov-Smirnov (KS) statistic to compare the ranked lists of "up-regulated" (uptags) and "down-regulated" (downtags) genes between two samples. This statistic captures whether the uptags of one sample are also highly ranked in the other, and vice-versa.
- Indirect Approach (Novel): The authors extend the direct approach by calculating the KS statistic for every pair of samples against the entire database. Then, they measure the similarity between two samples (query
qand targett) by computing the Spearman rank correlation between their respective lists of KS statistics against all other samples in the database. Essentially, ifqandtare similar to the same set of other samples in the database, they are considered similar to each other. This indirect comparison leverages the collective information within the database to filter out noise and experimental artifacts.
Datasets
The authors evaluate their method on several datasets:
- Vanda GEPedia Dataset: A large, proprietary dataset from Vanda Pharmaceuticals, used for drug discovery. This dataset is described as having a substantial number of replicates for many compounds.
- Broad Institute Dataset: A public dataset containing 453 samples, released with the Connectivity Map tool.
- Three smaller public datasets:
- Molecular Classification of Cancer: Distinguishing between acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL).
- Predicting Drug Sensitivity/Resistance: Identifying sensitivity or resistance to glucocorticoid treatment in childhood ALL.
- Classifying (Related) Cell Types: Distinguishing between megakaryocyte-erythrocyte progenitors (MEP), megakaryocytes, and erythrocytes.
Key Insights and Ideas
- Leveraging the Collective: The most significant insight is that comparing samples not in isolation, but by how they relate to the entire dataset, can reveal more robust similarities. This "wisdom of the crowd" approach effectively filters out noise specific to individual experiments.
- Robustness to Experimental Barriers: The indirect method demonstrably outperforms direct methods in scenarios with known experimental noise (vehicle and batch effects), suggesting its applicability to real-world, heterogeneous biological data.
- Improved Discovery Potential: By recalling more true positives (similar drugs, same class samples), the indirect method offers a more powerful tool for hypothesis generation and scientific discovery, potentially saving significant experimental resources.
- Generalizability: The method's success across diverse tasks (drug discovery, cancer classification, cell type differentiation) highlights its broad applicability to various gene expression analysis problems.
Key Results
- Drug Discovery (Vanda GEPedia): The indirect method achieved a 97.03% improvement in recall at rank 10 for similar drugs across different vehicles and batches compared to the direct method.
- Drug Discovery (Broad Dataset): A 49.44% improvement in recall at rank 10 was observed on this public dataset.
- Classification Tasks:
- Cancer Classification (AML vs. ALL): The indirect method showed a statistically significant improvement in AUC (0.829 vs. 0.727).
- Drug Sensitivity/Resistance: The indirect method also showed a statistically significant improvement in AUC (0.635 vs. 0.608).
- Cell Type Classification: The indirect method outperformed the direct method with a statistically significant p-value.
- Special Cases: The indirect method was able to find relevant similarities in cases where the direct method returned zero results.
Limitations and Future Work
- Computational Complexity: While the indirect method maintains the same O(n²) time complexity order as direct methods, the constant factors can be higher due to the initial KS statistic calculation. However, the authors note that their current implementation benefits from parallelization.
- Dependence on Direct Method: The indirect method's performance is dependent on the quality of the underlying direct similarity measure. If better direct methods emerge, the indirect approach can be readily adapted.
- Optimal k Selection: The paper uses a fixed
k(number of top/bottom probes) for defining uptags/downtags and a fixed rank threshold for evaluation. Future work could explore more sophisticated methods for selecting the optimalkfor different datasets. - Broader Comparisons: The authors suggest future research could involve more complex comparisons, such as analyzing groups of compounds together, and a more thorough evaluation of selecting the optimal
k.
Summary updated