TL;DR
A recent paper claimed that 2D embeddings like t-SNE and UMAP are misleading for single-cell data, comparing them to an arbitrary elephant shape. This paper refutes that claim, showing that while these embeddings distort distances, they effectively preserve crucial biological information like cell neighborhoods and types, making them valuable tools for exploration.
What Problem Does This Paper Address?
In single-cell genomics, researchers often use dimensionality reduction techniques like t-SNE and UMAP to visualize high-dimensional gene expression data in 2D. This allows for easier identification of cell populations and patterns. However, a recent study by Chari and Pachter argued that these 2D embeddings are fundamentally flawed, producing "arbitrary" and "specious" shapes that don't reflect true biological structure. They proposed that these embeddings are no better than forcing data into an arbitrary shape, like an elephant, using a method called Picasso. This paper directly challenges this conclusion, arguing that the metrics used in the original study were insufficient and failed to capture the true utility of t-SNE and UMAP for biological interpretation.
Methods
The authors critically re-evaluate the metrics used by Chari and Pachter, which focused solely on preserving distances between data points. They argue that this is an incomplete picture for biological data. To provide a more comprehensive assessment, they introduce and utilize four additional metrics commonly used in benchmark studies:
- k-nearest-neighbor (kNN) accuracy: Measures how often the 2D neighbors of a data point belong to the same biological class.
- kNN recall: Measures how often the 2D neighbors of a data point are also neighbors in the original high-dimensional space.
- Silhouette coefficient: Quantifies how compact and well-separated cell clusters are in the 2D embedding.
- Adjusted Mutual Information (AMI): Evaluates how well the 2D clusters recover the known biological classes.
They also reproduced the original study's findings using their metrics to demonstrate the limitations of their approach.
Datasets
The study used three publicly available single-cell RNA sequencing (scRNA-seq) datasets:
- Ex Utero dataset
- MERFISH dataset
- Smart-seq dataset
To address potential biases from using pre-defined cluster labels, they also generated a simulated dataset using negative binomial sampling based on the Ex Utero dataset, creating a scenario with known ground truth classes.
Key Insights and Ideas
- Distance preservation is not the only, or even the most important, metric for biological embeddings: While t-SNE and UMAP distort high-dimensional distances, this doesn't render them useless. The original study's focus on distance preservation led to a misinterpretation of their utility.
- t-SNE and UMAP excel at preserving local neighborhoods and class structure: The new metrics demonstrate that these methods are highly effective at ensuring that cells of the same type are clustered together in the 2D space, and that these clusters are visually distinct.
- The "elephant" metaphor is misleading: The arbitrary elephant embedding performed poorly on biologically relevant metrics, highlighting that t-SNE and UMAP capture meaningful structure, unlike a truly arbitrary shape.
- High-dimensional data can suffer from the "curse of dimensionality": In some cases, the 2D embeddings (t-SNE and UMAP) actually performed better than the original high-dimensional space on certain metrics like kNN accuracy, suggesting that dimensionality reduction can sometimes improve interpretability by mitigating the effects of too many dimensions.
- 2D embeddings are valuable for hypothesis generation: While not suitable for precise quantitative downstream analysis, t-SNE and UMAP are excellent tools for exploratory analysis, data quality control, and generating hypotheses about biological relationships.
Key Results
- kNN accuracy: t-SNE and UMAP consistently achieved >90% kNN accuracy across all datasets, meaning 2D neighbors are highly likely to be from the same cell type. This was significantly better than 2D PCA (<62%) and the Picasso elephant.
- Silhouette coefficient and AMI: t-SNE and UMAP showed significantly higher silhouette scores and AMI values compared to 2D PCA and the Picasso elephant, indicating better cluster separation and recovery of class structure.
- kNN recall: While kNN recall was lower for all methods (below 40%), this metric was less informative for biological interpretation than kNN accuracy.
- Comparison to high-dimensional space: t-SNE and UMAP often showed similar or higher kNN accuracy and silhouette coefficients than the original high-dimensional gene space, suggesting that preserving exact high-dimensional distances might not always be desirable for visualization.
Limitations and Future Work
- Limitations of class labels: The authors acknowledge that the class labels used for evaluation were derived from clustering in original publications and may not perfectly represent biological ground truth. They addressed this by using a simulated dataset with known ground truth.
- Focus on exploration, not quantification: The authors agree with Chari and Pachter that 2D embeddings should not be used for quantitative downstream analysis. Their endorsement is specifically for exploratory purposes.
- Future work: The paper implicitly suggests that further research could focus on developing even more robust metrics for evaluating embeddings that capture a wider range of biological features, and on refining methods for translating insights from 2D back to high-dimensional data for validation.
Summary updated