TL;DR
This paper introduces weighted t-SNE (wt-SNE), a visualization technique that integrates feature importance scores directly into the dimensionality reduction process. By scaling input dimensions based on their relevance before embedding, wt-SNE allows researchers to visually compare different feature scorers and gain intuitive insights into how machine learning models organize high-dimensional data.
What Problem Does This Paper Address?
In the world of high-dimensional data (like genomics or astrophysics), we often use "feature scorers"—algorithms that rank the importance of different variables. However, evaluating these scorers is notoriously difficult. There is rarely a "ground truth" for feature importance, and researchers typically rely on indirect metrics like downstream classification accuracy.
The problem is that two models can achieve the same accuracy while relying on completely different sets of features. Standard visualization tools like t-SNE or UMAP treat all features as equally important, which can mask the specific patterns a model has learned. There is a clear gap for a tool that allows for the direct, visual inspection of how different scoring strategies (e.g., Mutual Information vs. Random Forest importance) actually structure the data.
Methods
The authors propose a simple yet effective modification to the standard t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm.
Standard t-SNE calculates similarities between points based on Euclidean distance across all dimensions. Weighted t-SNE modifies this by incorporating a weight vector $\omega$ (the feature importance scores) into the distance calculation:
$$d_w(\phi, \psi, \omega) = \sqrt{\sum_{i=1}^N (\omega[i](\psi[i] - \phi[i]))^2}$$
In practice, this is equivalent to performing a Hadamard product (element-wise multiplication) between the dataset and the importance weights before running the t-SNE optimization. Features with a weight near zero effectively disappear, while those with a weight of one retain their full influence on the final 2D projection.
To quantify the quality of these visualizations, the authors use the Silhouette Coefficient (SC) to measure how well the importance scores separate known classes in the 2D space, and Trustworthiness to ensure the projection faithfully represents the high-dimensional relationships.
Datasets
The study evaluates wt-SNE across a diverse range of numerical and tabular datasets:
- Synthetic: An XOR-based classification task (highly non-linear), a high-dimensional classification set (5000 features), and a sparse regression dataset.
- Biological Microarray: Liver and prostate cancer datasets characterized by low sample sizes but tens of thousands of features.
- Single-cell RNA-seq: A mouse cortex dataset with over 23,000 cells and 45,000 features across 23 distinct cell types.
Key Insights and Ideas
- Visualizing "Importance" vs. "Selection": Unlike feature selection, which simply discards data, wt-SNE preserves the relative impact of every feature. This reveals "hidden" structures; for instance, in the XOR dataset, a good scorer reveals four distinct clusters (representing the four logic states) even when there are only two output classes.
- Interpretability of the "Black Box": wt-SNE can be used to peek inside complex models. By extracting importance scores from a neural network at different training epochs, researchers can literally watch the model "learn" to disentangle classes over time.
- Identifying Model Bias: The authors demonstrate that two Random Forests with identical F1-scores can produce wildly different wt-SNE plots. One might preserve the natural sub-clusters of the data, while the other might strictly (and perhaps erroneously) collapse the data into binary labels, helping users choose the more robust model.
- Outlier Detection: In the liver cancer dataset, wt-SNE helped identify a specific sample that appeared as an outlier only after feature weighting, suggesting it may have been mislabeled or contaminated—a detail invisible in standard t-SNE.
Key Results
The experiments compared nine different feature scorers, ranging from simple statistical filters (Kruskal-Wallis) to embedded model-based scorers (Lasso, SVM, Neural Networks).
- Correlation with Performance: There was a strong correlation between the visual clarity of the wt-SNE clusters (measured by the Silhouette Coefficient) and the downstream F1-score of a k-Nearest Neighbors classifier.
- Neural Network Superiority: For non-linear tasks like the XOR dataset, Neural Networks and Random Forests provided the most meaningful visualizations, whereas linear methods like SVM failed to separate the data.
- Regression Visualization: wt-SNE successfully visualized continuous target variables. As shown in the figure below, the weighted projection (right) clearly organizes samples along a color gradient representing the target value, whereas the unweighted version (left) is a disorganized jumble.
Limitations and Future Work
While wt-SNE is a powerful inspection tool, the authors note several areas for further development:
- Dependency on Scores: The quality of the visualization is entirely dependent on the quality of the precomputed feature scores. If the scorer is poor, the visualization will be misleading.
- Supervised Nature: Currently, wt-SNE relies on labeled data to generate importance scores. Future work could explore integrating unsupervised feature scoring techniques.
- Data Types: The current implementation is optimized for tabular/numerical data. Adapting wt-SNE for unstructured data like images or text (perhaps by weighting deep learning feature maps) remains an open challenge.
- Hyperparameters: Like standard t-SNE, wt-SNE remains sensitive to hyperparameters like perplexity and learning rate.
Summary updated