Assessing feature scorer results on high-dimensional datasets with t-SNE

Bruno Iochins Grisci, Mario Inostroza-Ponta, Márcio DornarXiv 2026 · DOI · arXiv

TL;DR

This paper introduces weighted t-SNE (wt-SNE), a visualization technique that integrates feature importance scores directly into the dimensionality reduction process. By scaling input dimensions based on their relevance before embedding, wt-SNE allows researchers to visually compare different feature scorers and gain intuitive insights into how machine learning models organize high-dimensional data.

What Problem Does This Paper Address?

In the world of high-dimensional data (like genomics or astrophysics), we often use "feature scorers"—algorithms that rank the importance of different variables. However, evaluating these scorers is notoriously difficult. There is rarely a "ground truth" for feature importance, and researchers typically rely on indirect metrics like downstream classification accuracy.

The problem is that two models can achieve the same accuracy while relying on completely different sets of features. Standard visualization tools like t-SNE or UMAP treat all features as equally important, which can mask the specific patterns a model has learned. There is a clear gap for a tool that allows for the direct, visual inspection of how different scoring strategies (e.g., Mutual Information vs. Random Forest importance) actually structure the data.

Methods

The authors propose a simple yet effective modification to the standard t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm.

Standard t-SNE calculates similarities between points based on Euclidean distance across all dimensions. Weighted t-SNE modifies this by incorporating a weight vector $\omega$ (the feature importance scores) into the distance calculation:

$$d_w(\phi, \psi, \omega) = \sqrt{\sum_{i=1}^N (\omega[i](\psi[i] - \phi[i]))^2}$$

In practice, this is equivalent to performing a Hadamard product (element-wise multiplication) between the dataset and the importance weights before running the t-SNE optimization. Features with a weight near zero effectively disappear, while those with a weight of one retain their full influence on the final 2D projection.

To quantify the quality of these visualizations, the authors use the Silhouette Coefficient (SC) to measure how well the importance scores separate known classes in the 2D space, and Trustworthiness to ensure the projection faithfully represents the high-dimensional relationships.

Datasets

The study evaluates wt-SNE across a diverse range of numerical and tabular datasets:

  • Synthetic: An XOR-based classification task (highly non-linear), a high-dimensional classification set (5000 features), and a sparse regression dataset.
  • Biological Microarray: Liver and prostate cancer datasets characterized by low sample sizes but tens of thousands of features.
  • Single-cell RNA-seq: A mouse cortex dataset with over 23,000 cells and 45,000 features across 23 distinct cell types.

Key Insights and Ideas

  • Visualizing "Importance" vs. "Selection": Unlike feature selection, which simply discards data, wt-SNE preserves the relative impact of every feature. This reveals "hidden" structures; for instance, in the XOR dataset, a good scorer reveals four distinct clusters (representing the four logic states) even when there are only two output classes.
  • Interpretability of the "Black Box": wt-SNE can be used to peek inside complex models. By extracting importance scores from a neural network at different training epochs, researchers can literally watch the model "learn" to disentangle classes over time.
  • Identifying Model Bias: The authors demonstrate that two Random Forests with identical F1-scores can produce wildly different wt-SNE plots. One might preserve the natural sub-clusters of the data, while the other might strictly (and perhaps erroneously) collapse the data into binary labels, helping users choose the more robust model.
  • Outlier Detection: In the liver cancer dataset, wt-SNE helped identify a specific sample that appeared as an outlier only after feature weighting, suggesting it may have been mislabeled or contaminated—a detail invisible in standard t-SNE.

Key Results

The experiments compared nine different feature scorers, ranging from simple statistical filters (Kruskal-Wallis) to embedded model-based scorers (Lasso, SVM, Neural Networks).

  • Correlation with Performance: There was a strong correlation between the visual clarity of the wt-SNE clusters (measured by the Silhouette Coefficient) and the downstream F1-score of a k-Nearest Neighbors classifier.
  • Neural Network Superiority: For non-linear tasks like the XOR dataset, Neural Networks and Random Forests provided the most meaningful visualizations, whereas linear methods like SVM failed to separate the data.
  • Regression Visualization: wt-SNE successfully visualized continuous target variables. As shown in the figure below, the weighted projection (right) clearly organizes samples along a color gradient representing the target value, whereas the unweighted version (left) is a disorganized jumble.
Fig. 1. Comparison between the visualization using weighted t-SNE of a synthetic regression dataset before and after considering the feature importance as learned by a neural network. Each figure is the visualization of the 2D embedding of a regression dataset (negative target values in blue, positive values in red) composed mostly of irrelevant features. On the left is the regular visualization, in which the samples are mixed. On the right, it is possible to visualize how the feature importance learned by a neural network disentangles the samples. This experiment is further detailed in Section 4. Interactive versions of all the plots in this publication are available at https://sbcblab.github.io/wtsne/.

Limitations and Future Work

While wt-SNE is a powerful inspection tool, the authors note several areas for further development:

  • Dependency on Scores: The quality of the visualization is entirely dependent on the quality of the precomputed feature scores. If the scorer is poor, the visualization will be misleading.
  • Supervised Nature: Currently, wt-SNE relies on labeled data to generate importance scores. Future work could explore integrating unsupervised feature scoring techniques.
  • Data Types: The current implementation is optimized for tabular/numerical data. Adapting wt-SNE for unstructured data like images or text (perhaps by weighting deep learning feature maps) remains an open challenge.
  • Hyperparameters: Like standard t-SNE, wt-SNE remains sensitive to hyperparameters like perplexity and learning rate.

Summary updated

Abstract

While vast literature on high-dimensional data visualization is available, there are not many works regarding the visualization of feature scorers and their results. Feature scorers are algorithms that assign numerical importance to each feature of multi-dimensional datasets. These importance scores can be used in several applications, such as feature selection, knowledge discovery, and machine learning interpretability. There are several feature scorers to choose from, and often no single metric or ground truth is available to guarantee the quality of their results. In this scenario, visualization can become valuable to support the decision of which method to choose and how good its results are. For this goal, this work presents "weighted t-SNE." It modifies the relationship between data points in the embedded 2D space to reflect the importance of each dimension of the original datasets as assessed by a feature scorer. This research discusses how to implement weighted t-SNE, proposes the silhouette coefficient as a numerical evaluation of the results, and shows several examples of its use in practice. Synthetic and real-world tabular datasets are used in the experiments together with nine feature scorers, ranging from Mutual Information to neural networks. Each feature scorer produces unique visualizations, and weighted t-SNE can be used to compare and choose the one that better suits a given dataset and task. Weighted t-SNE can also visually show the importance of features learned by machine learning models and help us see how they are organizing the data, increasing their interpretability.
BibTeX
@misc{grisci2026assessing,
  author = {Bruno Iochins Grisci and Mario Inostroza-Ponta and Márcio Dorn},
  title = {{Assessing feature scorer results on high-dimensional datasets with t-SNE}},
  year = {2026},
  eprint = {2610.39009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39009},
}
Figure