NOMAD Projection

Brandon Duderstadt, Zach Nussbaum, Laurens van der MaatenarXiv 2025 · OpenReview

TL;DR

This paper introduces NOMAD Projection, a novel nonlinear dimensionality reduction technique designed for massive datasets. It achieves unprecedented scalability by enabling multi-GPU training and efficiently approximating a contrastive learning objective, making it possible to visualize datasets previously too large for existing methods.

What Problem Does This Paper Address?

The explosion in the size of datasets used in AI, particularly with the rise of generative models, has outpaced the capabilities of traditional data visualization tools like t-SNE and UMAP. These methods struggle with large datasets due to poor scaling in both computation time and memory usage. This limitation hinders AI explainability, which often relies on these visualization techniques for exploratory data analysis. The core challenge is to develop a visualization method that can handle massive datasets efficiently, even across multiple GPUs, without sacrificing the quality of the visualization.

Methods

NOMAD Projection is a new method for unstructured data visualization that tackles the scalability issue by approximating an upper bound on the InfoNC-t-SNE loss. InfoNC-t-SNE is a contrastive learning-based approach where data points are encouraged to be close if they are neighbors in a kNN graph and far apart otherwise.

The key innovations of NOMAD Projection are:

  • Distributed Training Strategy: It uses a K-Means based Approximate Nearest Neighbor (ANN) index. This allows for sharding clusters across GPUs, ensuring that a point's neighbors reside on the same device, thus avoiding inter-device communication during the computationally intensive positive spring force calculation.
  • Efficient Negative Force Calculation: NOMAD Projection minimizes a surrogate loss that approximates an upper bound of the InfoNC-t-SNE loss. This surrogate loss only requires the cluster means to be all-gathered during negative force calculation, significantly reducing communication overhead compared to methods that need to communicate individual point embeddings.
  • Theoretical Grounding: The paper provides theoretical justification, showing that NOMAD Projection approximates an upper bound on the InfoNC-t-SNE loss, connecting it to established contrastive learning principles.

Datasets

The paper evaluates NOMAD Projection on the following datasets:

  • ArXiv: High-dimensional vectors obtained using the Nomic Embed model.
  • ImageNet: High-dimensional vectors obtained using the OpenClip model.
  • PubMed: Vectorized using a custom BERT model.
  • Multilingual Wikipedia: This is a newly created benchmark dataset, representing the largest data map computed to date, showcasing the scalability of NOMAD Projection.

Key Insights and Ideas

  • Scalability through Approximation: The core insight is that by approximating the InfoNC-t-SNE loss with a surrogate that relies on cluster means rather than individual negative samples, significant computational and communication efficiencies can be achieved. This allows for parallelization across multiple GPUs.
  • Multi-GPU Advantage: NOMAD Projection can effectively leverage multiple GPUs, leading to substantial speedups and enabling the visualization of datasets that would otherwise be impossible due to GPU memory limitations.
  • Trade-off in Visualization Quality: While multi-GPU NOMAD Projection improves speed and neighborhood preservation, it might slightly decrease random triplet accuracy. The authors attribute this to the partitioning approximation, which can reduce the influence of global coherence information. This highlights an interesting trade-off between local and global structure preservation.
  • Surfacing Counterintuitive Relationships: The visualization of Multilingual Wikipedia revealed unexpected relationships within the embedding model (BGE-M3), such as a connection between frog taxonomy and Greek mythology. This demonstrates the power of NOMAD Projection in uncovering subtle or counterintuitive patterns in high-dimensional data.

Key Results

  • Speed and Performance: NOMAD Projection achieves comparable or superior neighborhood preservation and random triplet accuracy to existing methods when run for sufficient epochs.
  • PubMed Corpus: NOMAD Projection computed a visualization of the entire PubMed corpus in under 1.5 hours, a task that took OpenTSNE 8 hours. Other GPU-accelerated methods failed due to out-of-memory errors.
  • Multilingual Wikipedia: The paper successfully computed the first complete data map of Multilingual Wikipedia, a 60 million point dataset, demonstrating the method's scalability.
  • Multi-GPU Speedup: Using multiple GPUs significantly improves NOMAD Projection's speed and neighborhood preservation performance. For instance, NOMAD Projection on 2xH100 GPUs shows improved neighborhood preservation over 1xH100 configurations of itself and other methods.

Limitations and Future Work

  • Single Node Limitation: The current implementation of NOMAD Projection is designed for a single node with multiple GPUs. Adapting it for multi-node distributed systems presents a significant future challenge due to differences in bandwidth and latency.
  • Code Optimization: The authors acknowledge that the NOMAD Projection code is largely unoptimized and that significant speedups can be achieved through profiling and further optimization.
  • Broader Applications: The theoretical connection to InfoNCE suggests potential applications beyond data mapping, including more general contrastive learning and language modeling.
  • Further Characterization: More extensive characterization of NOMAD Projection's performance and behavior across various datasets and configurations is needed.

Summary updated

Abstract

The rapid adoption of generative AI has driven an explosion in the size of datasets consumed and produced by AI models. Traditional methods for unstructured data visualization, such as t-SNE and UMAP, have not kept up with the pace of dataset scaling. This presents a significant challenge for AI explainability, which relies on methods such as t-SNE and UMAP for exploratory data analysis. In this paper, we introduce Negative Or Mean Affinity Discrimination (NOMAD) Projection, the first method for unstructured data visualization via nonlinear dimensionality reduction that can run on multiple GPUs at train time. We provide theory that situates NOMAD Projection as an approximate upper bound on the InfoNC-t-SNE loss, and empirical results that demonstrate NOMAD Projection's superior performance and speed profile compared to existing state-of-the-art methods. We demonstrate the scalability of NOMAD Projection by computing the first complete data map of Multilingual Wikipedia.
BibTeX
@misc{duderstadt2025nomad,
  author = {Brandon Duderstadt and Zach Nussbaum and Laurens van der Maaten},
  title = {{NOMAD Projection}},
  year = {2025},
  eprint = {2505.15511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.15511},
}
Figure