Abstract
We consider the problems of clustering, classification, and visualization of high-dimensional data when no straightforward Euclidean representation exists. Typically, these tasks are performed by first reducing the high-dimensional data to some lower dimensional Euclidean space, as many manifold learning methods have been developed for this task. In many practical problems however, the assumption of a Euclidean manifold cannot be justified. In these cases, a more appropriate assumption would be that the data lies on a statistical manifold, or a manifold of probability density functions (PDFs). In this paper we propose using the properties of information geometry in order to define similarities between data sets using the Fisher information metric. We will show this metric can be approximated using entirely non-parametric methods, as the parameterization of the manifold is generally unknown. Furthermore, by using multi-dimensional scaling methods, we are able to embed the corresponding PDFs into a low-dimensional Euclidean space. This not only allows for classification of the data, but also visualization of the manifold. As a whole, we refer to our framework as Fisher Information Non-parametric Embedding (FINE), and illustrate its uses on a variety of practical problems, including bio-medical applications and document classification.
Introduction
In modern data science, high-dimensional datasets often consist of collections of observations rather than single vectors. For instance, in flow cytometry, a single patient is represented by thousands of cell measurements. Traditionally, researchers handle such data through ad hoc feature extraction—calculating means or variances to force the data into a flat Euclidean space. This "dirty laundry" of machine learning often ignores the underlying statistical structure.
Existing manifold learning techniques like Isomap assume data lies on a Euclidean manifold. However, when data points are themselves probability distributions, they naturally reside on a statistical manifold. The FINE (Fisher Information Non-parametric Embedding) framework addresses this by providing a principled, non-parametric way to perform dimensionality reduction using information geometry.
The FINE Framework
FINE shifts the perspective from treating data as vectors to treating data as probability density functions (PDFs). The framework follows a four-step pipeline:
- Density Estimation: Estimate the PDF for each dataset using Kernel Density Estimation (KDE) for continuous data or normalized counts for discrete data.
- Local Distance Approximation: Instead of Euclidean distance, FINE uses information-theoretic divergences (like Kullback-Leibler or Hellinger) to approximate the Fisher Information Metric (FIM) between neighboring PDFs.
- Geodesic Distance Estimation: To account for the curvature of the statistical manifold, FINE constructs a neighborhood graph and calculates the shortest path (geodesic) between all pairs of PDFs.
- Embedding: Finally, Multi-Dimensional Scaling (MDS) or Laplacian Eigenmaps (LEM) project these geodesic distances into a low-dimensional Euclidean space for visualization.
As shown in the figure above, the Fisher Information Metric defines a specific geometry on the space of parameters. By approximating this metric non-parametrically, FINE captures the intrinsic relationships between distributions without requiring a predefined parametric model.
Experimental Results
Bio-medical Visualization
In flow cytometry tasks involving patients with CLL leukemia and MCL lymphoma, FINE successfully revealed a natural separation between the two groups in a 2D embedding. Traditional 2D scatter plots of raw markers often show massive overlap, making clinical differentiation difficult. FINE’s ability to treat each patient as a distribution allowed it to uncover structural differences that were previously invisible.
Document Classification
When applied to the 20 Newsgroups corpus, FINE demonstrated superior performance in low-data regimes. By reducing the dimensionality of word-count PDFs first, the framework limited variance and allowed simple classifiers to outperform complex methods like Diffusion Kernels. For example, with only 40 training samples, FINE achieved approximately 82% accuracy compared to 75% for competing methods.
Conclusion
FINE provides a robust framework for dimensionality reduction on statistical manifolds. By leveraging the Fisher Information Metric non-parametrically, it avoids the pitfalls of ad hoc feature extraction and Euclidean assumptions. Future work aims to scale density estimation for higher-dimensional individual observations and explore spherical constraints for multinomial data structures.
Summary updated