Normalized Space Alignment: A Versatile Metric for Representation Space Discrepancy Minimization

Danish Ebadulla, Ambuj SinghOpenReview 2024 · Submission to ICLR 2024 · Rejected · OpenReview

Abstract

We introduce a manifold analysis technique for quantifying the discrepancy between two representation spaces. Normalized Space Alignment (NSA) aims to compare pairwise distance between two point clouds. Our technique provides a robust means of comparing representations across different layers and models, with a particular focus on Graph Neural Networks (GNNs) to explore their unique capabilities. We show that our technique acts as a pseudometric, satisfies the properties of a similarity metric, is continuous and differentiable. We also demonstrate that NSA can serve as an effective loss function by utilizing it in autoencoders to preserve representation structure for dimensionality reduction. Furthermore, our empirical analysis showcases that NSA consistently outperforms or matches the results of previous techniques while offering computational efficiency. Its versatility extends to robustness analysis and various neural network training and representation learning applications, highlighting its wide applicability and potential to enhance the performance of neural networks.
BibTeX
@misc{ebadulla2024normalized,
  author = {Danish Ebadulla and Ambuj Singh},
  title = {{Normalized Space Alignment: A Versatile Metric for Representation Space Discrepancy Minimization}},
  year = {2024},
  url = {https://openreview.net/forum?id=5HGPR6fg2S},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}
Figure