On Geodesic Distances and Contextual Embedding Compression for Text Classification

Rishi Jha, Kai Mihata

Abstract

In some memory-constrained settings like IoT devices and over-the-network data pipelines, it can be advantageous to have smaller contextual embeddings. We investigate the efficacy of projecting contextual embedding data (BERT) onto a manifold, and using nonlinear dimensionality reduction techniques to compress these embeddings. In particular, we propose a novel post-processing approach, applying a combination of Isomap and PCA. We find that the geodesic distance estimations, estimates of the shortest path on a Riemannian manifold, from Isomap's k-Nearest Neighbors graph bolstered the performance of the compressed embeddings to be comparable to the original BERT embeddings. On one dataset, we find that despite a 12-fold dimensionality reduction, the compressed embeddings performed within 0.1% of the original BERT embeddings on a downstream classification task. In addition, we find that this approach works particularly well on tasks reliant on syntactic data, when compared with linear dimensionality reduction. These results show promise for a novel geometric approach to achieve lower dimensional text embeddings from existing transformers and pave the way for data-specific and application-specific embedding compressions.

Introduction

Contextual embeddings, such as those generated by BERT, have set new benchmarks in Natural Language Processing. However, these embeddings typically reside in high-dimensional spaces (e.g., 768 dimensions for BERT-base), imposing a significant "memory tax" on resource-constrained environments like IoT devices, mobile platforms, and high-throughput data pipelines. While model-level compression techniques like distillation exist, they often require intensive retraining. This work explores a post-processing alternative: compressing existing embeddings using a hybrid of linear and nonlinear dimensionality reduction to preserve the underlying geometric structure of language.

Methodology: Capturing the Manifold

The core hypothesis of this research is that language data lies on a lower-dimensional "manifold." Standard linear techniques like Principal Component Analysis (PCA) excel at capturing global variance but often fail to resolve complex, non-linear relationships. To address this, we employ Isomap, a nonlinear technique that utilizes a k-Nearest Neighbors (k-NN) graph to estimate geodesic distances—the shortest path along the manifold's surface.

Comparison of PCA and Isomap projections, demonstrating Isomap's superior ability to capture non-linear class separation.

As shown in the visualization above, Isomap provides a much clearer separation of data classes compared to PCA by respecting the local geometry of the embedding space. To leverage the strengths of both methods, we propose a Concatenated Embedding approach. By combining 16 Isomap dimensions with 48 PCA dimensions, we create a 64-dimensional vector that captures both local "curved" geometry and global variance.

Experimental Results

We evaluated our method on two distinct tasks from the GLUE benchmark: the Corpus of Linguistic Acceptability (CoLA), which focuses on syntax, and the Stanford Sentiment Treebank (SST-2), which focuses on semantics.

The results demonstrate that a 12-fold reduction in dimensionality (from 768 to 64) can be achieved with minimal impact on performance. Notably, on the SST-2 task, our concatenated embeddings achieved an accuracy of 0.846, nearly matching the original BERT performance of 0.847.

Performance comparison across different embedding compression techniques on CoLA and SST-2 benchmarks.

The table highlights that the concatenated approach significantly outperforms pure PCA at the same dimensionality, particularly on the syntactic CoLA task (0.421 vs 0.339 MCC). This suggests that the geodesic distances captured by Isomap are highly sensitive to the grammatical structures of language.

Conclusion

This study confirms that contextual embeddings possess a non-linear "shape" that can be exploited for efficient compression. By preserving geodesic distances through a hybrid PCA-Isomap approach, we can reduce embedding size by an order of magnitude while maintaining performance parity with full-scale models. Future work will focus on scaling these graph-based methods to larger datasets and exploring non-Euclidean geometries for even more compact representations.

Summary updated

Abstract

In some memory-constrained settings like IoT devices and over-the-network data pipelines, it can be advantageous to have smaller contextual embeddings. We investigate the efficacy of projecting contextual embedding data (BERT) onto a manifold, and using nonlinear dimensionality reduction techniques to compress these embeddings. In particular, we propose a novel post-processing approach, applying a combination of Isomap and PCA. We find that the geodesic distance estimations, estimates of the shortest path on a Riemannian manifold, from Isomap's k-Nearest Neighbors graph bolstered the performance of the compressed embeddings to be comparable to the original BERT embeddings. On one dataset, we find that despite a 12-fold dimensionality reduction, the compressed embeddings performed within 0.1% of the original BERT embeddings on a downstream classification task. In addition, we find that this approach works particularly well on tasks reliant on syntactic data, when compared with linear dimensionality reduction. These results show promise for a novel geometric approach to achieve lower dimensional text embeddings from existing transformers and pave the way for data-specific and application-specific embedding compressions.
BibTeX
@inproceedings{jha2021geodesic,
  author = {Rishi Jha and Kai Mihata},
  title = {{On Geodesic Distances and Contextual Embedding Compression for Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure