Transformation of dense and sparse text representations

Mengyu Wang, Wenpeng Hu, Bing Liu, Jinwen Ma, Dongyan Zhao, Rui Yan, Haiqing Chen

Abstract

Modern Natural Language Processing (NLP) relies heavily on dense vector embeddings, which offer high performance but lack interpretability. Conversely, sparse representations are more transparent but often struggle to match the accuracy of dense models. This paper presents the Semantic Transformation (ST) framework, which enables bidirectional movement between dense and sparse spaces. By employing a novel activation function to induce sparsity rather than traditional $L_1$ penalties, the ST framework achieves state-of-the-art results on sentiment analysis benchmarks while providing a 2-3x speedup over recurrent architectures and offering clear, visualizable semantic components.

Introduction

In the current NLP landscape, words and sentences are typically represented as dense vectors. While these "black box" embeddings are mathematically efficient for machine learning models, they are nearly impossible for humans to interpret. A 300-dimensional vector provides little insight into why a model classifies a specific sentence as negative or positive.

Sparse representations offer a solution by mapping concepts to specific dimensions (semantemes), where most values are zero. However, creating high-quality sparse representations that maintain the performance of dense embeddings has remained a challenge. The Semantic Transformation (ST) framework addresses this by providing a robust mathematical bridge between these two domains.

The Semantic Transformation Framework

The ST framework is inspired by signal processing techniques, such as the Fourier Transform, which transition data between different domains. It consists of three primary components:

  1. Semantic Forward Transformation (SFT): This process maps dense embeddings into a higher-dimensional sparse space.
  2. Semantic Backward Transformation (SBT): This inverse operation reconstructs the original dense vector from its sparse counterpart, ensuring that no critical information is lost during the transformation.
  3. Semantic Combination in Sparse Space (SCSS): This method aggregates word-level sparse vectors into sentence-level representations, utilizing "negative semantemes" to handle linguistic nuances like negation.
The ST framework facilitates bidirectional movement between dense and sparse semantic spaces.

Inducing Sparsity via Activation

A key innovation of the ST framework is the move away from traditional $L_1$ regularization, which can lead to unstable training. Instead, the authors introduce a specialized activation function, $S(x)$, designed to naturally encourage sparsity.

As shown in the figure above, the function acts as a filter that highlights specific "peaks" ($\pm\gamma/\beta$) while suppressing noise. This approach makes the optimization process smoother and results in a representation where over 96% of the values are effectively zero.

Experimental Results

The ST model was evaluated against several benchmarks, including SNLI, MR, SST1, SST2, and TREC.

Performance and Accuracy

The ST framework outperformed LSTMs on three of the five datasets (SST1, SST2, and MR) and remained competitive on SNLI and TREC. Notably, it consistently surpassed more complex architectures like Transformers and Capsule Networks on these specific sentiment-related tasks.

Efficiency and Interpretability

Beyond accuracy, the ST model offers significant practical advantages:

  • Speed: Due to the avoidance of heavy matrix multiplications in the sparse combination phase, the model runs 2 to 3 times faster than standard LSTMs.
  • Reconstruction Fidelity: The reconstruction error between the original dense vectors and those processed through the SBT was approximately 5%, confirming the framework's ability to preserve semantic integrity.
  • Visualizable Semantics: By analyzing the learned sparse dimensions, researchers identified specific "semantic bases" that correspond to sentiment polarities, providing a level of transparency absent in traditional dense models.

Conclusion

The Semantic Transformation framework represents a significant step toward making high-performance NLP models more interpretable. By replacing traditional sparsity penalties with a tailored activation function, the authors have created a system that is both faster and more transparent than existing recurrent models. Future research will explore the application of ST to generative tasks and its potential for automated style transfer through the manipulation of specific semantic bits.

Summary updated

Abstract

Sparsity is regarded as a desirable property of representations, especially in terms of explanation. However, its usage has been limited due to the gap with dense representations. Most NLP research progresses in recent years are based on dense representations. Thus the desirable property of sparsity cannot be leveraged. Inspired by Fourier Transformation, in this paper, we propose a novel Semantic Transformation method to bridge the dense and sparse spaces, which can facilitate the NLP research to shift from dense space to sparse space or to jointly use both spaces. The key idea of the proposed approach is to use a Forward Transformation to transform dense representations to sparse representations. Then some useful operations in the sparse space can be performed over the sparse representations, and the sparse representations can be used directly to perform downstream tasks such as text classification and natural language inference. Then, a Backward Transformation can also be carried out to transform those processed sparse representations to dense representations. Experiments using classification tasks and natural language inference task show that the proposed Semantic Transformation is effective.
BibTeX
@inproceedings{wang2020transformation,
  author = {Mengyu Wang and Wenpeng Hu and Bing Liu and Jinwen Ma and Dongyan Zhao and Rui Yan and Haiqing Chen},
  title = {{Transformation of dense and sparse text representations}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.290},
}
Figure