Discrete Cosine Transform as Universal Sentence Encoder

Nada AlMarwani, Mona T. Diab

Abstract

Modern sentence encoders are used to generate dense vector representations that capture the underlying linguistic characteristics for a sequence of words, including phrases, sentences, or paragraphs. These kinds of representations are ideal for training a classifier for an end task such as sentiment analysis, question answering and text classification. Different models have been proposed to efficiently generate general purpose sentence representations to be used in pretraining protocols. While averaging is the most commonly used efficient sentence encoder, Discrete Cosine Transform (DCT) was recently proposed as an alternative that captures the underlying syntactic characteristics of a given text without compromising practical efficiency compared to averaging. However, as with most other sentence encoders, the DCT sentence encoder was only evaluated in English. To this end, we utilize DCT encoder to generate universal sentence representation for different languages such as German, French, Spanish and Russian. The experimental results clearly show the superior effectiveness of DCT encoding in which consistent performance improvements are achieved over strong baselines on multiple standardized datasets.

Introduction

Modern Natural Language Processing (NLP) relies heavily on dense vector representations of sentences for downstream tasks such as sentiment analysis, question answering, and text classification. While heavily parameterized models like BERT or RoBERTa provide high-quality embeddings, training them for every language is computationally expensive and environmentally taxing. A common, efficient alternative is vector averaging (AVG) of pre-trained word embeddings; however, this method discards crucial structural and syntactic information like word order.

This paper explores the use of the Discrete Cosine Transform (DCT) as a lightweight, non-parameterized sentence encoder. Originally proposed for English, the authors investigate whether DCT can serve as a universal, language-independent encoder that preserves structural data across diverse linguistic typologies without the overhead of neural network training.

Methodology: Text as a Signal

In signal processing, the Discrete Cosine Transform (DCT) decomposes a signal into its component frequencies. This research treats a sentence—a sequence of $d$-dimensional word embeddings—as a multi-dimensional signal over time. By applying a Type II DCT along the feature dimensions, the sequence is compressed into a set of coefficients:

  • $c[0]$ (The first coefficient): Represents the normalized sum of the sequence, which is mathematically proportional to simple vector averaging.
  • Lower-order coefficients ($c[1], c[2]$, etc.): Capture lower signal frequencies corresponding to overall structural patterns and word order.

By concatenating the first $K$ coefficients (e.g., $c[0:2]$), the authors generate fixed-length sentence vectors that retain significantly more information than simple averaging. As shown in the heatmaps below, adding more coefficients consistently improves the model's ability to capture complex linguistic features.

Extracted figure from page 3 of Discrete Cosine Transform as Universal Sentence Encoder

Experimental Setup

The DCT encoder was evaluated across two primary benchmarks:

  1. Multilingual Probing: Using the SentEval framework, the authors tested 9 probing tasks (e.g., Sentence Length, Tree Depth, Tense) across English, French, German, Spanish, and Russian.
  2. Cross-lingual Sentence Retrieval: Parallel datasets from WMT'13 and WMT'17 were used to align vector spaces and evaluate retrieval accuracy across language pairs like German-English and Latvian-English.

The experiments utilized various embedding types, including pre-trained FastText and multilingual BERT (mBERT).

Key Results and Insights

The experimental results demonstrate that treating text as a signal is a universally effective approach. The $c[0]$ DCT model alone outperformed standard vector averaging (AVG) by approximately 30 percentage points on surface-level tasks across all tested languages. As more coefficients ($K$) were added, performance on complex syntactic and semantic tasks consistently improved.

A major highlight of the study is the performance achieved when combining DCT with contextualized embeddings. By applying DCT to mBERT embeddings (using just $c[0:1]$), the authors achieved a state-of-the-art average accuracy of 91.83% across all translation directions, significantly outperforming previous baselines like ELMo.

Extracted table from page 5 of Discrete Cosine Transform as Universal Sentence Encoder

Furthermore, the study found that sentence-level mapping via DCT provides better disambiguation for cross-lingual alignment than traditional word-level dictionary methods. However, language typology does play a role; morphologically rich languages with flexible word order, such as Russian, showed more performance variance compared to more rigid languages like English.

Conclusion

The Discrete Cosine Transform offers a powerful, "ready-to-use" alternative to parameterized sentence encoders. By capturing structural information through frequency decomposition, it enables high-quality cross-lingual knowledge transfer with minimal computational cost. This approach is particularly promising for low-resource languages, where the massive compute overhead of training large transformer models is often prohibitive.

Summary updated

Abstract

Modern sentence encoders are used to generate dense vector representations that capture the underlying linguistic characteristics for a sequence of words, including phrases, sentences, or paragraphs. These kinds of representations are ideal for training a classifier for an end task such as sentiment analysis, question answering and text classification. Different models have been proposed to efficiently generate general purpose sentence representations to be used in pretraining protocols. While averaging is the most commonly used efficient sentence encoder, Discrete Cosine Transform (DCT) was recently proposed as an alternative that captures the underlying syntactic characteristics of a given text without compromising practical efficiency compared to averaging. However, as with most other sentence encoders, the DCT sentence encoder was only evaluated in English. To this end, we utilize DCT encoder to generate universal sentence representation for different languages such as German, French, Spanish and Russian. The experimental results clearly show the superior effectiveness of DCT encoding in which consistent performance improvements are achieved over strong baselines on multiple standardized datasets.
BibTeX
@inproceedings{almarwani2021discrete,
  author = {Nada AlMarwani and Mona T. Diab},
  title = {{Discrete Cosine Transform as Universal Sentence Encoder}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=xZOH2I0ubN},
}
Figure