TL;DR
Training massive neural sentence encoders for every language is computationally prohibitive and environmentally costly, yet simple word vector averaging loses crucial word order and syntax. This paper demonstrates that Discrete Cosine Transform (DCT) provides an efficient, parameter-free sentence encoder that preserves structural information across multiple languages. Concatenating low-frequency DCT coefficients consistently outperforms standard averaging on multilingual probing and achieves state-of-the-art cross-lingual translation retrieval.
What Problem Does This Paper Address?
Generating dense, fixed-length sentence representations is foundational for natural language understanding tasks like sentiment analysis, question answering, and cross-lingual transfer. However, standard solutions present a dilemma:
- Heavy parameterized models (like transformers or bilingual encoders) require substantial compute, labeled/unlabeled parallel text, and significant energy to train per language.
- Simple pooling heuristics (such as averaging pre-trained word vectors) are parameter-free and fast, but treat sentences as unordered "bags of words," discarding vital syntactic and word order dynamics.
While prior work introduced the Discrete Cosine Transform as a structure-sensitive, parameter-free encoder for English, whether DCT representations generalize universally across typologically diverse languages—and whether they support cross-lingual alignment—remained untested.
Methods
The authors adopt a signal-processing perspective: treating a sentence matrix $N \times d$ ($N$ words, $d$ embedding dimensions) as a multi-dimensional discrete signal over time.
- DCT Formulation: They apply the 1D Discrete Cosine Transform (Type II) along the sentence length ($N$) independently for each embedding dimension: $$c[K] = \sqrt{\frac{2}{N}} \sum_{n=0}^{N-1} v_n \cos\left(\frac{\pi}{N}\left(n + \frac{1}{2}\right)K\right)$$
- Sentence Representation ($c[0:K]$): The first coefficient $c[0]$ corresponds to the length-normalized sum of the word vectors (proportional to vector averaging). The subsequent lower-order coefficients ($c[1], c[2], \dots$) capture low-frequency trends and sequential transitions (e.g., word order and phrase structure). The first $K+1$ coefficients are concatenated into a single vector of size $(K+1)d$.
- Evaluation Protocols:
- Multilingual Probing: Extracted DCT sentence representations feed into a single-layer MLP classifier to probe for specific linguistic properties.
- Cross-Lingual Retrieval: Sentence vectors are mapped into a shared bilingual space via linear transformations learned from parallel sentences, evaluated on nearest-neighbor sentence translation retrieval.
- Word Embeddings Tested: Public out-of-domain FastText, in-domain FastText, and contextualized embeddings from the final layer of multilingual BERT (mBERT).
Datasets
- Multilingual Probing Benchmark (Ravishankar et al., 2019): Wikipedia-derived datasets across 5 languages (English, French, German, Spanish, Russian) targeting 9 probing tasks: surface features (
SentLen,WC), syntactic features (BShift,TreeDepth,Tense,CoordInv), and semantic features (SubjNum,ObjNum,SOMO). - Cross-Lingual Retrieval Tasks:
- WMT’13: English (EN), Spanish (ES), and German (DE) using 1M parallel sentences from Common Crawl for alignment and the WMT'13 test set for evaluation.
- WMT’17: 5 additional language pairs paired with English: German, Latvian (LV), Finnish (FI), Czech (CS), and Russian (RU), using 1M parallel training sentences each.
Key Insights and Ideas
- Frequency decomposition maps to linguistic hierarchy: The zeroth coefficient ($c[0]$) excels at surface tasks like sentence length (
SentLen) and word content (WC), where it beats mean averaging by ~30 percentage points. Meanwhile, higher-order coefficients ($c[1], c[2], \dots$) capture structural and word-order cues, driving performance gains on syntactic tasks (TreeDepth,CoordInv,BShift). - Language typology impacts structural gain: French and Spanish track English performance most closely across probing tasks. Russian exhibits higher variance and lower syntactic probing parity relative to English, likely reflecting Russian's rich morphology and flexible word order.
- Complementary to contextualized representations: Even when using contextualized embeddings from mBERT—which already encode context—applying DCT ($c[0:1]$) still improves cross-lingual retrieval over simple mean pooling, showing that frequency compression preserves valuable structural boundaries.
Key Results
- Multilingual Probing: DCT representations consistently outperform simple vector averaging (AVG) across French, German, Spanish, and Russian. While $c[0]$ alone achieves the best results on surface tasks, increasing $K$ (up to $K=4$) yields the highest scores on syntactic and semantic probing tasks.
- Cross-Lingual Retrieval (Out-of-Domain FastText): DCT models substantially outshine AVG across all language pairs:
- For Latvian-to-English (
LV→EN), $c[0:3]$ reaches 60.16% accuracy compared to 21.26% for AVG (a ~38% absolute gain). - For Czech-to-English (
CS→EN), $c[0:3]$ improves accuracy from 41.87% (AVG) to 55.05%. - Zero-shot non-English pairs also improve:
DE→ESjumps from 43.80% (AVG) to 51.20% ($c[0:2]$). - State-of-the-Art Retrieval: In-domain FastText with $c[0:1]$ reaches 80.39% average retrieval accuracy across all translation directions (vs. 76.92% for FastText AVG). Combining mBERT with $c[0:1]$ sets a new benchmark at 91.83% average accuracy, comfortably surpassing prior ELMo-based sentence mapping (84.03%).
Limitations and Future Work
- Vector Dimensionality vs. Efficiency: Retaining $K$ coefficients multiplies the sentence embedding dimension by $K$, increasing downstream memory footprints.
- Diminishing Returns with Higher $K$: Benefits plateau around $K=2$ or $K=3$, with performance slightly decaying on certain tasks when $K \ge 4$.
- Complex Morphology: The relative performance dip on Russian syntax suggests that pure linear frequency decomposition struggles more with highly inflected, free-word-order languages.
- Downstream Scope: Evaluations were limited to probing tasks and translation retrieval; testing DCT embeddings on downstream downstream tasks (e.g., cross-lingual NLI or question answering) remains open for future research.
Summary updated