Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity

Ivan Rep, David Dukic, Jan Snajder

Abstract: While BERT produces high-quality sentence embeddings, its pre-training computational cost is a significant drawback. In contrast, ELECTRA delivers a cost-effective pre-training objective and downstream task performance improvements, but not as performant sentence embeddings. The community tacitly stopped utilizing ELECTRA's sentence embeddings for semantic textual similarity (STS). We notice a significant drop in performance when using the ELECTRA discriminator's last layer in comparison to earlier layers. We explore this drop and devise a way to repair ELECTRA's embeddings, proposing a novel truncated model fine-tuning (TMFT) method. TMFT improves the Spearman correlation coefficient by over 8 points while increasing parameter efficiency on the STS benchmark dataset. We extend our analysis to various model sizes and languages. Further, we discover the surprising efficacy of ELECTRA's generator model, which performs on par with BERT, using significantly fewer parameters and a substantially smaller embedding size. Finally, we observe further boosts by combining TMFT with a word similarity task or domain adaptive pre-training.

Introduction

In the landscape of Natural Language Processing, BERT remains the gold standard for generating high-quality sentence embeddings. However, its pre-training computational cost is a significant hurdle. ELECTRA emerged as a more efficient alternative, utilizing a "Discriminator" to detect replaced tokens rather than predicting masked ones. Despite its efficiency and strong performance on token-level tasks, the NLP community observed a significant deficiency: ELECTRA’s sentence embeddings performed poorly on Semantic Textual Similarity (STS) tasks. Consequently, researchers largely abandoned ELECTRA for tasks requiring robust sentence-level representations. This study investigates the root cause of this failure and proposes a method to "repair" these embeddings without sacrificing efficiency.

Methodology: Truncated Model Fine-Tuning (TMFT)

The authors hypothesize that the performance drop in ELECTRA's sentence embeddings is due to the final layers becoming over-specialized for the pre-training task (Replaced Token Detection), thereby losing general semantic information. To address this, they introduce Truncated Model Fine-Tuning (TMFT).

Overview of the Truncated Model Fine-Tuning (TMFT) architecture compared to standard pre-training.

Unlike standard fine-tuning, which utilizes the output of the final layer ($L$), TMFT extracts the output from an earlier, more semantically rich layer ($l$). This output undergoes mean pooling, and the model is fine-tuned only up to that specific layer. Additionally, the researchers evaluated the ELECTRA Generator—a smaller model typically discarded after pre-training—and explored hybrid enhancements by combining TMFT with word similarity tasks or Domain Adaptive Pre-training (DAPT) using a standard Masked Language Modeling (MLM) objective.

Key Insights: The Specialization Drop

A critical discovery of this research is the "Specialization Drop." Using Centered Kernel Alignment (CKA) analysis, the authors demonstrated that the ELECTRA discriminator's final layers diverge significantly from BERT-like representations. These layers become hyper-focused on detecting "fake" tokens, which degrades their ability to capture the broader semantic relationships necessary for sentence similarity.

Performance of various models across layers under different training regimes (STSB, word similarity, and MLM).

As shown in Figure 5, the performance of the ELECTRA discriminator (and notably, DeBERTa-V3) peaks at intermediate layers and drops sharply toward the final layer. This confirms that "less is more" for these architectures; for instance, the 3rd layer of the ELECTRA discriminator often outperforms the 12th. Truncating the model not only improves accuracy but also enhances inference efficiency by reducing the number of active parameters.

The Hidden Gem: ELECTRA Generator

The study also reveals that the ELECTRA Generator is a highly effective, parameter-efficient model for sentence representations. Because it is trained to predict plausible replacement tokens (a task similar to BERT's MLM), it retains high-quality semantic information. The ELECTRA-base generator, with only 33M parameters, performs on par with the 108M-parameter BERT-base, offering a 3x reduction in size while producing more compact embeddings.

Key Results

The application of TMFT yields substantial improvements across several benchmarks:

  • Massive Gains: Applying TMFT to the ELECTRA discriminator improved the Spearman correlation on the STS Benchmark by over 8 points.
  • Efficiency King: The ELECTRA-base generator performed on par with BERT-base while being roughly 3x smaller and producing smaller embeddings.
  • Discriminator Recovery: By truncating the ELECTRA-large discriminator to its best-performing layer, the authors saw a staggering jump in Spearman correlation—from a "broken" 25.84 to a competitive 80.90.
  • Hybrid Success: Combining TMFT with word similarity fine-tuning or MLM-based domain adaptation allowed ELECTRA to outperform BERT on several layers of the STSB task.

Conclusion and Future Work

This research demonstrates that ELECTRA's sentence embeddings are not "broken" but rather hidden in its intermediate layers. By employing TMFT, researchers can leverage ELECTRA's efficiency for sentence-level tasks. Future work should expand this analysis to broader information retrieval tasks and investigate if other discriminator-style models, such as the wider DeBERTa family, exhibit similar final-layer degradation. Further hyperparameter optimization could potentially unlock even greater performance from these truncated models.

Summary updated

Abstract

While BERT produces high-quality sentence embeddings, its pre-training computational cost is a significant drawback. In contrast, ELECTRA delivers a cost-effective pre-training objective and downstream task performance improvements, but not as performant sentence embeddings. The community tacitly stopped utilizing ELECTRA's sentence embeddings for semantic textual similarity (STS). We notice a significant drop in performance when using the ELECTRA discriminator's last layer in comparison to earlier layers. We explore this drop and devise a way to repair ELECTRA's embeddings, proposing a novel truncated model fine-tuning (TMFT) method. TMFT improves the Spearman correlation coefficient by over 8 points while increasing parameter efficiency on the STS benchmark dataset. We extend our analysis to various model sizes and languages. Further, we discover the surprising efficacy of ELECTRA's generator model, which performs on par with BERT, using significantly fewer parameters and a substantially smaller embedding size. Finally, we observe further boosts by combining TMFT with a word similarity task or domain adaptive pre-training.
BibTeX
@inproceedings{rep2024are,
  author = {Ivan Rep and David Dukic and Jan Snajder},
  title = {{Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=QLB9oM9qa9},
}
Figure