Abstract
While BERT produces high-quality sentence embeddings, its pre-training computational cost is a significant drawback. In contrast, ELECTRA provides a cost-effective pre-training objective and downstream task performance improvements, but worse sentence embeddings. The community tacitly stopped utilizing ELECTRA’s sentence embeddings for semantic textual similarity (STS). We notice a significant drop in performance for the ELECTRA discriminator’s last layer in comparison to prior layers. We explore this drop and propose a way to repair the embeddings using a novel truncated model fine-tuning (TMFT) method. TMFT improves the Spearman correlation coefficient by over 8 points while increasing parameter efficiency on the STS Benchmark. We extend our analysis to various model sizes, languages, and two other tasks. Further, we discover the surprising efficacy of ELECTRA’s generator model, which performs on par with BERT, using significantly fewer parameters and a substantially smaller embedding size. Finally, we observe boosts by combining TMFT with word similarity or domain adaptive pre-training.
Introduction
For years, the NLP community has had a love-hate relationship with ELECTRA. While it reaches BERT-level performance on the GLUE benchmark with a fraction of the pre-training compute, its sentence embeddings—the vector representations used to compare how similar two sentences are—have been notoriously "broken." While BERT produces high-quality embeddings for Semantic Textual Similarity (STS), ELECTRA’s embeddings were so poor that researchers essentially stopped trying to use them.
A recent study presented at EMNLP 2024 identifies that ELECTRA’s poor sentence embeddings are caused by its final layers becoming "over-specialized" for its unique pre-training task. By introducing Truncated Model Fine-Tuning (TMFT) and utilizing the often-discarded "generator" model, the researchers show that ELECTRA can match or outperform BERT in sentence similarity tasks while being significantly more parameter-efficient.
The Specialization Gap in ELECTRA
In the world of Sentence Transformers (bi-encoders), we typically take a pre-trained model like BERT, pass a sentence through it, and "pool" the output of the very last layer to get a single vector. This works effectively for models trained with Masked Language Modeling (MLM).
However, ELECTRA uses a different objective: Replaced Token Detection (RTD). Instead of predicting a missing word, a "discriminator" model must decide if a word is original or was swapped out by a "generator" model. While this makes pre-training fast, it creates a "specialization gap." The final layers of the discriminator become hyper-focused on spotting "fake" tokens, a task fundamentally different from understanding holistic sentence meaning. Consequently, the standard approach of using the last layer for embeddings fails.
Methods: Truncated Model Fine-Tuning (TMFT)
The authors propose Truncated Model Fine-Tuning (TMFT) to address this specialization. The method involves three key components:
- Layer-wise Analysis: Systematically testing the embedding quality of every layer in ELECTRA revealed that performance improves through the model but "falls off a cliff" in the final 2-3 layers.
- Truncation: Instead of using the full model, it is truncated at the best-performing intermediate layer (e.g., layer 9 instead of 12), and only this truncated version is fine-tuned for the similarity task.
- Hybrid Improvements: The researchers also experimented with combining TMFT with auxiliary tasks, such as word similarity or a brief stint of "Domain Adaptive Pre-training" (DAPT) using the classic MLM objective to "remind" the model how to create general-purpose representations.
Key Insights and Results
The most striking finding is the visualization of the "Last Layer Trap." In the ELECTRA-base discriminator, the Spearman correlation on the STS Benchmark (STSB) climbs steadily until layer 9, then plummets by over 11 points by the final layer. Using Centered Kernel Alignment (CKA), the authors proved that ELECTRA’s final layers look fundamentally different from BERT’s, as the RTD task forces the model to discard general semantic information in favor of token-level "detective work."
Key results include:
- Massive Gains: Applying TMFT to the ELECTRA discriminator improved Spearman correlation by over 8 points on STSB.
- Efficiency: The best-performing truncated ELECTRA models used significantly fewer parameters than their full-sized counterparts while achieving better results.
- The Generator as a Hidden Gem: The ELECTRA-base generator has only ~33M parameters (compared to BERT-base’s 108M) but performs almost identically to BERT on similarity tasks. It does not suffer from the "last layer drop" because it was trained to predict tokens, similar to MLM.
- Cross-Linguistic Consistency: This "last layer drop" was observed across all tested languages (English, Spanish, German, Korean) and various model sizes (Small, Base, Large).
Conclusion
The study demonstrates that ELECTRA's sentence embeddings are not beyond repair. By simply stopping a few layers short of the finish line and utilizing the generator model, ELECTRA becomes a highly efficient alternative to BERT for sentence similarity tasks. This work encourages the community to reconsider discarded models and look deeper into layer-wise specializations.
Summary updated