VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding

Dou Hu, Xiaolong Hou, Xiyang Du, Mengyuan Zhou, Lianxin Jiang, Yang Mo, Xiaofeng ShiEMNLP 2022 · DOI

Introduction

Pre-trained language models (PLMs) such as BERT and RoBERTa have revolutionized natural language processing. However, their performance often degrades when applied to specialized domains like medicine or finance due to "domain shift." While "continual pre-training" on domain-specific corpora is a common remedy, it typically requires vast amounts of data. In many specialized fields, data is scarce or expensive to acquire. Standard Masked Autoencoders (MAEs) perform rigid point estimation, which can lead to sparse embeddings that fail to capture the nuanced contextual meanings of tokens in low-data regimes.

The VarMAE Framework

To address these challenges, we introduce VarMAE (Variational Masked Autoencoder), a novel framework designed to improve domain adaptation through smoother latent representations. The core of VarMAE is the Context Uncertainty Learning (CUL) module, which shifts the modeling paradigm from point estimation to distribution estimation.

The VarMAE architecture for pre-training and fine-tuning.

Context Uncertainty Learning (CUL)

Instead of mapping a token's context to a single vector, the CUL module encodes it into a latent Gaussian distribution characterized by a mean ($\mu$) and variance ($\sigma$). This approach explicitly models "aleatoric uncertainty"—the inherent noise in the data. By treating masked and unmasked tokens differently, the model acknowledges that masked positions carry higher uncertainty.

The training objective combines a standard reconstruction loss with a Kullback–Leibler (KL) divergence term. This variational objective regularizes the latent space, ensuring that similar contexts are mapped to nearby regions in a continuous distribution, thereby "smoothing" the representation and preventing overfitting to limited co-occurrence patterns.

Experimental Evaluation

VarMAE was evaluated across two distinct domains:

  • Science Domain (English): Using 0.6 million abstracts, tested on tasks like citation intent (ACL-ARC, SciCite) and bio-entity recognition (JNLPBA, EBM-NLP).
  • Finance Domain (Chinese): Using 2 million sentences, tested on intent classification (OIR), topic classification (MTC), entity extraction (IEE), and search matching (PSM).

Key Results and Insights

VarMAE consistently outperformed established baselines, including RoBERTa and standard domain-adaptive pre-training (DAPT).

Performance comparison of VarMAE against baselines across Science and Finance domains.

Superior Performance and Data Efficiency

As shown in the results table, VarMAE achieved an average absolute improvement of 1.41% in the science domain and 3.09% in the finance domain. Notably, in the finance search match (PSM) task, VarMAE outperformed DAPT by over 7 points in F1 score.

A critical finding is VarMAE's robustness to data scarcity. Experiments revealed that VarMAE trained on only one-third of the available corpus could still surpass standard DAPT trained on the full dataset. This suggests that the variational smoothing effectively compensates for limited training samples.

Robustness and Optimization

Ablation studies confirmed that a 15% masking ratio remains optimal for VarMAE, mirroring the standard BERT configuration. Furthermore, the model demonstrated superior capability in multi-label disambiguation, identifying multiple relevant tags in classification tasks where baseline models often focused only on the most prominent features.

Conclusion and Future Work

VarMAE provides a powerful solution for domain adaptation, particularly when data is limited. By modeling context as a distribution rather than a point, it learns more robust and generalizable representations. While currently focused on Natural Language Understanding (NLU) tasks, future work will explore extending this variational approach to generative models and incorporating structured domain knowledge to further enhance performance on complex entities.

Summary updated

BibTeX
@inproceedings{hu2022varmae,
  author = {Dou Hu and Xiaolong Hou and Xiyang Du and Mengyuan Zhou and Lianxin Jiang and Yang Mo and Xiaofeng Shi},
  title = {{VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.468},
}
Figure