Graduated Non-Convexity for Robust Self-Trained Language Understanding

Hongyin Luo, Yoon Kim, James R. GlassICLR 2023 · OpenReview

Abstract

Self-training has been proved an efficient strategy for unsupervised fine-tuning of language models using unlabeled data and model-generated pseudo-labels. However, the performance of self-trained models is unstable under different settings of training and evaluation data, influenced by both data distribution and pseudo-label accuracy. In this work, we propose an outlier robust self-training method based on graduated non-convexity (GNC) to mitigate the problem. We construct self-training as a non-convex optimization problem with outlier training examples. The models are self-trained with robust cost functions based according to Black-Rangarajan Duality. The algorithm learns slack variables as the loss weights for all training samples. The slack variables are used to calibrate the loss items during training to update the model parameters. The calibrated loss items lead to more robust self-trained models against different training and evaluation data and tasks. We conducted experiments on few-shot natural language understanding tasks with labeled and unlabeled data examples. Experiment results show that the proposed loss calibration method improves the performance and stability of self-training under different training tasks and data examples, and also benefits the robustness against adversarial evaluation corpora.
BibTeX
@inproceedings{luo2023graduated,
  author = {Hongyin Luo and Yoon Kim and James R. Glass},
  title = {{Graduated Non-Convexity for Robust Self-Trained Language Understanding}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=XG_LmeoU8Xq},
}

Loading related content...

Figure