Context-Aware Temperature for Language Modeling
Lists
Only you see this
Abstract
Current practices to apply temperature scaling assume either a fixed, or a manually-crafted dynamically changing schedule. However, our studies indicate that the individual optimal trajectory for each class can change with the context. To this end, we propose context-aware temperature, a generalized approach to provide an individual optimal temperature trajectory over the context for each vocabulary, while allowing the temperature to be learned along with the remaining model parameters during training. Experiment results confirm that the proposed method significantly improves state-of-the-art language models, achieving a perplexity of 19.90 on Penn Treebank, 33.88 on WikiText-2, and 4.7 on WikiText-103.
BibTeX
@misc{wang2021context,
author = {Pei-Hsin Wang and Sheng-Iou Hsieh and Shih-Chieh Chang and Yu-Ting Chen and Da-Cheng Juan and Jia-Yu Pan and Wei Wei},
title = {{Context-Aware Temperature for Language Modeling}},
year = {2021},
}Loading related content...