On Randomized Classification Layers and Their Implications in Natural Language Generation

Gal-Lev Shalev, Gabi Shalev, Joseph KeshetNAACL 2021

TL;DR

This paper proposes a simple yet effective method for natural language generation (NLG) tasks: randomly initializing and fixing the target word embeddings (classification layer weights) during training. This significantly reduces the number of learnable parameters without a substantial drop in performance, making models more efficient.

What Problem Does This Paper Address?

In natural language generation (NLG) tasks, like machine translation or image captioning, a neural network decoder predicts the next word in a sequence. This prediction relies on a "classification layer," which is essentially a matrix of target word embeddings. These embeddings, along with all other model parameters, are typically learned during training. The authors question the necessity of learning these target word embeddings, hypothesizing that their learned properties might be redundant and that fixing them could offer significant efficiency gains. This is particularly relevant in low-resource scenarios where parameter reduction is crucial.

Methods

The core contribution of this paper is the proposal to randomly initialize the target word embeddings and keep them fixed throughout the training process. This means that the weights of the classification layer, which map the decoder's internal representation to probabilities over the target vocabulary, are not updated during backpropagation. Instead, only the rest of the model's parameters (encoder, decoder, attention mechanisms, and input embeddings) are optimized. The authors explore this "fixed target embedding" approach across various NLG tasks.

Datasets

The paper evaluates its method on the following datasets:

  • Image Captioning: MSCOCO dataset.
  • Machine Translation:
    • Multi30k
    • IWSLT 2014
    • WMT-14

Key Insights and Ideas

  • Redundancy of Learned Target Embeddings: The most striking insight is that the specific properties learned by target word embeddings (like word frequency and semantic relationships) are surprisingly redundant for achieving high performance in NLG tasks. Random initialization seems to capture enough signal.
  • Parameter Reduction: Fixing the target embeddings drastically reduces the number of learnable parameters, especially in large vocabularies. This is a significant advantage for model efficiency and deployment.
  • Improved Diversity in Image Captioning: Interestingly, fixing target embeddings led to more diverse and less frequent word usage in image captions, suggesting that random initialization encourages the model to find more discriminative representations.
  • Magnitude Matters (for fixed embeddings): For fixed target embeddings, the authors found that smaller vector magnitudes generally lead to better performance in machine translation. This suggests a need for normalization.

Key Results

  • Performance: The proposed method achieved results comparable to or even slightly better than models with learned target embeddings across image captioning and machine translation tasks. For instance, in image captioning, the "fixed" models showed comparable BLEU and ROUGE scores. In machine translation, the fixed models often outperformed non-fixed and tied-embedding models.
  • Parameter Efficiency: Table 3 shows significant reductions in learnable parameters, ranging from 19% to 33% depending on the task and model.
  • Word Relationships: The fixed target vectors were found to correlate better with human judgments of word relationships than input vectors in 3 out of 4 tested models.
  • Vocabulary Usage: Image captioning models with fixed target embeddings showed an increased vocabulary usage rate and generated low-frequency words more often.

Limitations and Future Work

  • Initialization Sensitivity: While random initialization works well, the paper doesn't deeply explore the impact of different initialization strategies beyond drawing from a uniform distribution.
  • Magnitude Normalization: The paper notes that for fixed target embeddings in machine translation, smaller magnitudes (achieved via L2 normalization) are beneficial. Further investigation into optimal magnitude strategies could be explored.
  • Theoretical Understanding: While empirically demonstrated, a deeper theoretical understanding of why random and fixed target embeddings are sufficient could be a future direction.
  • Broader NLG Tasks: While image captioning and machine translation are covered, exploring this approach on other NLG tasks like text summarization or dialogue generation would be valuable.

Summary updated

BibTeX
@inproceedings{shalev2021randomized,
  author = {Gal-Lev Shalev and Gabi Shalev and Joseph Keshet},
  title = {{On Randomized Classification Layers and Their Implications in Natural Language Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure