Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling

Aaron Mueller, Mark Dredze

Abstract

Neural Topic Models (NTMs) have recently leveraged pre-trained transformer encoders to generate document representations. However, these models often rely on frozen embeddings, which can lead to suboptimal topic coherence and poor cross-lingual transfer. This paper investigates fine-tuning strategies for multilingual encoders, demonstrating that auxiliary supervised tasks—particularly Natural Language Inference (NLI)—significantly improve both monolingual coherence and zero-shot alignment across languages. We identify "vertical striping" as a key failure mode in frozen models and show how fine-tuning enables more robust semantic clustering.

Introduction

Topic modeling is a fundamental unsupervised task for discovering latent themes in text. Modern Contextualized Topic Models (CTMs) improve upon traditional bag-of-words methods by using embeddings from pre-trained models like BERT or XLM-R. Despite their success, using frozen embeddings often results in representations that are not optimized for the specific semantic clusters of a target corpus. This leads to two primary challenges: suboptimal coherence in the discovered topics and a failure to align topics correctly in zero-shot cross-lingual settings, where foreign-language documents are often misassigned to a small subset of generic topics.

Methodology

The authors explore four strategies to introduce supervision into the encoder:

  1. Fine-tuning on Related Tasks: Utilizing supervised datasets like MLDoc (document classification) or NLI to refine the encoder's semantic space.
  2. Pseudo-label Fine-tuning: Using traditional LDA to generate topic labels for a corpus and fine-tuning the encoder to predict these labels.
  3. Continued Pre-training (CPT): Adapting the encoder to a specific domain through additional Masked Language Modeling (MLM).
  4. Topic Classification Contextualized Topic Model (TCCTM): A novel architecture that incorporates a classification head to simultaneously reconstruct the bag-of-words and predict auxiliary topic labels.

Experimental Setup

Evaluations were conducted using the following datasets:

  • Wikipedia: 100,000 English articles for training, with aligned test sets in French, German, Portuguese, and Dutch for zero-shot evaluation.
  • CORD-19: COVID-19 academic articles for out-of-domain testing.
  • Auxiliary Tasks: MLDoc for news classification and SNLI/MultiNLI for inference tasks.

Analysis of Embedding Alignment

A critical finding is the "vertical striping" phenomenon observed in frozen models. Qualitative analysis of confusion matrices shows that without fine-tuning, the model tends to ignore the specific content of foreign documents, collapsing them into a few "garbage" topics. Fine-tuning the encoder effectively "spreads out" these representations, allowing the model to utilize the full topic space across different languages.

Confusion matrices showing the 'vertical striping' phenomenon in frozen models (left) vs. improved alignment after fine-tuning (right).

Interestingly, performance on Semantic Textual Similarity (STS) benchmarks was found to be a strong predictor of cross-lingual transfer ability ($ \rho > 0.93 $), though it did not correlate with topic coherence (NPMI). This suggests that while good similarity embeddings facilitate transfer, they do not inherently guarantee the discovery of distinct, coherent topics.

Key Results

Fine-tuning consistently outperformed frozen baselines across all metrics. In monolingual settings on Wikipedia, fine-tuned models achieved an NPMI of 0.160, surpassing the 0.144 of frozen CTMs and 0.129 of traditional LDA.

The impact on cross-lingual alignment was even more pronounced. The "Match" metric, which measures how often translated document pairs are assigned the same topic, increased from 33.3% for frozen models to 55.3% with NLI fine-tuning.

Cross-lingual performance metrics (Match and KL divergence) across different languages and fine-tuning strategies.

XLM-RoBERTa (XLM-R) generally outperformed multilingual BERT (mBERT), likely due to its more extensive pre-training and better-aligned cross-lingual space. Surprisingly, NLI fine-tuning—a task seemingly unrelated to topic modeling—proved to be the most effective strategy for cross-lingual transfer, suggesting that high-level relational understanding is vital for aligning topic spaces.

Conclusion

This work highlights fine-tuning as a crucial step for neural topic modeling, particularly for cross-lingual applications. While NLI emerged as a powerful auxiliary task, future research is needed to understand why entailment tasks align topic spaces so effectively. Additionally, extending the framework to support simultaneous multi-way transfer and parallel topic generation remains an open area for exploration.

Summary updated

BibTeX
@inproceedings{mueller2021fine,
  author = {Aaron Mueller and Mark Dredze},
  title = {{Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=aBdGTThk_7V},
}
Figure