Using neural topic models to track context shifts of words: a case study of COVID-related terms before and after the lockdown in April 2020

Olga Kellert, Md Mahmud Uz ZamanACL 2022

TL;DR

This paper introduces a novel approach using neural topic models (specifically BERTopic) to track how word meanings shift over time, particularly in response to major events like the COVID-19 pandemic. By analyzing global word associations within documents rather than just local co-occurrences, the method offers a more nuanced understanding of semantic change, especially for words whose core meaning remains but whose usage context dramatically alters.

What Problem Does This Paper Address?

Understanding how language evolves is crucial for many fields, from linguistics to social sciences. Traditional methods for tracking semantic change, like frequency analysis or n-gram analysis, often struggle to capture subtle shifts in word meaning. These methods primarily focus on local lexical and syntactic differences. While more recent approaches using contextualized word embeddings (like BERT) can capture some non-local relations, they may still miss broader contextual shifts. For instance, the word "mask" might retain its core meaning but be used in entirely new social and pragmatic contexts after a pandemic. This paper aims to address this gap by developing a method that can capture these more global, context-driven meaning changes.

Methods

The core of this paper's contribution lies in its application of BERTopic, a neural topic modeling technique. BERTopic leverages transformer-based language models (like BERT) to generate contextualized word embeddings. These embeddings are then clustered using a density-based algorithm (HDBScan) to form topics. Crucially, BERTopic uses a class-based TF-IDF (c-TF-IDF) approach to extract topic words, which considers the entire cluster as a document. This allows for the identification of words that are representative of a specific topic or context.

The authors contrast this with:

  • Traditional methods: Relative frequency analysis and n-gram analysis, which are good for detecting local changes but miss broader context.
  • Word Embeddings: While better at capturing some non-local relations, they might not fully grasp global contextual shifts.

The paper's novel contribution is using these neural topics as proxies for word contexts to track semantic change, focusing on how word distributions across topics change over time.

Datasets

The study utilizes a newly created dataset of English tweets from the social media platform Twitter. This dataset is divided into two periods:

  • Before the COVID-19 lockdown in April 2020: Data collected from October 2017 to June 2019.
  • After the COVID-19 lockdown in April 2020: Data collected from April 2020 onwards.

The data is sourced from four countries: the United States, Canada, the UK, and Australia, with a roughly equal number of tweets per country and period. The pre-processing resulted in approximately 90,000 unique words across both datasets.

Key Insights and Ideas

  • Global vs. Local Context: The paper strongly argues that understanding semantic change requires looking beyond local word co-occurrences. Neural topic models, by capturing broader thematic associations within documents, offer a more holistic view.
  • Topics as Semantic Frames: The authors propose that topics generated by BERTopic can serve as proxies for semantic frames or pragmatic contexts, allowing them to track how the "use" of a word changes.
  • COVID-19 as a Catalyst: The pandemic provided a unique opportunity to observe rapid and significant shifts in word usage, particularly for terms related to public health, social behavior, and daily life.
  • Limitations of Traditional Methods: The study effectively demonstrates how frequency and n-gram analyses, while useful, can be insufficient for capturing the nuanced semantic shifts observed during the pandemic.

Key Results

The study presents both quantitative and qualitative findings:

  • Quantitative:
    • Relative Frequency: Words like "quarantine," "covid," "coronavirus," and "stayhome" showed significant increases in relative frequency after the lockdown, as expected.
  • N-gram Analysis: For "distance," the top collocations shifted from "walking" and "summerofyes" to "social" and "safe." For "mask," collocations moved from "blackface" and "firespitter" to "face" and "covid."
    • Word Embeddings: Analysis of "distance" showed a shift in associated words from "hiking" and "film" to "distancing" and "practicing."
    • Topic Modeling (BERTopic):
      • The word "mask" appeared in significantly more topics (33) after the lockdown compared to before (3).
  • Topics related to "mask" after lockdown included preventive measures, but also everyday activities like Easter, food, and photography, indicating a broader contextualization.
  • Words like "quarantine" and "stay" appeared in topics related to suggestions for staying home and healthy living.
  • Qualitative: The analysis of topic distributions revealed how words like "mask" and "stay" became associated with a much wider range of everyday conversations and activities post-lockdown, highlighting a significant pragmatic shift.

Limitations and Future Work

The authors acknowledge several limitations:

  • Approximation of Pragmatic Aspect: The neural topic modeling approach is an "approximate" method for capturing pragmatic aspects of meaning change.
  • Data Size Dependency: The quantity of topics and their specification are influenced by the dataset size and document length.
  • Implicit Discourse Relations: The method does not fully capture implicit discourse relations between topic words (e.g., causal relationships).

Future work suggested includes:

  • Capturing Implicit Discourse: Exploring unsupervised approaches to capture discourse relations between topic words within the same topic.
  • Further Pragmatic Analysis: Developing methods to more directly capture the pragmatic aspect of meaning change, moving beyond lexical semantic change.

Summary updated

BibTeX
@inproceedings{kellert2022using,
  author = {Olga Kellert and Md Mahmud Uz Zaman},
  title = {{Using neural topic models to track context shifts of words: a case study of COVID-related terms before and after the lockdown in April 2020}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure