BrainStorm @ iREL at #SMM4H 2024: Leveraging Translation and Topical Embeddings for Annotation Detection in Tweets

Manav Chaudhary, Harshit Gupta, Vasudeva VarmaACL 2024

TL;DR

This paper tackles the challenge of distinguishing human-generated annotations from those made by Large Language Models (LLMs) in COVID-19 symptom tweets. They propose a method combining translated tweets, domain-specific BERT models, and topical embeddings, finding only marginal improvements but highlighting the potential of these approaches.

What Problem Does This Paper Address?

As LLMs become increasingly capable of automating tasks like data annotation, a critical question arises: how can we trust these machine-generated annotations? This paper focuses on a specific, yet crucial, aspect of this problem: identifying whether annotations for COVID-19 symptoms in Latin American Spanish tweets were made by human experts or by an LLM. This distinction is vital for ensuring the reliability of datasets used in health-related NLP applications, where errors or biases could have significant consequences. The authors note that while much research focuses on detecting machine-generated text, detecting machine-generated annotations is a newer, important frontier.

Methods

The core of the BrainStorm @ iREL approach involves several key steps:

  • Translation: Latin American Spanish tweets are translated into English using Google Translate. This allows for the application of models trained on English data and facilitates cross-lingual comparisons.
  • Domain-Specific Embeddings: Two BERT models are used:
  • dccuchile/bert-base-spanish-wwm-cased: A Spanish BERT model for processing original Spanish tweets.
  • vinai/bertweet-covid19-base-cased: A BERT model specifically pre-trained on English COVID-19 related tweets, used for the translated English tweets.
  • Topical Embeddings with BERTopic: To capture the underlying thematic structure of tweets, the BERTopic library is employed. This process extracts latent topics from the text, and the embeddings of these topic labels are appended to the tweet's tokenized embeddings.
  • Feature Concatenation: The topical embeddings are concatenated with the pooled output of the BERT models. This combined, "feature-rich" embedding is then fed into a classification layer to predict whether the annotation was human or LLM-generated.

The underlying hypothesis is that human annotations exhibit a more consistent and coherent topical structure than LLM annotations.

Datasets

The study uses a dataset consisting of tweets related to COVID-19 symptoms written in Latin American Spanish. Each tweet is labeled as either "human" or "machine" (specifically, annotated by GPT-4). The paper does not introduce a new benchmark dataset but rather utilizes data from the SMM4H-2024 Shared Task 7.

Key Insights and Ideas

  • Topical Coherence as a Differentiator: The central idea is that human annotators, by virtue of their domain expertise and understanding of context, imbue their annotations with a distinct topical coherence that LLMs might struggle to replicate perfectly.
  • Translation as a Bridge: Translating tweets to English opens up the possibility of leveraging powerful, pre-trained English models (like BERTweet) for a task originally in Spanish, acting as a bridge for cross-lingual analysis.
  • Human Annotations as Gold Standard: The authors explicitly treat human annotations as the ground truth, acknowledging that LLM annotations might contain both correct and incorrect information.

Key Results

The evaluation was performed using accuracy on the test set.

  • Baseline Spanish: Achieved an accuracy score of 0.50.
  • Topical Spanish (with BERTopic): Also achieved a score of 0.50, indicating that topical embeddings did not improve performance on the original Spanish tweets.
  • Topical English (translated tweets with BERTopic): Achieved a score of 0.51, showing a marginal improvement. This suggests that the combination of translated tweets and topical embeddings holds some potential.

While the improvements are slight, they underscore the difficulty of this annotation detection task.

Limitations and Future Work

  • Marginal Improvements: The primary limitation is that the proposed method, while showing some promise, only yielded marginal improvements in accuracy. This suggests that more sophisticated techniques are needed for significant gains.
  • Ablation Studies: The authors mention that an ablation study using just the translated tweets and the BERTweet model was left for future exploration.
  • Further Exploration: Future work could involve:
  • Investigating more advanced topic modeling techniques.
  • Integrating other context-aware embeddings.
  • Conducting detailed ablation studies to isolate the impact of each component (translation, topical embeddings, different models).
  • Exploring multilingual and domain-adaptive approaches more deeply.

The authors believe that addressing these areas can lead to more reliable methods for distinguishing human and machine annotations, ultimately contributing to more trustworthy NLP systems in critical domains like healthcare.

Summary updated

BibTeX
@inproceedings{chaudhary2024brainstorm,
  author = {Manav Chaudhary and Harshit Gupta and Vasudeva Varma},
  title = {{BrainStorm @ iREL at \#SMM4H 2024: Leveraging Translation and Topical Embeddings for Annotation Detection in Tweets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure