GPT is an effective tool for multilingual psychological text analysis

Steve Rathje, Dan-Mircea Mirea, Ilia Sucholutsky, Raja Marjieh, Claire Robertson, Jay Joseph Van Bavel

TL;DR

This paper demonstrates that large language models like GPT can effectively perform psychological text analysis across multiple languages with high accuracy, often outperforming traditional dictionary-based methods and rivaling fine-tuned machine learning models. This breakthrough democratizes advanced NLP capabilities, making sophisticated text analysis more accessible for researchers, especially those working with understudied languages.

What Problem Does This Paper Address?

The social and behavioral sciences increasingly rely on automated text analysis to understand psychological constructs within large datasets. However, existing methods face significant challenges: dictionary-based approaches lack accuracy due to their inability to grasp context, while traditional machine learning models are often resource-intensive, require extensive coding expertise, and demand vast amounts of labeled data. A critical limitation is their poor performance on multilingual data, leading to a bias towards English and Western contexts. This paper addresses the need for a more accurate, accessible, and multilingual tool for psychological text analysis.

Methods

The researchers evaluated three versions of GPT (3.5 Turbo, GPT-4, and GPT-4 Turbo) for their ability to detect psychological constructs in text. They employed a "zero-shot" learning approach, meaning GPT was prompted to perform tasks without any prior specific training on the datasets. Prompts were designed to mimic those given to human annotators, asking GPT to classify text for sentiment, discrete emotions, offensiveness, and moral foundations. Performance was measured using accuracy and F1 scores, comparing GPT's outputs against human annotations, which are considered the gold standard. They also explored "few-shot" learning (providing a few examples in the prompt) and assessed test-retest reliability.

Datasets

The study utilized 15 diverse datasets comprising approximately 47,925 manually annotated tweets, news headlines, and Reddit comments across 12 languages. These datasets covered various psychological constructs:

  • Sentiment: English (2017), Arabic (2017), Swahili, Hausa, Amharic, Yoruba, Igbo, Twi, Kinyarwanda, Tsonga.
  • Discrete Emotions: English tweets (2020), Indonesian tweets (2020), News headlines (2023).
  • Offensiveness: English tweets (2019), Turkish tweets (2020).
  • Moral Foundations: Reddit comments (2022).

Crucially, datasets 7-16 were not publicly available on the internet before GPT's training cutoff, mitigating concerns about data contamination.

Key Insights and Ideas

  • GPT as a Universal Translator for Emotions: GPT demonstrates remarkable cross-lingual capabilities, accurately assessing sentiment and emotions even in languages with fewer speakers and less representation in typical training data.
  • Beyond Keywords: Context Matters: GPT's performance significantly surpasses dictionary-based methods, highlighting its ability to understand nuanced language and context, a major limitation of older techniques.
  • Democratizing NLP: The ease of use with simple prompts and minimal coding requirements makes advanced NLP accessible to a broader range of social scientists, reducing the barrier to entry.
  • Rapid Evolution: The consistent improvement in performance across GPT versions (3.5 to 4 to 4 Turbo) underscores the rapid advancement in LLM capabilities, particularly for underrepresented languages.

Key Results

  • Superior to Dictionaries: GPT's performance in detecting psychological constructs was significantly better than English-language dictionary analysis, with correlation coefficients ranging from 0.59 to 0.77 for GPT compared to 0.20 to 0.30 for dictionaries.
  • Competitive with ML Models: GPT performed nearly as well as, and in some cases better than, top-performing fine-tuned machine learning models, especially when no additional training data was available for GPT. For instance, in African languages, GPT-4 Turbo achieved an average F1 score of 0.603, while fine-tuned models averaged 0.720.
  • Multilingual Success: GPT showed strong performance across 12 languages, with notable improvements in lesser-spoken languages with each successive model version. For example, GPT-3.5's average F1 for African languages was 0.455, while GPT-4 Turbo reached 0.600.
  • Reliable and Consistent: GPT exhibited very high test-retest reliability (Cohen's kappa between 0.93 and 0.99) even when prompts were translated between languages, indicating robust and reproducible results.
  • Cost-Effective: While not free, GPT's API costs are presented as significantly lower than the cost of hiring human annotators or developing custom ML models.

Limitations and Future Work

  • Fine-tuning Advantage: While GPT is powerful in a zero-shot setting, fine-tuned models (including fine-tuned GPT) can achieve higher accuracy, especially for complex constructs or specific datasets.
  • Potential Biases: The paper acknowledges that LLMs can reflect human biases, and researchers should be mindful of this when interpreting results, particularly concerning WEIRD populations.
  • "Black Box" Nature: GPT's internal workings are not transparent, making it difficult to understand why it produces certain outputs, which can be a limitation for interpretability compared to dictionary methods.
  • Evolving Models: The rapid evolution of LLMs means that findings may need to be re-evaluated with newer versions. The authors encourage researchers to stay updated with new models like GPT-40.
  • Future Directions: The authors suggest further research into GPT's accuracy across a wider range of languages and cultures, exploring prompt engineering for improved performance, and investigating potential biases in more depth. They also highlight the need to compare GPT with other emerging LLMs.

Summary updated

Abstract

The social and behavioral sciences have been increasingly using automated text analysis to measure psychological constructs in text. We explore whether GPT, the large-language model (LLM) underlying the AI chatbot ChatGPT, can be used as a tool for automated psychological text analysis in several languages. Across 15 datasets (<i>n</i> = 47,925 manually annotated tweets and news headlines), we tested whether different versions of GPT (3.5 Turbo, 4, and 4 Turbo) can accurately detect psychological constructs (sentiment, discrete emotions, offensiveness, and moral foundations) across 12 languages. We found that GPT (<i>r</i> = 0.59 to 0.77) performed much better than English-language dictionary analysis (<i>r</i> = 0.20 to 0.30) at detecting psychological constructs as judged by manual annotators. GPT performed nearly as well as, and sometimes better than, several top-performing fine-tuned machine learning models. Moreover, GPT's performance improved across successive versions of the model, particularly for lesser-spoken languages, and became less expensive. Overall, GPT may be superior to many existing methods of automated text analysis, since it achieves relatively high accuracy across many languages, requires no training data, and is easy to use with simple prompts (e.g., "is this text negative?") and little coding experience. We provide sample code and a video tutorial for analyzing text with the GPT application programming interface. We argue that GPT and other LLMs help democratize automated text analysis by making advanced natural language processing capabilities more accessible, and may help facilitate more cross-linguistic research with understudied languages.
BibTeX
@inproceedings{rathje2024gpt,
  author = {Steve Rathje and Dan-Mircea Mirea and Ilia Sucholutsky and Raja Marjieh and Claire Robertson and Jay Joseph Van Bavel},
  title = {{GPT is an effective tool for multilingual psychological text analysis}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2024},
  doi = {10.1073/pnas.2308950121},
}
Figure