Generating Mental Health Transcripts with SAPE (Spanish Adaptive Prompt Engineering)

Daniel Cabrera Lozoya, Alejandro Berazaluce, Juan Perches, Eloy Lúa, Mike Conway, Simon D'AlfonsoNAACL 2024 · DOI

TL;DR

This paper introduces SAPE, a novel method for automatically generating effective prompts for Large Language Models (LLMs) to create synthetic Spanish psychotherapy transcripts. SAPE uses genetic algorithms to evolve prompts, and human evaluations by mental health professionals show that SAPE-generated transcripts are perceived as more authentic than those produced by other leading prompt engineering techniques.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful tools for data augmentation, especially in domains where data is scarce or sensitive. However, their performance heavily relies on the quality of the prompts used to guide them. Manually crafting these prompts, known as prompt engineering, is time-consuming and often yields suboptimal results. This is particularly true for specialized domains like mental health, where collecting authentic data (e.g., therapy transcripts) is challenging due to privacy concerns and the presence of protected health information. While existing prompt engineering techniques exist, they are predominantly English-centric and often require manual effort. This paper addresses the need for an automated, language-adaptive approach to generate high-quality synthetic data for mental health applications, specifically in Spanish, a language less explored in this context.

Methods

The core innovation is SAPE (Spanish Adaptive Prompt Engineering), a method that employs genetic algorithms to automate the generation and selection of prompts. SAPE works by:

  • Initialization: Starting with a base prompt and a set of "mutation prompts" (which are themselves evolved).
  • Evolutionary Process:
    • Mutation: Individuals (prompts) are mutated using various strategies inspired by biological reproduction (sexual, asexual, selective breeding, environmental adaptation, taught behaviors). These mutations modify the instruction prompt.
  • Fitness Evaluation: The quality of the generated text (therapy transcripts) is evaluated using a reward model trained on human feedback. This feedback is collected from mental health professionals who rank pairs of synthetic transcripts.
  • Selection: Individuals with higher fitness scores are more likely to survive and reproduce, driving the population of prompts towards better performance.
  • Adaptive Prompting: Unlike previous methods that focused on closed-ended tasks with clear correct answers, SAPE is designed for open-ended generation tasks like creating therapy transcripts, where "correctness" is subjective and requires human judgment.

Datasets

  • Training Data: 30 hours of Spanish counseling session videos from publicly available YouTube content were used. Utterances were extracted and annotated by a psychologist and a co-author, focusing on three therapeutic interaction types: "Mood check," "Change methods," and "Set goals." Protected health information was excluded.
  • Synthetic Data Generation: The text-davinci-003 model was used to generate synthetic therapy transcripts.
  • Human Feedback Data: A dataset of synthetic transcripts was created for training a Reinforcement Learning from Human Feedback (RLHF) model. This involved presenting pairs of synthetic transcripts to annotators and having them indicate a preference.
  • Evaluation Data: 180 synthetic therapy transcripts were generated using SAPE, Chain-of-Thought (CoT), and Reflexion prompts. These were then ranked by 8 psychologists.

Key Insights and Ideas

  • Automated Prompt Engineering for Open-Ended Tasks: SAPE demonstrates that genetic algorithms can be effectively applied to optimize prompts for generative tasks that lack a single "correct" answer, moving beyond previous applications in arithmetic or classification.
  • Importance of Human Feedback: The reliance on mental health professionals for feedback was crucial in training the reward model and evaluating the quality of synthetic transcripts, ensuring relevance to the domain.
  • Language Specificity Matters: While LLMs are often trained on vast English datasets, adapting prompt engineering techniques to specific languages like Spanish is essential for broader applicability and to avoid potential performance degradation.
  • Genetic Algorithms for Diversity: The use of a genetic algorithm with various mutation strategies helps explore a wider prompt space, potentially avoiding local optima and discovering more robust prompts.

Key Results

  • Qualitative Preference: Mental health professionals found SAPE-generated Spanish counseling transcripts to be more closely resembling authentic therapy transcripts compared to those generated using Chain-of-Thought (CoT) and Reflexion techniques.
  • Statistical Significance: Friedman tests revealed statistically significant differences in preferences for the generated texts across all therapeutic interaction types and for the cumulative rankings.
  • SAPE Outperforms: Post-hoc analysis showed significant preferences for SAPE over Reflexion and, in most cases, over CoT. Specifically, for "Mood check," SAPE was preferred over Reflexion. For "Change Method," SAPE was preferred over both Reflexion and CoT. For cumulative scores, SAPE was significantly preferred over both Reflexion and CoT.
  • "Set Goals" Anomaly: SAPE did not achieve the highest ranking for the "Set goals" prompt, which the authors attribute to the prompt's explicit instructional preamble, suggesting that prompt structure can influence performance.

Limitations and Future Work

  • Financial Constraints: The use of a paid OpenAI API limited the exploration of hyperparameter tuning (e.g., mutation rates, population size, generations), suggesting that further optimization could yield even better prompts.
  • Annotator/Evaluator Pool Size: The number of psychologists involved in RLHF and evaluation was limited by time and budget, suggesting that a larger pool could provide more robust results.
  • Language Bias: While SAPE is designed to be language-agnostic, using an LLM primarily trained on English to optimize Spanish prompts might lead to suboptimal performance compared to optimizing for English. Future work should explore developing LLM and NLP tools tailored for a wider range of languages.
  • Bias Assessment: SAPE's search strategy does not explicitly assess or mitigate potential biases introduced by the generated prompts. Future work should investigate methods to identify and prevent the propagation of biases in synthetic data.
  • Synthetic vs. Organic Data: While SAPE shows promising results, further research is needed to rigorously compare the quality of SAPE-generated synthetic transcripts with real-world organic data.
  • Prompt Evolution Strategy: The authors suggest exploring modifications to the "sexual reproduction" mutation prompt, as it consistently yielded lower fitness.

Summary updated

BibTeX
@inproceedings{lozoya2024generating,
  author = {Daniel Cabrera Lozoya and Alejandro Berazaluce and Juan Perches and Eloy Lúa and Mike Conway and Simon D'Alfonso},
  title = {{Generating Mental Health Transcripts with SAPE (Spanish Adaptive Prompt Engineering)}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.285},
}
Figure