Adapting Large Language Models via Multi-Task Learning for Mental Health Problem, Emotion, and Sentiment Detection

Amir Rafiq Bhat, Rahim Dehkharghani, Zeki ErdemarXiv 2026 · arXiv

This paper explores how to effectively adapt Large Language Models (LLMs) for the sensitive and interconnected tasks of mental health detection, emotion recognition, and sentiment analysis. By investigating sequential multi-task learning (MTL), the authors demonstrate that the order in which a model learns these tasks is just as important as the data itself.

TL;DR

The order in which you fine-tune an LLM on related tasks—like mental health detection and emotion analysis—critically determines its final performance. Using sequential Parameter-Efficient Fine-Tuning (PEFT), the authors show that a "curriculum" approach outperforms simultaneous multi-task learning, which often leads to model confusion and "generative format collapse."

What Problem Does This Paper Address?

Detecting mental health issues from social media text is notoriously difficult due to linguistic nuance, data imbalance, and the inherent overlap between a person's emotional state, their sentiment, and potential psychiatric conditions.

While LLMs like Llama 3.1 are powerful generalists, they aren't specialized for clinical contexts. Full fine-tuning of these 8-billion-parameter models is computationally expensive, and traditional "Joint Multi-Task Learning" (training on all tasks at once) often suffers from negative transfer, where learning a simple task (like sentiment) interferes with the model's ability to learn a complex one (like multi-label emotion detection). The authors set out to find if a sequential approach—learning one task after another—could mitigate these issues.

Figure 1: Motivation and overview of the proposed methodology.

Methods

The researchers used the Llama 3.1 8B Instruct model as their foundation. To make adaptation efficient, they employed QLoRA (Quantized Low-Rank Adaptation), which combines 4-bit quantization with LoRA to update less than 1% of the model's parameters.

The core of the experiment was comparing two sequential fine-tuning "curricula":

  1. Seq1 (MHP-first): Mental Health Problem $\rightarrow$ Emotion $\rightarrow$ Sentiment.
  2. Seq2 (Emo-first): Emotion $\rightarrow$ Mental Health Problem $\rightarrow$ Sentiment.

In each sequence, the model retained the learned "adapters" from the previous step. They compared these against a Joint-MTL baseline, where the model was prompted to output all three labels simultaneously in a single pass.

Datasets

The study utilized the Extended SWMH (SuicideWatch and Mental Health) dataset, consisting of 4,243 English social media posts.

  • Primary Task: Mental Health Problem (MHP) detection (Anxiety, Bipolar, Depression, or SuicideWatch).
  • Auxiliary Task 1: Multi-label Emotion detection (8 categories including anger, fear, joy, and sadness).
  • Auxiliary Task 2: Binary Sentiment analysis (Positive or Negative).

Key Insights and Ideas

  • Task Order is a Critical Design Parameter: There is no "one-size-fits-all" sequence. If the primary goal is clinical accuracy (MHP), training on that task first yields the best results. However, starting with Emotion detection creates a "semantic bridge" that significantly boosts performance on the other two tasks.
  • The "Simplicity" Interference: Fine-tuning on a simple binary task (sentiment) after a complex multi-label task (emotion) can actually degrade the model's nuanced emotional representations. The model essentially "overwrites" complex knowledge to optimize for the simpler objective.
  • Generative Format Collapse: One of the most striking findings was that Joint-MTL caused the model to "collapse." When asked to perform all tasks at once, the base model often hallucinated invalid labels or failed to follow the output format entirely. Sequential training completely eliminated this failure mode.
  • Correlation as a Guide: The authors used a correlation matrix to show that classes like 'depression' strongly correlate with 'hopelessness' and 'sadness.' This relatedness is what makes multi-task learning effective, provided the sequence is handled correctly.
Overall methodology and sequential multi-task learning workflow

Key Results

  • MHP Detection: The MHP-first sequence (Seq1) achieved the highest Micro-F1 score of 0.7624, a significant jump from the base model's zero-shot performance of 0.6188.
  • Emotion & Sentiment: The Emo-first sequence (Seq2) was superior for the auxiliary tasks, reaching an F1 of 0.6052 for Emotion and a near-perfect 0.9459 for Sentiment.
  • Baseline Failure: The Joint-MTL model's performance on MHP detection catastrophically collapsed to an F1 of 0.1953, proving that "prompt packing" multiple complex tasks can overwhelm the model's representational capacity.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Data Imbalance: The model still struggled with minority classes (like 'joy' or 'calmness') because they appeared so rarely in the clinical dataset.
  • Computational Constraints: Due to the high cost of training 8B parameter models, the study relied on single training runs; future work should include multiple runs to establish statistical significance.
  • Curriculum Exploration: Only two of the six possible task permutations were tested. Exploring the full space of task orderings remains an open area for research.
  • Ethics: The paper stresses that this is a research tool, not a diagnostic one. Any real-world application must include "human-in-the-loop" oversight to prevent harms from misclassification.

Summary updated

Abstract

This study investigates sequential multi-task learning using Parameter-Efficient Fine-Tuning (PEFT), LoRA, and 4-bit quantization on Llama 3.1 8B Instruct for mental health problem detection, emotion detection, and sentiment analysis. Using 4,243 English social media posts, it compares two task sequences and demonstrates that task ordering significantly affects knowledge transfer and task interference. The MHP-first sequence achieves the best mental health problem detection (Micro-F1: 0.7624), while the Emotion-first sequence performs better in emotion (F1: 0.6052) and sentiment analysis (F1: 0.9459). A Joint-MTL baseline further highlights the challenges of simultaneous learning, including task interference and generative format collapse. Overall, the findings demonstrate the effectiveness of sequential PEFT for mental health NLP while showing that task order is critical to balancing performance across related tasks. The integration of LoRA and 4-bit quantization also reduces computational and memory requirements, supporting the adaptation of large language models in resource-constrained environments.
BibTeX
@misc{bhat2026adapting,
  author = {Amir Rafiq Bhat and Rahim Dehkharghani and Zeki Erdem},
  title = {{Adapting Large Language Models via Multi-Task Learning for Mental Health Problem, Emotion, and Sentiment Detection}},
  year = {2026},
  eprint = {2611.10797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10797},
}
Figure