TL;DR
This paper explores how to make AI dialogue systems more personal and engaging in long-term conversations (Longitudinal Dialogues or LDs). They fine-tune large language models (LLMs) on personal user history, testing different ways to represent this knowledge, and find that structured representations improve performance but a significant gap remains compared to human-like interaction.
What Problem Does This Paper Address?
Current dialogue systems, whether task-oriented or for general chit-chat, are largely stateless. They don't remember past interactions or build a deep understanding of the user over time. This limits their ability to engage in truly personal conversations, which are characterized by recalling past events, understanding personal thoughts and emotions, and maintaining a consistent persona. Longitudinal Dialogues (LDs) present a significant challenge because they require systems to interact uniquely with each user over extended periods, learning and adapting based on a sparse sequence of dialogue sessions. The paper aims to address this by investigating how to generate responses in LDs that are not only appropriate and accurate but also personalized and engaging, by leveraging the user's past interactions as knowledge.
Methods
The core of this research involves fine-tuning two pre-trained language models (PLMs): GePpeTto (based on GPT-2) and iT5 (based on T5), both adapted for Italian. The key innovation lies in how they represent the "personal knowledge" extracted from previous user dialogue turns. They experiment with three methods:
- RAW: The raw text of the user's previous responses is fed directly to the model.
- Bag of Head Nouns (BOH): A more distilled representation where key nouns from the user's responses are extracted. This aims to capture important entities and concepts.
- Personal Space Graph (PSG): A structured representation where events (predicates) form edges between participants (nodes). This graph is then linearized to be compatible with LLMs.
The models are fine-tuned on a dataset of LDs and then evaluated using both automatic metrics (perplexity) and human evaluations across several criteria.
Datasets
The primary dataset used is a collection of Longitudinal Dialogues (LDs) in Italian, specifically from the mental health domain. This dataset consists of two dialogue sessions per user, where the first session contains personal life events, thoughts, and emotions. The user responses from the first session are used as the "personal knowledge" for generating responses in the second session. The authors note that this dataset is relatively small, which influences their experimental setup.
Key Insights and Ideas
- Structured Knowledge Matters: Simply feeding raw text from past conversations isn't as effective as using more structured representations like BOH or PSG. These refined methods help the model better utilize personal history.
- Trade-offs in Grounding: While grounding responses in personal knowledge generally improves appropriateness and engagement, it can sometimes decrease grammatical correctness and increase "hallucinations" (generating non-existent or contradictory information).
- Model Architecture Differences: The auto-regressive GePpeTto model showed more diverse responses compared to the encoder-decoder iT5 model, which tended to produce more similar responses.
- Human Evaluation is Crucial: Automatic metrics like perplexity are useful but don't fully capture the nuances of conversational quality. Human evaluations revealed significant gaps in contextualization and appropriateness, highlighting the difficulty of achieving human-level personalization.
- Contextualization is Subjective: The Inter-Annotator Agreement (IAA) for contextualization was notably low, indicating that assessing whether a response is truly contextualized is highly subjective and challenging for annotators.
Key Results
- iT5-Base Outperforms: The iT5-Base model, especially when grounded with refined knowledge representations (BOH and PSG), generally achieved better results in human evaluations for appropriateness, contextualization, and listening compared to GePpeTto.
- PSG Shows Promise: The Personal Space Graph (PSG) representation, in particular, led to the highest levels of contextualization and listening among the grounded models, and also improved correctness.
- Grounding Reduces Genericness: Grounding responses in personal knowledge helped reduce generic responses, but it also increased the instances of hallucinations, especially with the RAW representation.
- Significant Gap Remains: Despite improvements, the performance of even the best-performing models is still far from human-level conversational abilities in LDs, particularly in the mental health domain.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Size and Domain: The dataset is small and specific to the Italian language and the mental health domain, which may limit the generalizability of the findings.
- Model Size: They were unable to experiment with larger versions of iT5 due to computational constraints.
- Language Specificity: The models are trained on Italian, and language-specific limitations might affect performance.
Future work could involve exploring larger and more diverse datasets, investigating more sophisticated knowledge representation techniques, and developing models that can better balance grounding with factual accuracy and avoid hallucinations. The significant gap between current models and human performance also suggests a need for entirely new approaches to modeling long-term, personal conversations.
Summary updated