Aligning to Adults Is Easy, Aligning to Children Is Hard: A Study of Linguistic Alignment in Dialogue Systems

Dorothea French, Sidney D’Mello, Katharina von der WenseACL 2024

TL;DR

This paper investigates how well state-of-the-art dialogue systems, ChatGPT and Llama2, linguistically align with adult and child users. While both systems show promising alignment with adults, they fall short when interacting with children, highlighting a significant gap in current AI capabilities for child-directed applications.

What Problem Does This Paper Address?

Conversational AI systems aim to mimic human interaction, and a key aspect of this is linguistic alignment. Linguistic alignment, where speakers unconsciously mirror each other's language (words, syntax, concepts), fosters shared understanding, increases engagement, and improves conversational outcomes. This is particularly crucial in contexts like language acquisition, where parental alignment significantly boosts a child's development. However, the NLP community has largely overlooked studying this phenomenon in advanced dialogue systems. This paper addresses this gap by asking: 1) How well do ChatGPT and Llama2 align with adult users, and how does this compare to human levels? 2) Do these systems exhibit high alignment with children, similar to how parents do, especially considering potential applications in child language acquisition?

Methods

The researchers developed a methodology to quantify linguistic alignment across three levels: syntactic, lexical, and semantic.

  • Syntactic Alignment: This measures the similarity in the grammatical structure of utterances. It involves tagging parts-of-speech (POS) and calculating the cosine similarity between the unique POS tag vectors of the context and the response.
  • Lexical Alignment: Similar to syntactic alignment, but instead of POS tags, it uses word lemmas to capture word choice similarity.
  • Semantic Alignment: This assesses the overlap in meaning between utterances. It uses word2vec embeddings to represent utterances as vectors and calculates the cosine similarity between these semantic vectors.

These metrics were applied to generated responses from ChatGPT and Llama2, comparing them against "gold" (human-provided) responses from existing dialogue datasets.

Datasets

The study utilized two primary datasets:

  • Switchboard Dialogue Acts Corpus (SWDA): This corpus consists of adult-adult phone conversations and was used to evaluate the systems' alignment with adult users (RQ1).
  • CHILDES Dataset: This dataset comprises conversations between caretakers and children and was used to assess the systems' alignment with children (RQ2).

The researchers extracted relevant dialogue excerpts, ensuring sufficient context (35 turns) for the models to generate responses.

Key Insights and Ideas

  • Adult Alignment is Approaching Human Levels: State-of-the-art models like ChatGPT are getting remarkably close to human levels of linguistic alignment when conversing with adults.
  • Llama2 Over-Aligns with Adults: Interestingly, Llama2 (especially the 7B and 13B versions) actually shows higher alignment than humans with adult users. This could suggest it might be less likely to introduce new topics, potentially making conversations less dynamic.
  • Children are a Major Challenge: Both ChatGPT and Llama2 exhibit significantly lower alignment levels when interacting with children compared to adults, and crucially, below human parent levels. This indicates a substantial gap in current AI's ability to effectively engage with young, non-fluent speakers.
  • Alignment vs. Quality Trade-off: While high alignment is generally good, the study hints at a potential trade-off. Llama2's mimicry of stylistic elements, while high in alignment, sometimes resulted in less constructive or novel responses, scoring lower in qualitative assessments compared to ChatGPT.
  • Systemic Response to Children: The study suggests that current systems respond with a similar level of alignment regardless of whether the target is an adult or a child, which is a key finding given the expectation of higher alignment in child-directed interactions.

Key Results

  • Adult Alignment (RQ1):
    • ChatGPT's alignment is close to human levels across syntactic, lexical, and semantic measures.
  • Llama2 (13B and 7B) shows elevated alignment compared to humans, particularly in lexical and semantic categories.
  • Child Alignment (RQ2):
    • ChatGPT's alignment with children is significantly lower than with adults, with syntactic alignment being 12% lower, lexical 37% lower, and semantic 17% lower than human parent levels.
  • Llama2 also shows decreased alignment with children, though its lexical and semantic alignment remain higher than ChatGPT's.
  • Qualitative assessments showed ChatGPT's responses to children were more convincing (average 3.86) than Llama2's (7B: 3.02, 13B: 3.35).
  • A higher percentage of Llama2 responses to children were rated as poor (41-63%) compared to ChatGPT (15%).

Limitations and Future Work

  • No Human Participants: The study relied on existing datasets rather than live human participants, which might mean systems behave differently when dynamically adapting to a real user.
  • Training Data Contamination: There's a possibility that the dialogue systems were trained on parts of the datasets used for evaluation, which could inflate alignment scores. However, qualitative assessment did not reveal high similarity between gold and generated responses.
  • Ambiguity of "Ideal" Alignment: The paper acknowledges that while alignment correlates with positive outcomes, the exact "ideal" level of alignment across different conversation types is still an open question.
  • Future Work: The authors plan to investigate alignment with adult learners or non-typical speakers, explore techniques for achieving closer-to-human alignment, and examine alignment in multi-turn conversational structures and their related outcomes. They also aim to explore additional datasets and models.

Summary updated

BibTeX
@inproceedings{french2024aligning,
  author = {Dorothea French and Sidney D’Mello and Katharina von der Wense},
  title = {{Aligning to Adults Is Easy, Aligning to Children Is Hard: A Study of Linguistic Alignment in Dialogue Systems}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure