TL;DR
This paper introduces Personality-1260, a novel Chinese dataset for personality assessment through human-agent dialogues, and investigates how interaction length and agent personality affect assessment accuracy. Surprisingly, they found that fewer interaction rounds (1-2) often lead to better personality assessment results, and that the agent's personality can significantly influence the accuracy of specific trait assessments.
What Problem Does This Paper Address?
Automatically assessing human personality through natural language interaction is crucial for developing user-centered AI systems, from personalized recommendations to mental health support. However, existing datasets for this task often lack genuine interactivity or rely on static text, failing to capture the dynamic nature of human communication. Furthermore, the impact of interaction dynamics, such as the number of conversational turns and the personality of the AI agent itself, on assessment accuracy remains underexplored. This paper aims to fill this gap by creating a more interactive dataset and experimentally investigating these crucial factors.
Methods
The authors developed a system that controls Large Language Models (LLMs) to exhibit specific Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) through personality prompts. They then designed a "Prisoner's Dilemma" game scenario to elicit natural dialogues between human users and these AI agents over multiple rounds. The core of their methodology involves:
- Dataset Creation: Building the Personality-1260 dataset, which consists of 1260 rounds of human-agent dialogues and corresponding Big Five personality questionnaire results from 42 participants.
- Experimental Design: Conducting experiments to systematically vary the number of interaction rounds and the personality of the interacting agent.
- LLM Evaluation: Utilizing several state-of-the-art LLMs (GPT-4.1-Nano, GPT-4.1-Mini, GPT-4.1, DeepSeek-V3, Gemini-2.5-Flash, Claude-4-Sonnet) to assess personality based on the collected dialogues.
- Evaluation Metrics: Employing Mean Absolute Error (MAE) to quantify the accuracy of personality assessments.
- Human Annotation: Validating their findings by having human experts annotate a subset of the data.
Datasets
- Personality-1260: A newly introduced Chinese dataset comprising 1260 interaction rounds between humans and AI agents with controlled personalities. This dataset is unique in its focus on dynamic, multi-round dialogues and includes participants' BFI-44 personality questionnaire results. It aims to support research on personality assessment in interactive settings.
- Existing Datasets (mentioned for context): The paper contrasts Personality-1260 with static text datasets (e.g., MBTI, articles) and manually annotated dialogue datasets from fictional sources (e.g., FriendsPersona), highlighting the limitations of these in capturing authentic, interactive personality expressions.
Key Insights and Ideas
- Less is More: The most surprising finding is that fewer interaction rounds (specifically 1-2) often yield better personality assessment accuracy. This challenges the intuition that more data always leads to better results and has significant implications for designing efficient assessment systems.
- Agent Personality Matters: The personality of the AI agent significantly influences the accuracy of personality assessment, particularly in the initial rounds. For instance, interacting with a neurotic agent improved the assessment of the user's neuroticism in the first round.
- Trait-Specific Dynamics: Different personality dimensions exhibit varying sensitivities to interaction duration. For example, Conscientiousness assessment accuracy tends to decrease with more rounds, while Openness may benefit from longer interactions.
- Neuroticism is Tricky: Neuroticism consistently shows the highest MAE across models, indicating it's the most challenging trait to assess accurately through dialogue analysis, likely due to its less explicit behavioral manifestations in text.
- LLM vs. Human: The study found that LLM-based personality assessments are comparable in accuracy to those of human annotators, validating the LLMs' capabilities in this domain.
Key Results
- Optimal Rounds: The optimal data requirement for personality assessment was found to be 1-2 rounds of interaction, with longer interactions often leading to decreased performance or no improvement.
- Agent Influence: Interacting with a neurotic agent led to significantly lower MAE for Neuroticism assessment in the first round compared to other agent personalities. Similarly, high Conscientiousness agents facilitated optimal Agreeableness assessment in the first round.
- Model Performance: While GPT-4.1 generally performed well, models with larger parameter counts (GPT-4.1, DeepSeek-V3, Claude-4-Sonnet) showed greater stability in assessments compared to smaller models.
- Human Agreement: Human annotators showed good to excellent levels of agreement (ICC ≥ 0.60) in their relative judgments of personality traits, though systematic differences in absolute rating levels were observed.
Limitations and Future Work
- Dataset Language: The Personality-1260 dataset is currently in Chinese. Future work should focus on building multilingual datasets to support cross-cultural personality assessment.
- Participant Diversity: The participant demographics were limited to university students. Expanding the participant pool to include more diverse backgrounds is necessary.
- Dataset Size: While sufficient for analysis, the dataset is relatively small compared to some existing benchmarks. Future work could involve expanding the dataset.
- Model Scope: The study primarily focused on closed-source LLMs. Further research could explore a wider range of open-source models.
- Interaction Complexity: The study suggests that the complex effects of interaction environment and round number need careful consideration in system design. Future work could explore more sophisticated interaction designs.
Summary updated