A Graph per Persona: Reasoning about Subjective Natural Language Descriptions

Eunjeong Hwang, Vered Shwartz, Dan Gutfreund, Veronika ThostACL 2024 · DOI

Abstract

Large Language Models (LLMs) often struggle with subjective reasoning—understanding a specific user's unique web of opinions and preferences. This paper introduces GOO (Graph per Persona), a framework that models an individual's subjective opinions and their implicit implications as a structured relational graph. By reasoning over these graphs using Graph Attention Networks, GOO outperforms state-of-the-art LLMs on personalization tasks while providing transparent, human-readable explanations.

Introduction

As LLMs are increasingly used for personalized assistants, their inability to handle subjective reasoning becomes a critical bottleneck. Current models face three major hurdles: the implicit nature of user preferences, a tendency toward demographic bias (relying on stereotypes rather than history), and a lack of interpretability in their "black box" decision-making. GOO addresses these by transforming a user's history into a structured "Persona Graph" that captures both explicit statements and their logical implications.

Methodology: The GOO Pipeline

The authors propose a multi-step pipeline to construct and reason over persona-specific graphs. First, past question-answer pairs are converted into declarative statements using an LLM. Next, to capture implicit knowledge, they use a Wizard-Vicuna-30B model to generate potential inferences from these explicit statements (e.g., "I do not use Instagram" implies "The person may not be interested in sharing photos"). These statements are then treated as nodes in a multi-relational graph. Edges between nodes represent the strength of logical relationships, predicted by a fine-tuned T5-base NLI (Natural Language Inference) model.

The GOO pipeline: from raw survey data to a reasoned answer via graph construction and GNN encoding.

Finally, a Graph Attention Network (GAT) processes the graph to identify relevance paths from the target question to potential answer choices. By learning "attention flows," the model can navigate through a user's logical web to find the most consistent answer. This path-based approach not only improves accuracy but also allows for the extraction of specific reasoning chains that explain the model's final prediction.

Evaluation and Results

The framework was tested on the OpinionQA benchmark, a large-scale dataset consisting of approximately 45,000 QA pairs from 5,000 users across 15 diverse topics (e.g., Guns, Automation, Gender, and Sexual Harassment) sourced from the Pew Research Center. GOO achieved an average accuracy of 54.9%, notably outperforming GPT-3.5 (51.0%) and Mistral-7B (52.8%). Interestingly, the authors found that GOO and LLMs often succeed on different types of questions; combining the two approaches could potentially push accuracy as high as 73%.

Beyond accuracy, GOO demonstrated significant improvements in:

  • Consistency: Performance was more stable across different demographic groups compared to the high variance seen in LLMs.
  • Explainability: The model extracts reasoning paths that human evaluators deemed reasonable 83% of the time.
A comparison of reasoning paths: GOO selects relevant past opinions to predict a target opinion, providing a transparent chain of logic.

Discussion and Key Insights

A central finding is that relational structure matters. Treating opinions as a structured graph rather than a flat list of text allows the model to better navigate the complex web of human logic. Furthermore, by grounding reasoning in a user's actual stated history, GOO effectively mitigates demographic hallucinations. The inclusion of LLM-generated "implications" was found to be particularly crucial for complex topics like global attitudes and sexual harassment, where opinions are rarely surface-level.

Conclusion

GOO represents a step forward in personalized AI by combining the generative power of LLMs with the structured reasoning of graph networks. While the pipeline's complexity remains a challenge for deployment, its ability to provide robust, interpretable, and personalized reasoning offers a promising alternative to standard black-box prompting.

Summary updated

BibTeX
@inproceedings{hwang2024graph,
  author = {Eunjeong Hwang and Vered Shwartz and Dan Gutfreund and Veronika Thost},
  title = {{A Graph per Persona: Reasoning about Subjective Natural Language Descriptions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.115},
}
Figure