Persona-SQ: A Personalized Suggested Question Generation Framework For Real-world Documents

Zihao Lin, Zichao Wang, Yuanting Pan, Varun Manjunatha, Ryan A. Rossi, Angela Lau, Lifu Huang, Tong Sun

TL;DR

This paper introduces Persona-SQ, a novel framework for generating personalized suggested questions (SQs) for AI-powered reading assistants. By incorporating user personas (professions and reading goals), Persona-SQ significantly improves the quality and diversity of SQs compared to existing methods, and also enables the training of highly efficient, small models for on-device deployment.

What Problem Does This Paper Address?

Current AI-powered reading applications, like those found in document assistants, often generate "suggested questions" (SQs) to help users engage with documents. However, these SQs are typically generic and fail to account for the diverse backgrounds, professions, and reading goals of individual users. This leads to homogeneous and often ineffective questions that don't cater to specific user needs, ultimately hindering user engagement and document comprehension. The challenge lies in personalizing these SQs without relying on explicit user profile information, which can be privacy-sensitive and difficult to collect.

Methods

Persona-SQ employs a pipeline approach to generate personalized SQs. The core steps are:

  1. Document Collection: Gathering sets of open-source documents from various domains.
  2. Persona Generation: Using a Large Language Model (LLM) to generate relevant professional roles and their corresponding reading goals for each document.
  3. Quality Control for Personas and Goals: Normalizing similar professions and filtering goals based on relevance to the persona and document.
  4. Personalized Question Generation: Leveraging an LLM to generate questions based on the document, the persona, and their associated goals.
  5. Quality Control for Questions: Implementing multiple filters, including length constraints, LLM-based relevance scoring (to both persona and document), and answerability verification.

A key innovation is the synthetic generation of persona-based SQs, which can then be used to fine-tune extremely small language models.

Datasets

  • Training and Evaluation Datasets: The paper utilizes three sets of public documents from the finance, legal, and academia domains. Specific datasets mentioned include:
  • Fns2020 (for finance)
    • CUAD (for legal)
    • qsper (for academia)
  • Synthetic Dataset: Persona-SQ is used to curate a large synthetic dataset of approximately 23,000 questions from around 1,600 documents, which is then used for fine-tuning smaller models.
  • SQuAD: Used as a baseline for fine-tuning models, repurposed for SQ generation.

Key Insights and Ideas

  • Persona Matters: Incorporating user personas (profession and goals) significantly improves the relevance and diversity of generated questions, moving beyond document-centric generation.
  • Synthetic Data for Small Models: Persona-SQ can generate high-quality synthetic data that enables training very small, efficient language models (e.g., 360 million parameters) to perform competitively with much larger models on the SQ generation task. This opens doors for on-device deployment.
  • Domain-Specific Persona Bias: Without persona guidance, SQ generation can exhibit a bias towards dominant personas within a domain (e.g., "lawyer" in legal documents), limiting personalization.
  • Quality Control is Crucial: A multi-stage quality control process is essential for ensuring the generated personas and questions are relevant, logical, and answerable.

Key Results

  • Improved SQ Quality and Diversity: Persona-SQ consistently produces more diverse and higher-quality SQs compared to baselines that do not use persona information, as validated by automatic and human evaluations. For instance, Table 1 shows Persona-SQ generates more diverse questions. Table 4 indicates Persona-SQ significantly improves question importance.
  • Competitive Small Models: Fine-tuned small models (e.g., SmolLM 360M) using Persona-SQ generated data outperform baseline models and approach the performance of larger models like GPT-4o on SQ generation. Table 8 shows a fine-tuned SmolLM model matching GPT-3.5 Turbo performance.
  • User Preference: Human evaluations show users prefer SQs generated by Persona-SQ over those from baseline systems (Table 3).
  • Coverage Ratio Improvements: Figures 22-24 demonstrate that Persona-SQ significantly increases the coverage ratio across various personas, indicating better alignment between generated questions and their intended users.

Limitations and Future Work

  • Synthetic Personas: The current Persona-SQ uses synthetically generated personas rather than actual user profiles. While effective, this limits true personalization. The authors suggest that this synthetic data can serve as an anchor for collecting real user preference signals in the future.
  • Latency: The persona generation and quality control steps can increase system latency. For cloud deployments, specialized hardware can mitigate this. For on-device deployments, the efficiency of small models suggests the entire pipeline can be made feasible.
  • Handling Longer Context: The current system splits long documents into chunks. Extending the model to handle longer contexts is a future research direction.
  • On-Device Optimization: Further optimization techniques like quantization-aware training could reduce model size and improve latency for on-device deployment.
  • True Personalization: Future work could explore integrating actual user interaction data to further refine persona generation and achieve true personalization.

Summary updated

Abstract

Suggested questions (SQs) provide an effective initial interface for users to engage with their documents in AI-powered reading applications. In practical reading sessions, users have diverse backgrounds and reading goals, yet current SQ features typically ignore such user information, resulting in homogeneous or ineffective questions. We introduce a pipeline that generates personalized SQs by incorporating reader profiles (professions and reading goals) and demonstrate its utility in two ways: 1) as an improved SQ generation pipeline that produces higher quality and more diverse questions compared to current baselines, and 2) as a data generator to fine-tune extremely small models that perform competitively with much larger models on SQ generation. Our approach can not only serve as a drop-in replacement in current SQ systems to immediately improve their performance but also help develop on-device SQ models that can run locally to deliver fast and private SQ experience.
BibTeX
@inproceedings{lin2025persona,
  author = {Zihao Lin and Zichao Wang and Yuanting Pan and Varun Manjunatha and Ryan A. Rossi and Angela Lau and Lifu Huang and Tong Sun},
  title = {{Persona-SQ: A Personalized Suggested Question Generation Framework For Real-world Documents}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure