Persona-SQ: A Personalized Suggested Question Generation Framework For Real-world Documents

Zihao Lin, Zichao Wang, Yuanting Pan, Varun Manjunatha, Ryan A. Rossi, Angela Lau, Lifu Huang, Tong Sun

Abstract

Suggested questions (SQs) provide an effective initial interface for users to engage with their documents in AI-powered reading applications. In practical reading sessions, users have diverse backgrounds and reading goals, yet current SQ features typically ignore such user information, resulting in homogeneous or ineffective questions. We introduce a pipeline that generates personalized SQs by incorporating reader profiles (professions and reading goals) and demonstrate its utility in two ways: 1) as an improved SQ generation pipeline that produces higher quality and more diverse questions compared to current baselines, and 2) as a data generator to fine-tune extremely small models that perform competitively with much larger models on SQ generation. Our approach can not only serve as a drop-in replacement in current SQ systems to immediately improve their performance but also help develop on-device SQ models that can run locally to deliver fast and private SQ experience.

Introduction

When users interact with AI assistants for document analysis, "Suggested Questions" (SQs) often serve as the primary entry point. However, current systems predominantly utilize a document-centric approach, extracting general information without considering the specific needs of the reader. A financial report, for instance, is read differently by a regulator than by a hedge fund manager. This lack of personalization leads to generic, repetitive suggestions that may not align with a user's professional background or specific reading goals.

Persona-SQ addresses this "homogeneity" problem by synthetically injecting user profiles into the generation process. This framework allows for the creation of highly relevant, diverse questions while maintaining user privacy, as it does not rely on tracking real-time user data for static documents like PDFs.

Methodology

The Persona-SQ framework utilizes a five-step pipeline designed to simulate diverse reader perspectives and generate targeted inquiries:

  1. Document Collection: Real-world documents are gathered from specialized domains, including finance, law, and academia.
  2. Persona Generation: A Large Language Model (LLM) brainstorms relevant professional roles and specific reading goals for each document (e.g., an "Auditor" seeking to "verify compliance").
  3. Persona Filtering: The system normalizes roles and filters out goals that lack high relevance to the assigned persona to ensure quality.
  4. Personalized Question Generation: The LLM generates questions based on the intersection of the document content and the specific persona/goal pair.
  5. Question Filtering: A rigorous quality control process checks for length, answerability within the document, and relevance to the intended persona using an LLM-based judge.
The Persona-SQ pipeline, illustrating the five stages from document input to personalized question output.

This pipeline serves both as a high-performance upgrade for cloud-based models like GPT-4o and as a synthetic data engine for training smaller, on-device models.

Experimental Setup

The study utilized three primary document sources: annual reports (Fns2020), legal contracts (CUAD), and research papers (qsper). Using the Persona-SQ pipeline, the researchers curated a synthetic dataset of 100,000 high-quality questions. This dataset was used to fine-tune SmolLM-360M, a "tiny" model optimized for local execution.

Results and Discussion

Diversity and Alignment

Experimental results indicate that without persona guidance, LLMs tend to converge on a "dominant" persona for specific domains—for example, legal questions often default to a lawyer's perspective, ignoring other stakeholders. Persona-SQ forces the model to adopt different lenses, significantly increasing the semantic diversity of the output and reducing redundancy.

Coverage and Human Preference

As shown in the evaluation of coverage ratios, the inclusion of personas allows the system to address a much wider range of document content relevant to specific professional needs. In A/B testing, human evaluators preferred Persona-SQ generated questions 75.8% of the time over standard baselines.

Comparison of coverage ratios with and without persona guidance across legal and financial domains, showing significant improvements when personas are utilized.

On-Device Efficiency

A key achievement of this research is the performance of TinyDocLM-SQ. Despite having only 360M parameters, the model matched the performance of GPT-3.5 Turbo in SQ generation. Occupying only ~760MB of space and delivering sub-second latency on standard laptop CPUs, it demonstrates a viable path for private, high-performance AI features that run entirely locally.

Conclusion

Persona-SQ demonstrates that incorporating synthetic reader profiles significantly enhances the utility and diversity of suggested questions in AI reading assistants. While the current personas are synthetic, they provide a robust proxy for real-world user needs. Future work will explore "closing the loop" by integrating anonymized user feedback to further refine these synthetic personas. Additionally, the researchers aim to distill the multi-stage filtering process directly into the models to reduce computational overhead for production environments.

Summary updated

Abstract

Suggested questions (SQs) provide an effective initial interface for users to engage with their documents in AI-powered reading applications. In practical reading sessions, users have diverse backgrounds and reading goals, yet current SQ features typically ignore such user information, resulting in homogeneous or ineffective questions. We introduce a pipeline that generates personalized SQs by incorporating reader profiles (professions and reading goals) and demonstrate its utility in two ways: 1) as an improved SQ generation pipeline that produces higher quality and more diverse questions compared to current baselines, and 2) as a data generator to fine-tune extremely small models that perform competitively with much larger models on SQ generation. Our approach can not only serve as a drop-in replacement in current SQ systems to immediately improve their performance but also help develop on-device SQ models that can run locally to deliver fast and private SQ experience.
BibTeX
@inproceedings{lin2025persona,
  author = {Zihao Lin and Zichao Wang and Yuanting Pan and Varun Manjunatha and Ryan A. Rossi and Angela Lau and Lifu Huang and Tong Sun},
  title = {{Persona-SQ: A Personalized Suggested Question Generation Framework For Real-world Documents}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure