Abstract
Suggested questions (SQs) provide an effective initial interface for users to engage with their documents in AI-powered reading applications. In practical reading sessions, users have diverse backgrounds and reading goals, yet current SQ features typically ignore such user information, resulting in homogeneous or ineffective questions. We introduce a pipeline that generates personalized SQs by incorporating reader profiles (professions and reading goals) and demonstrate its utility in two ways: 1) as an improved SQ generation pipeline that produces higher quality and more diverse questions compared to current baselines, and 2) as a data generator to fine-tune extremely small models that perform competitively with much larger models on SQ generation. Our approach can not only serve as a drop-in replacement in current SQ systems to immediately improve their performance but also help develop on-device SQ models that can run locally to deliver fast and private SQ experience.
Introduction
When users interact with AI assistants for document analysis, "Suggested Questions" (SQs) often serve as the primary entry point. However, current systems predominantly utilize a document-centric approach, extracting general information without considering the specific needs of the reader. A financial report, for instance, is read differently by a regulator than by a hedge fund manager. This lack of personalization leads to generic, repetitive suggestions that may not align with a user's professional background or specific reading goals.
Persona-SQ addresses this "homogeneity" problem by synthetically injecting user profiles into the generation process. This framework allows for the creation of highly relevant, diverse questions while maintaining user privacy, as it does not rely on tracking real-time user data for static documents like PDFs.
Methodology
The Persona-SQ framework utilizes a five-step pipeline designed to simulate diverse reader perspectives and generate targeted inquiries:
- Document Collection: Real-world documents are gathered from specialized domains, including finance, law, and academia.
- Persona Generation: A Large Language Model (LLM) brainstorms relevant professional roles and specific reading goals for each document (e.g., an "Auditor" seeking to "verify compliance").
- Persona Filtering: The system normalizes roles and filters out goals that lack high relevance to the assigned persona to ensure quality.
- Personalized Question Generation: The LLM generates questions based on the intersection of the document content and the specific persona/goal pair.
- Question Filtering: A rigorous quality control process checks for length, answerability within the document, and relevance to the intended persona using an LLM-based judge.
This pipeline serves both as a high-performance upgrade for cloud-based models like GPT-4o and as a synthetic data engine for training smaller, on-device models.
Experimental Setup
The study utilized three primary document sources: annual reports (Fns2020), legal contracts (CUAD), and research papers (qsper). Using the Persona-SQ pipeline, the researchers curated a synthetic dataset of 100,000 high-quality questions. This dataset was used to fine-tune SmolLM-360M, a "tiny" model optimized for local execution.
Results and Discussion
Diversity and Alignment
Experimental results indicate that without persona guidance, LLMs tend to converge on a "dominant" persona for specific domains—for example, legal questions often default to a lawyer's perspective, ignoring other stakeholders. Persona-SQ forces the model to adopt different lenses, significantly increasing the semantic diversity of the output and reducing redundancy.
Coverage and Human Preference
As shown in the evaluation of coverage ratios, the inclusion of personas allows the system to address a much wider range of document content relevant to specific professional needs. In A/B testing, human evaluators preferred Persona-SQ generated questions 75.8% of the time over standard baselines.
On-Device Efficiency
A key achievement of this research is the performance of TinyDocLM-SQ. Despite having only 360M parameters, the model matched the performance of GPT-3.5 Turbo in SQ generation. Occupying only ~760MB of space and delivering sub-second latency on standard laptop CPUs, it demonstrates a viable path for private, high-performance AI features that run entirely locally.
Conclusion
Persona-SQ demonstrates that incorporating synthetic reader profiles significantly enhances the utility and diversity of suggested questions in AI reading assistants. While the current personas are synthetic, they provide a robust proxy for real-world user needs. Future work will explore "closing the loop" by integrating anonymized user feedback to further refine these synthetic personas. Additionally, the researchers aim to distill the multi-stage filtering process directly into the models to reduce computational overhead for production environments.
Summary updated