HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models

Swaroop Mishra, Elnaz Nouri

Abstract

Large Language Models (LLMs) offer immense potential for content creation, yet non-expert users often struggle with "prompt engineering" and the cognitive burden of identifying necessary details for specific tasks. This paper presents HELP ME THINK, a user-centric prompting strategy that flips the traditional interaction model. Instead of requiring complex instructions, the model adopts an expert persona to ask the user clarifying questions, then synthesizes the answers into high-quality customized content. Evaluation across diverse creative tasks demonstrates high relevance, coherence, and effective knowledge absorption.

Introduction: Overcoming the Thinking Burden

While LLMs are capable of generating professional bios or travel itineraries, the average person often faces "writer's block" when trying to provide the exact instructions and constraints required for high-quality output. This "thinking burden" stems from the user needing to know what information is necessary before they can even begin prompting.

Comparison of standard prompting versus the HELP ME THINK strategy.

Existing techniques like few-shot prompting or negative constraints are often too complex for casual users. HELP ME THINK addresses this by automating the information-gathering phase, making AI more accessible by guiding the user through the creative process.

Methodology

The researchers propose a model-agnostic, three-stage framework:

  1. Generate Questions: The model adopts an expert persona (e.g., "expert travel planner") and is instructed to ask the user clarifying questions to gather necessary data.
  2. Collect Answers: The user provides conversational responses. In experimental settings, crowdworkers simulated real-world usage to provide these answers.
  3. Generate Task-Specific Output: The model is prompted with the original goal and the collected question-answer pairs to synthesize the final product.

Data and Evaluation

To evaluate the strategy, a new dataset was constructed covering six creative domains: bio generation, travel planning, dialogue, poetry, event summaries, and story generation. The dataset comprises approximately 2,000 question-answer pairs and 180 task-specific outputs. The method's versatility was further tested on 57 additional tasks, ranging from bridge construction planning to sports team formation.

Key Results

Human evaluation revealed that the HELP ME THINK strategy significantly improves the quality and relevance of AI-generated content.

Performance metrics across various creative tasks.

Key performance metrics include:

  • 100% Relevance: All model-generated questions were found to be valid and relevant to the task.
  • 99% Coherence: The model successfully generated coherent sentences, adding appropriate context beyond the raw user input.
  • 70% Knowledge Absorption: On average, the model successfully transferred the majority of user-provided facts into the final output.
  • 94% Fact Accuracy: Most outputs contained no hallucinations or irrelevant information beyond the user's provided data.

Insights and Discussion

The core insight is that for personalized content, the AI's ability to ask the right questions is as valuable as its ability to generate text. By breaking complex tasks into manageable pieces, the model reduces the user's cognitive load.

Interestingly, the study found that LLMs (specifically GPT-3) act as natural polishers, correcting typos and grammar in user answers in roughly 83% of cases. However, there is a trade-off: providing too many details at once can decrease "knowledge absorption," suggesting that decomposing tasks into smaller chunks or fewer questions can improve final accuracy.

Limitations and Future Work

While promising, the 70% retention rate indicates room for improvement in handling large volumes of facts. Future research should explore multi-turn dialogues for deeper refinement, expand the strategy to non-English languages, and develop automated frameworks for large-scale stress-testing across niche domains.

Summary updated

Abstract

Controlling the text generated by language models and customizing the content has been a long-standing challenge. Existing prompting techniques proposed in pursuit of providing control are task-specific and lack generality; this provides overwhelming choices for non-expert users to find a suitable method for their task. The effort associated with those techniques, such as in writing examples, explanations, instructions, etc. further limits their adoption among non-expert users. In this paper, we propose a simple prompting strategy HELP ME THINK where we encourage GPT3 to help non-expert users by asking a set of relevant questions and leveraging user answers to execute the task. We demonstrate the efficacy of our technique HELP ME THINK on a variety of tasks. Specifically, we focus on tasks that are hard for average humans and require significant thinking to perform. We hope our work will encourage the development of unconventional ways to harness the power of large language models.
BibTeX
@inproceedings{mishra2023help,
  author = {Swaroop Mishra and Elnaz Nouri},
  title = {{HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=izyh0fbyoQa},
}
Figure