HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models

Swaroop Mishra, Elnaz Nouri

Abstract

Controlling the text generated by language models and customizing the content has been a long-standing challenge. Existing prompting techniques proposed in pursuit of providing control are task-specific and lack generality; this provides overwhelming choices for non-expert users to find a suitable method for their task. The effort associated with those techniques, such as in writing examples, explanations, instructions, etc. further limits their adoption among non-expert users. In this paper, we propose a simple prompting strategy HELP ME THINK where we encourage GPT3 to help non-expert users by asking a set of relevant questions and leveraging user answers to execute the task. We demonstrate the efficacy of our technique HELP ME THINK on a variety of tasks. Specifically, we focus on tasks that are hard for average humans and require significant thinking to perform. We hope our work will encourage the development of unconventional ways to harness the power of large language models.

Introduction

Large Language Models (LLMs) are powerful tools, yet eliciting specific, high-quality, and personalized content remains a significant challenge for non-expert users. This difficulty often stems from "prompt engineering" hurdles, such as the "Writer's Block" problem—where users are unsure what information the model needs—and the complexity of following multi-step instructions or negative constraints.

Comparison of traditional prompting vs. the Help Me Think strategy.

The "HELP ME THINK" strategy flips the script on traditional prompting. Instead of the user struggling to provide all necessary context upfront, the model is prompted to lead the information-gathering process by asking the user relevant questions. This shifts the cognitive burden from the user to the AI, democratizing the creation of customized content like professional biographies, travel itineraries, and creative stories.

Methodology

The HELP ME THINK framework operates through a structured three-stage interaction:

  1. Stage 1: Question Generation: The user prompts the LLM to adopt an expert persona (e.g., "I am an expert travel planner") and instructs it to generate a list of questions required to complete the specific task.
  2. Stage 2: Information Collection: The user provides brief, straightforward answers to the model's questions.
  3. Stage 3: Final Generation: The model synthesizes the original task, the generated questions, and the user's answers to produce the final output.
The structured prompt template used in the Help Me Think strategy.

This approach leverages the model's internal knowledge of a task's requirements to guide the user, ensuring all necessary "ingredients" are present for a high-quality result.

Experimental Evaluation

The researchers evaluated the strategy across six core customized content generation tasks: biography generation, travel planning, dialogue creation, poetry, event summarization, and story generation. The evaluation involved a dataset of approximately 2,000 question-answer pairs and 180 task-specific outputs generated using GPT-3 (text-davinci-002). The strategy's versatility was further tested on 57 additional diverse tasks, including complex scenarios like bridge construction planning.

Key Results and Insights

Human evaluation revealed several significant advantages of the HELP ME THINK strategy:

  • High Relevance and Validity: 100% of the model-generated questions were found to be valid and relevant. Furthermore, approximately 94% of final outputs were free of irrelevant hallucinations.
  • Robustness and Editing: The model acted as an effective editor, correcting typos or grammatical errors in user answers in 82.65% of cases.
  • Coherence: The strategy achieved a 99% coherence rate, successfully weaving brief user responses into rich, contextually appropriate prose.
  • Knowledge Absorption: The model successfully incorporated provided facts in about 70% of cases. While performance was high for creative tasks like poetry (90%), it faced challenges in highly detailed tasks like travel planning (3.33% for full detail retention), indicating a trade-off between the volume of information and the model's memory.

Conclusion and Future Work

HELP ME THINK demonstrates that LLMs are exceptionally capable of identifying their own information needs. While highly effective for general customization, future work is needed to address the "absorption trade-off" in detail-heavy tasks, perhaps through task decomposition. Additionally, expanding the evaluation to multilingual contexts and newer models like GPT-4 will be crucial for further validating the strategy's generalizability.

Summary updated

Abstract

Controlling the text generated by language models and customizing the content has been a long-standing challenge. Existing prompting techniques proposed in pursuit of providing control are task-specific and lack generality; this provides overwhelming choices for non-expert users to find a suitable method for their task. The effort associated with those techniques, such as in writing examples, explanations, instructions, etc. further limits their adoption among non-expert users. In this paper, we propose a simple prompting strategy HELP ME THINK where we encourage GPT3 to help non-expert users by asking a set of relevant questions and leveraging user answers to execute the task. We demonstrate the efficacy of our technique HELP ME THINK on a variety of tasks. Specifically, we focus on tasks that are hard for average humans and require significant thinking to perform. We hope our work will encourage the development of unconventional ways to harness the power of large language models.
BibTeX
@inproceedings{mishra2023help,
  author = {Swaroop Mishra and Elnaz Nouri},
  title = {{HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=izyh0fbyoQa},
}
Figure