Promptile: Recovering and Reusing Prompting Strategies from LLM Interaction Histories

Qingyang Wan, Chang Liu, Zhoutong Ye, Chengwen Zhang, Yanru Qian, Zhike Wang, Jie Cai, Chun Yu, Yuanchun ShiarXiv 2026 · arXiv

TL;DR

Promptile is a new framework that automatically mines your past LLM chat histories to extract reusable "prompting strategies"—like specific personas, formatting rules, or instructional constraints. By transforming scattered interaction history into a library of modular, editable fragments, it allows users to curate and assemble high-quality prompts for new tasks with significantly less manual typing and fewer rounds of refinement.

What Problem Does This Paper Address?

As we gain experience with LLMs, we develop "mental recipes" for what works: a specific way to ask for a summary, a persona that gives the best advice, or a set of constraints that prevents the model from hallucinating. However, these insights are usually buried in thousands of lines of old chat logs.

Current personalization efforts (like ChatGPT's "Memory") often focus on what you talked about (facts and preferences) rather than how you talked to the model (strategies). This forces users into a cycle of "repetitive prompting," where they must manually re-type or copy-paste complex instructions across different sessions. The authors aim to bridge this gap by treating prompting experience as a reusable asset.

Methods

The researchers developed Promptile, a system that operates in three main stages:

  1. Extraction & Taxonomy: Based on a formative study of 1,445 real-world prompts, the team created a taxonomy to categorize prompt fragments into four dimensions: Context (domain/topic), Instruction (tasks/personas), Supporting Info (examples/background), and Output (format/style).
  2. The Pipeline: As shown in Figure 1, the system normalizes chat history and uses LLM agents to "mine" valid, effective fragments. When a user starts a new task, Promptile uses vector similarity (cosine similarity of embeddings) to retrieve the top 15 most relevant fragments from the user's history.
  3. Interactive Assembly: Instead of "automagically" writing the prompt, Promptile provides a UI (Figure 2) where users can drag and drop these fragments into their input box. A "Polish" button then uses an LLM to weave these fragments into a coherent, natural-language prompt.
Figure 1: The Pipeline of Promptile System

Datasets

The authors evaluated Promptile using two primary sources:

  • WildChat: A public dataset of 1 million real-world ChatGPT interaction logs used to test the system's ability to recover strategies from longitudinal histories.
  • User-Specific Histories: For the user study, 16 participants exported their own recent ChatGPT histories (averaging 192 prompts per person) to serve as the "source material" for their personalized version of Promptile.

Key Insights and Ideas

  • From Writing to Curating: The most profound shift is moving the user's role from writing prompts from scratch to curating existing strategies. Users spent more time in the input box selecting and editing fragments, but this resulted in much lower frustration because they weren't stuck in a "back-and-forth" loop of fixing bad model outputs.
  • Performance-Driven Evolution: The study found that users' best strategies aren't static; they evolve. A user might start with "be simple" and eventually refine it to "explain like a PI instructing a grad student." Promptile captures this refined version as the "proven" strategy.
  • Malleable Memory: Participants valued retrieved fragments even when they didn't use them verbatim. The fragments served as "externalized memory," reminding users of requirements they might have otherwise forgotten to include.

Key Results

In a controlled study comparing Promptile to standard ChatGPT with global memory enabled:

  • Reduced Effort: Manual typing was cut nearly in half, from an average of 91.6 characters to 51.4 characters per task.
  • Higher Quality, Faster: Final prompts were longer and more detailed, yet required fewer interaction turns to reach a satisfactory result (1.88 turns vs. 2.69 turns).
  • Subjective Success: As seen in Figure 5, Promptile scored significantly higher in perceived performance and capability, while significantly lowering user frustration.
Figure 5: Results of the comprehensive subjective evaluation (N=16). The chart compares Promptile and the Baseline across NASA-TLX workload dimensions, system usability, and study-specific user experience metrics on a 7-point Likert scale. Note that standard NASA-TLX scales are reversed here: higher scores correspond to lower workload. Asterisks denote statistical significance (* p < 0.05, ** p < 0.01).

Limitations and Future Work

The authors noted that the system struggles with "noisy" histories—prompts filled with typos, fragmented phrases, or very minimal wording make it difficult for the mining agent to identify a clear strategy. Additionally, the boundary between an explicit instruction (which the system is told to extract) and an implicit preference inferred from context can be blurry, sometimes leading the system to "hallucinate" a strategy that wasn't strictly there.

Future work includes handling these ambiguous cases and exploring how prompting strategies might be shared or transferred between different users in collaborative environments.

Summary updated

Abstract

As people use large language models (LLMs) over time, they develop their own ways of prompting to achieve desired results. These useful experiences often remain scattered and unorganized across conversations. Yet existing personalization approaches primarily capture user preferences or explicitly defined workflows. Through a formative study, we found that users develop prompting strategies that range from implicit instructions to explicit templates from these experiences. We further find that interaction behaviors provide evidence for these strategies, which can be decomposed and recombined for reuse across tasks. Building on these findings, we present Promptile, a framework that identifies and validates users' prompting strategies from interaction histories. Promptile organizes these strategies into reusable components that can be retrieved, combined, and adapted to users' current tasks. Our evaluation shows that Promptile accurately recovers and reuses these strategies, better aligning model outputs with users' expectations while reducing repetitive prompting effort.
BibTeX
@misc{wan2026promptile,
  author = {Qingyang Wan and Chang Liu and Zhoutong Ye and Chengwen Zhang and Yanru Qian and Zhike Wang and Jie Cai and Chun Yu and Yuanchun Shi},
  title = {{Promptile: Recovering and Reusing Prompting Strategies from LLM Interaction Histories}},
  year = {2026},
  eprint = {2610.26485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26485},
}
Figure