TL;DR
Promptile is a new framework that automatically mines your past LLM chat histories to extract reusable "prompting strategies"—like specific personas, formatting rules, or instructional constraints. By transforming scattered interaction history into a library of modular, editable fragments, it allows users to curate and assemble high-quality prompts for new tasks with significantly less manual typing and fewer rounds of refinement.
What Problem Does This Paper Address?
As we gain experience with LLMs, we develop "mental recipes" for what works: a specific way to ask for a summary, a persona that gives the best advice, or a set of constraints that prevents the model from hallucinating. However, these insights are usually buried in thousands of lines of old chat logs.
Current personalization efforts (like ChatGPT's "Memory") often focus on what you talked about (facts and preferences) rather than how you talked to the model (strategies). This forces users into a cycle of "repetitive prompting," where they must manually re-type or copy-paste complex instructions across different sessions. The authors aim to bridge this gap by treating prompting experience as a reusable asset.
Methods
The researchers developed Promptile, a system that operates in three main stages:
- Extraction & Taxonomy: Based on a formative study of 1,445 real-world prompts, the team created a taxonomy to categorize prompt fragments into four dimensions: Context (domain/topic), Instruction (tasks/personas), Supporting Info (examples/background), and Output (format/style).
- The Pipeline: As shown in Figure 1, the system normalizes chat history and uses LLM agents to "mine" valid, effective fragments. When a user starts a new task, Promptile uses vector similarity (cosine similarity of embeddings) to retrieve the top 15 most relevant fragments from the user's history.
- Interactive Assembly: Instead of "automagically" writing the prompt, Promptile provides a UI (Figure 2) where users can drag and drop these fragments into their input box. A "Polish" button then uses an LLM to weave these fragments into a coherent, natural-language prompt.
Datasets
The authors evaluated Promptile using two primary sources:
- WildChat: A public dataset of 1 million real-world ChatGPT interaction logs used to test the system's ability to recover strategies from longitudinal histories.
- User-Specific Histories: For the user study, 16 participants exported their own recent ChatGPT histories (averaging 192 prompts per person) to serve as the "source material" for their personalized version of Promptile.
Key Insights and Ideas
- From Writing to Curating: The most profound shift is moving the user's role from writing prompts from scratch to curating existing strategies. Users spent more time in the input box selecting and editing fragments, but this resulted in much lower frustration because they weren't stuck in a "back-and-forth" loop of fixing bad model outputs.
- Performance-Driven Evolution: The study found that users' best strategies aren't static; they evolve. A user might start with "be simple" and eventually refine it to "explain like a PI instructing a grad student." Promptile captures this refined version as the "proven" strategy.
- Malleable Memory: Participants valued retrieved fragments even when they didn't use them verbatim. The fragments served as "externalized memory," reminding users of requirements they might have otherwise forgotten to include.
Key Results
In a controlled study comparing Promptile to standard ChatGPT with global memory enabled:
- Reduced Effort: Manual typing was cut nearly in half, from an average of 91.6 characters to 51.4 characters per task.
- Higher Quality, Faster: Final prompts were longer and more detailed, yet required fewer interaction turns to reach a satisfactory result (1.88 turns vs. 2.69 turns).
- Subjective Success: As seen in Figure 5, Promptile scored significantly higher in perceived performance and capability, while significantly lowering user frustration.
Limitations and Future Work
The authors noted that the system struggles with "noisy" histories—prompts filled with typos, fragmented phrases, or very minimal wording make it difficult for the mining agent to identify a clear strategy. Additionally, the boundary between an explicit instruction (which the system is told to extract) and an implicit preference inferred from context can be blurry, sometimes leading the system to "hallucinate" a strategy that wasn't strictly there.
Future work includes handling these ambiguous cases and exploring how prompting strategies might be shared or transferred between different users in collaborative environments.
Summary updated