Language-model agents that read a user's email, calendar, or notes routinely pass private details to people who should not see them, even when the same models can correctly explain the relevant privacy norms. The best current fix teaches a model to reason explicitly about who is sharing what with whom before it acts. That training rests on a few hundred synthetic scenarios set in ordinary present-day settings. This note proposes a different source of training scenarios. We would generate design fictions, meaning short future-dated stories that place a technology inside everyday life and then show what goes wrong. Each fiction would be built from a real, retrieved AI failure report and would deliberately stage a "crisis of routine": an illness, a visitor, or a custody dispute that breaks the system's assumptions.
The bet is that privacy failures in deployment come from disrupted, multi-party situations that present-day synthetic vignettes rarely contain. If that is right, a small set of incident-anchored, disruption-centred fictions should teach structured privacy reasoning that transfers better to existing agent benchmarks than the same number of generic synthetic examples. The comparison is a controlled one: we keep the model, the reward, and the number of examples fixed, and vary only where the training scenarios come from.
Core Claim
Training-data content, not only the reasoning format, limits how well structured privacy reasoning generalizes. Holding the reasoning template and reinforcement-learning recipe of Contextual Integrity in LLMs via Reasoning and Reinforcement Learning fixed, we predict that training on 729 fictions grounded in incidents and organized around crises of routine will lower leakage on PrivacyLens and ConfAIde. We predict a larger reduction than training on the original 729 synthetic examples. We also predict a larger reduction than training on 729 fictions that share the same crisis structure but are generated without retrieval, and that this will hold without a loss in helpfulness. If the ungrounded fictions match the grounded ones, retrieval contributes nothing and the claim reduces to "disruption-centred scenarios help." That outcome would still be useful, but it is a weaker claim, and the experiments are designed to tell the two apart.
How this relates to prior work
Three bodies of work motivate the proposal.
Design fiction. Design-fiction research shows that structured speculative scenarios bring out failure modes that abstract questions miss, but every study in the file writes those scenarios by hand. Learning Systems versus Future Everyday Domestic Life contrasts "happy path" and "crisis of routine" narratives from three viewpoints: the inhabitant, the learning system, and the designer. It does this by hand for three domestic practices. Particip-AI collects speculative use cases and harms from lay survey respondents. Studying Up Public Sector AI uses hypothetical projects in interviews to get past institutional defensiveness. We borrow the crisis-of-routine and three-perspective structure as a generation template. This automates something those studies did manually, and we use the output as training data rather than as a discussion prompt.
Privacy benchmarks. Two benchmarks show the failure we target. Can LLMs Keep a Secret? introduces ConfAIde and finds that unstructured "think step by step" prompting can increase leakage: models note that a fact is confidential and then disclose it anyway. PrivacyLens finds that models answer privacy-norm questions almost perfectly but leak during tool use. These results constrain the design. Free-form reasoning is a known hazard, so we keep the structured reasoning template fixed and change only the data.
Structured reasoning training. Contextual Integrity in LLMs via Reasoning and Reinforcement Learning supplies the method. It trains with a reasoning template, Contextual Integrity chain-of-thought (CI-CoT), that labels each piece of data as necessary, helpful, or inappropriate. It optimizes with Group Relative Policy Optimization (GRPO), a reinforcement-learning method that scores several sampled answers against each other, using rule-based rewards. It trains on the ContextualIntegritySyntheticDataset. That paper leaves open where training scenarios should come from, and that is our variable.
Incident archive. The AI Incident Database supplies the retrieval corpus. It is a curated archive of more than 1,000 real AI failure reports that its creators built as collective memory. It has never been used as a retrieval store for generating scenarios.
Research Program
The generator works in three stages.
Retrieval. We filter the AI Incident Database to incidents involving personal data, surveillance, or unwanted disclosure. For each target domain, we retrieve one to three incidents using hybrid lexical and dense search with reranking, following the pattern in Boros et al..
Generation. A large model, prompted with a fixed structured template, writes a future-dated fiction. In order, the template asks it to:
- name the system and its data flows as contextual-integrity five-tuples (data type, subject, sender, recipient, transmission principle), in the same form as PrivacyLens seeds;
- write the happy path;
- introduce a crisis of routine that reproduces the retrieved incident's mechanism in a new setting;
- describe the crisis from the inhabitant, learning-system, and designer viewpoints;
- emit an agent task with gold labels marking which data items are necessary and which are inappropriate to disclose.
That final output format matches the CI-RL training format exactly, so the reward function does not change.
Filtering and checks. Before training, we:
- drop any fiction whose five-tuples or surface text overlap PrivacyLens seeds or ConfAIde vignettes above a similarity threshold;
- check each fiction's gold labels with a second model and a human sample;
- have practitioners rate plausibility on a held-out sample. Plausibility matters here because ungrounded speculation tends to drift into science-fiction clichés.
Analysis. Beyond headline leakage, we will tag every PrivacyLens and ConfAIde item by whether it involves a disruption, a third party, or a change in role. We will then test whether any gains concentrate on those items, which is what the claim predicts. A uniform gain across all items would instead suggest a generic data-diversity effect.
Baselines
Naive reference. Qwen2.5-7B with direct prompting, plus a version given an explicit privacy-preserving system prompt. For context, PrivacyLens reports that privacy-enhancing prompts still leave leakage above 25%, with GPT-4 at a 25.68% leakage rate (PrivacyLens). On ConfAIde Tier 4, GPT-4 leaks 39% of the time under explicit instructions (Can LLMs Keep a Secret?). We also run unstructured chain-of-thought prompting, because ConfAIde reports that it can make leakage worse.
Strongest existing method. CI-RL trained on the original 729-example dataset with the authors' recipe, which reports up to a 40% reduction in PrivacyLens leakage relative to the base model and a 7B model surpassing a 14B baseline (Contextual Integrity in LLMs via Reasoning and Reinforcement Learning). There is no reported result found on Lacuna for CI-RL's exact ConfAIde scores, so we will measure those ourselves. We reproduce CI-RL before comparing. Every trained condition uses:
- the same base model;
- the same 729-example budget;
- the same reward;
- the same number of GRPO steps;
- the same hyperparameter sweep, of equal size per condition;
- three seeds.
Beating CI-RL. Our method beats CI-RL if it shows lower PrivacyLens leakage at equal or better helpfulness, and the difference holds across seeds with non-overlapping bootstrap intervals. Helpfulness is measured by PrivacyLens's helpfulness rating together with the rate of required information included. The helpfulness check matters because PrivacyLens notes that Llama-3-8B gets low leakage by refusing to share even non-sensitive information.
Ablations isolating the key ingredients. Two ablations separate the contributions of retrieval and of the crisis structure. In the first, the same template generates 729 fictions without retrieved incidents, which isolates grounding. In the second, retrieved incidents are kept but the generator writes ordinary present-day happy-path vignettes, which isolates the crisis-of-routine structure.
Why not just…?
Why not just generate more ordinary synthetic data? This is the most serious threat. CI-RL's gains may be limited by data quantity rather than content, so we run a scaled condition with 2,916 original-style synthetic examples, four times the budget. If four times as many generic examples match 729 fictions, the fictions' advantage is only efficiency. We would report that result rather than claim better generalization.
Why not just prompt the model to imagine edge cases, without retrieval? That is exactly the ungrounded-fiction ablation. Retrieval earns its place only if grounded fictions beat ungrounded ones on the benchmarks, or if they are clearly more plausible and more diverse under human rating. If neither holds, we drop retrieval from the method.
Why not just retrieve incidents or norms at inference time instead of training on them? Retrieval at inference time is a natural rival. We run CI-CoT prompting and the CI-RL model with retrieved incident summaries placed in the agent's context. Two findings give reason to doubt this will be enough. PrivacyLens shows that extra context in tool-using trajectories tends to trigger leakage. ConfAIde shows that awareness stated in the model's reasoning does not reliably control what it does. Even so, the experiment settles the question, and a positive result there would be cheaper and preferable.
Proposed Experiments
Primary evaluation. Leakage rate and helpfulness on the full PrivacyLens agent trajectories, plus ConfAIde Tiers 3 and 4 leakage, for every baseline and ablation above. Neither benchmark was used to design the generator, and overlap filtering is enforced.
Secondary evaluation. Transfer from text to tool calling, following CI-RL's own generalization test. We also run a held-out set of 200 incident-grounded fictions drawn from incidents excluded from training. We treat that set only as a diagnostic, because it shares the generator's biases.
Decisive checks. The proposal succeeds only if all three of the following hold:
- grounded crisis fictions beat CI-RL-original at equal helpfulness;
- they beat both ablations;
- they match or beat the scaled-synthetic condition.
The tagged-subset analysis should also show the gain concentrated on items involving disruption, third parties, or role changes. A gain only on our own held-out fictions, with none on PrivacyLens or ConfAIde, would count as a failure.
References
- Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
- Learning Systems versus Future Everyday Domestic Life: A Designer's Interpretation of Social Practice Imaginaries
- Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits
- Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
- PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action
- Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database
- Fine-Tuning and Retrieval Augmented Generation for Question Answering Using Affordable Large Language Models