When a clinician or lawyer asks a language model for help, the details that make the question answerable are often the same details they are nervous to share: a patient's age and HIV status, the amount in a private loan dispute, the name of an employer. Privacy research has found a "transparency paradox." Explaining how a system handles your data often makes people share less, not trust more. Research on large language models (LLMs) has found a matching weakness from the other side. These models already struggle in professional domains and need complete, specific case facts to have any chance. Put the two findings together and a concrete risk appears. A privacy warning attached to an expert LLM assistant could lead professionals to leave out or falsify exactly the facts the model needs, making the tool less accurate while protecting nothing that mattered.
We propose to change what the disclosure says. Current tools describe risk ("this text contains PII; the provider may retain it"). Our assistant would instead estimate, before asking, how much a specific sensitive detail changes the model's answer on this kind of case. It would request only details whose estimated benefit is statistically distinguishable from zero, and it would tell the user that benefit in plain terms. The core claim is testable: on professional tasks, utility-grounded disclosure recovers the accuracy that risk warnings cost, and it also reduces the sharing of sensitive details that do not help. Risk warnings alone cannot achieve both.
What Exists and What Is Missing
The closest system is CLEAR. It detects PII in user prompts with a pattern-matching recognizer, pulls the relevant privacy-policy passages, and generates concrete risk explanations. In its ChatGPT study, 12 of 13 participants said they would delete or fake sensitive data after seeing its alerts. CLEAR treats that change in behavior as success. It never asks whether the withheld data was needed, and it studied lay users on everyday tasks. Our proposal takes CLEAR as the strongest risk-based comparison and measures the cost it leaves out: task accuracy after users self-censor.
The mechanism comes from Participatory Personalization in Classification and Participatory Systems for Personalized Prediction. These papers let individuals choose which group attributes to report to a clinical classifier. A hypothesis test prunes any request that gives no significant accuracy gain, and every user is guaranteed at least the accuracy of a generic model. On cardio_eicu, their sequential system reduced error by 11.3% relative to a one-hot model while using 16.7% less data. That work covers tabular classifiers, fixed attribute sets, and users assumed to act rationally. We carry its gain-then-prune logic over to a generative model reading free-text case descriptions, and we test the rationality assumption with real people. It could fail if seeing any sensitive request at all triggers sensitization.
Three findings shape how the disclosure should be worded. Explaining mechanisms tends to backfire: justifications for mouse tracking lowered participation below giving no explanation at all (Privacy Attitudes toward Mouse-Tracking Paradata Collection), and better understanding of differential privacy reduced willingness to share (The influence of explanation designs...). Describing consequences works better, as shown by outcome-framed differential privacy explanations in Models Matter and by concrete inference visualizations in The Curious Case of the PDF Converter that Likes Mozart. Our disclosure is a consequence statement aimed at the benefit side ("including the patient's age changed the model's recommendation in 31% of similar validation cases and raised accuracy by X points"). Those papers framed consequences around risk. A Transparency Paradox? is a warning for us. Specific explanations that exposed harmless errors raised anxiety even when outcomes were safe. Utility numbers could expose the model's imperfection in the same way, so we measure trust and anxiety directly rather than assuming the numbers reassure.
Core Claim
For a case with a set of candidate sensitive details, we define the marginal utility of detail $a$ for case type $g$ as
$$\Delta_g(a) = \mathrm{Acc}_g(\text{with } a) - \mathrm{Acc}_g(\text{without } a),$$
where $\mathrm{Acc}_g$ is the model's accuracy on labeled validation cases of type $g$. The assistant requests $a$ only if a one-sided test rejects $\Delta_g(a) \le 0$, and when it does, it shows the estimate. The claim is about the joint effect of pruning and truthful benefit framing. It should beat three things: risk warnings on end-task accuracy, silent redaction on accuracy, and a request for every flagged detail on how many unnecessary sensitive details are shared.
Research Program
The first part is offline and needs no users. We build text versions of the participatory clinical benchmarks (apnea, cardio_eicu, cardio_mimic, saps, coloncancer, lungcancer) by serializing each record into a short case vignette. The group attributes those papers treat as sensitive become removable phrases. We use MIMIC-Extract where ICU time series need standardizing. For law, we use AppealCase first-instance reversal prediction. There, sensitive spans are party identities, amounts, employer and relationship details, and "new information" annotations. For pharmacology we use NLPEC (Towards Medical Machine Reading Comprehension...), with patient attributes inside the question stems. On each benchmark we estimate $\Delta_g(a)$ with an open model and a frontier model, apply the pruning test with multiple-comparison correction, and record which requests survive.
Free text has no fixed attribute list, so a detector must propose candidate details. We compare the Presidio-style recognizer that CLEAR uses with an LLM tagger prompted for domain-sensitive, non-PII content, such as symptoms or legal claims. Case type $g$ is defined by coarse, non-sensitive features, so estimating it requires no disclosure.
The second part is a between-subjects user study with practitioners-in-training: pharmacy or medical students on NLPEC and clinical vignettes, and law students on AppealCase-derived cases. Each vignette arrives complete. The participant decides what to pass to the assistant, may edit or fake details, sees the answer, and submits a final decision. We log exactly what was withheld or falsified. We also compute the counterfactual accuracy the model would have had with the full vignette, which lets us split lost accuracy into model error and self-censorship.
Baselines
The do-nothing reference is the assistant with no disclosure interface. Users type whatever they choose and see no warning. This shows natural disclosure and accuracy, and it is the condition that risk warnings are meant to improve.
The strongest existing method is CLEAR (CLEAR). We reimplement its three stages: PII detection, contextual policy extraction, and risk generation grounded in a privacy knowledge base. It uses the same underlying LLM, policy text, and vignettes as our system. It is the strongest competitor because it is the only published just-in-time LLM disclosure tool with evidence that it changes behavior. The only reported result is behavioral intention, 12 of 13 participants. No reported result on Lacuna measures CLEAR's effect on downstream task accuracy. We count our method as beating it if final-decision accuracy is significantly higher and the number of disclosed zero-gain sensitive details is no higher. Winning on accuracy alone by encouraging users to share everything does not count.
For the offline accuracy side, we add the participatory tabular systems as run in their papers. The reported reference is the 11.3% error reduction with 16.7% less data on cardio_eicu. We rerun them on the same splits so the LLM version can be compared on both accuracy and data minimization. On AppealCase, the reported reference is that all 20 tested models scored below 50% F1 on first-instance reversal prediction. Our aim there is not a higher score. It is to avoid falling further below the full-information model after pruning.
There are two ablations that isolate our key ingredients. Prune-only drops zero-gain requests without showing benefit numbers. Utility-without-pruning shows benefit estimates for every flagged detail. A third ablation replaces the estimated $\Delta_g(a)$ with the LLM's own judgment of whether a detail is relevant, which tests whether measured utility is needed at all. All conditions use the same models, vignettes, participant pool, and prompt-tuning budget, fixed on a separate pilot set.
Why not just…?
Why not just redact every sensitive detail automatically and never ask? Redaction removes the paradox by removing the choice, but it quietly discards information. The participatory papers found that when withheld attributes are imputed, some subgroups do worse than with a generic model. This includes a gain of -24 for one subgroup under naive personalization. A redacted LLM could fail in the same way on details like age that matter clinically. Offline, we measure the gap between accuracy under full redaction and under full information for each benchmark. If that gap is negligible everywhere, redaction wins and our interface is unnecessary. We treat that as a genuine possible outcome.
Why not just keep CLEAR-style warnings but make them clearer? The literature points the other way. Clearer explanations of mechanisms tend to reduce sharing (The influence of explanation designs...; Privacy Attitudes toward Mouse-Tracking Paradata Collection), and honest short notices penalized transparent vendors (User Choices and Regret). A warning that is only about risk gives users no reason to share a detail that helps. The CLEAR-versus-ours comparison tests this directly.
Why not just let the LLM ask clarifying questions about what it needs? This is the most serious threat. A capable model might request the right details without any held-out estimate. However, model self-assessment is unreliable in exactly these domains. Adding the needed theorems to the prompt did not help TheoremQA (TheoremQA), which suggests models misjudge which information they can actually use. The self-judged-relevance ablation settles the question. If its requests match the tested set's accuracy and disclosure counts, the estimation machinery adds nothing, and we would recommend the simpler design.
Proposed Experiments and Decisive Checks
The first decisive check is offline. Does the estimated $\Delta_g(a)$ predict the accuracy change on held-out cases, measured by correlation and by calibration across case types? And does pruning keep accuracy within a small margin of the full-information model while requesting fewer sensitive details? If held-out utility estimates are noise, the user study is not worth running.
The second check is the user study's primary comparison. Final-decision accuracy and zero-gain disclosures are measured jointly for each condition, with accuracy loss split into model error and self-censorship using the full-vignette counterfactual. Secondary outcomes are falsification rate, reliance on correct versus incorrect answers, and anxiety and trust on short standard scales. These tell us whether benefit numbers themselves produce the "safe but scared" effect seen in A Transparency Paradox?.
The third check is a short longitudinal arm of about two weeks of repeated sessions. It asks whether users habituate to the utility disclosure or keep responding to it. Neither CLEAR nor the participatory work addresses this.
The proposal is falsified in any of three cases: CLEAR matches our accuracy, redaction costs nothing, or self-judged relevance matches measured utility. Each outcome would itself be a useful result for designing privacy interfaces for expert LLMs.
References
- CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applications
- Participatory Personalization in Classification
- Participatory Systems for Personalized Prediction
- Privacy Attitudes toward Mouse-Tracking Paradata Collection
- The influence of explanation designs on user understanding differential privacy and making data-sharing decision
- Models Matter: Setting Accurate Privacy Expectations for Local and Central Differential Privacy
- The Curious Case of the PDF Converter that Likes Mozart: Dissecting and Mitigating the Privacy Risk of Personal Cloud Apps
- A Transparency Paradox? Investigating the Impact of Explanation Specificity and Autonomous Vehicle Perceptual Inaccuracies on Passengers
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III
- AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
- User Choices and Regret: Understanding Users' Decision Process about Consensually Acquired Spyware
- TheoremQA: A Theorem-driven Question Answering Dataset