InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy

Vishnu Vinod, Krishna Pillutla, Abhradeep Guha Thakurta

Abstract

As Large Language Model (LLM) applications expand into retrieval-augmented generation (RAG) and long-form reasoning, the secure handling of sensitive information remains a significant challenge. We introduce InvisibleInk, a scalable framework for long-form text generation that provides rigorous differential privacy (DP) guarantees. InvisibleInk optimizes the exponential mechanism over LLM logits through two key innovations: DClip, which isolates and clips only the sensitive delta between private and public logits, and Top-k+ sampling, which improves text quality by restricting sampling to a high-probability token subset. Our evaluations demonstrate an 8x reduction in computational overhead compared to state-of-the-art baselines, achieving high-utility private text generation at less than 10x the cost of non-private methods.

Introduction

The integration of LLMs into specialized domains like medicine and law often requires processing sensitive documents. While paradigms like RAG enhance model utility, they also risk leaking private data through generated outputs. Differential Privacy (DP) provides a rigorous framework to mitigate this risk, but traditional DP text generation methods have historically been impractical due to extreme computational costs (often 100x overhead) and poor output quality (coherence issues).

InvisibleInk addresses these "deal-breakers" by treating token sampling as an iterative application of the exponential mechanism. By focusing privacy-preserving noise only where it is needed most and employing smarter truncation strategies, InvisibleInk enables high-quality, private long-form generation at a fraction of the previous cost.

The InvisibleInk Framework

Targeted Clipping (DClip)

A core insight of InvisibleInk is that the majority of an LLM's output is governed by general linguistic patterns—grammar, syntax, and common knowledge—which are not private. Standard DP methods apply noise to the entire logit distribution. In contrast, DClip isolates the difference between the "private" logits (conditioned on sensitive data) and "public" logits (conditioned only on general context). By clipping only this residual difference, the framework can use a much tighter clip norm, significantly reducing the amount of noise required to satisfy DP guarantees.

Top-k+ Sampling

Standard DP generation typically requires sampling from the entire vocabulary to avoid infinite privacy loss (which occurs if a token's probability is zero in one distribution but non-zero in another). However, sampling from the "long tail" of a 32k+ token vocabulary often leads to incoherent "gibberish."

InvisibleInk introduces Top-k+ sampling, which constructs a tight superset of the most likely private tokens using only public information. This allows the model to utilize truncated decoding—a standard technique in non-private LLMs—to maintain high text quality and coherence without compromising the privacy budget.

Efficient Privacy Accounting

Unlike previous state-of-the-art methods that rely on computationally expensive grid searches to calibrate privacy parameters, InvisibleInk utilizes a non-adaptive accounting procedure based on zero-Concentrated Differential Privacy (zCDP). This allows for off-the-shelf deployment and predictable privacy budgets.

Experimental Evaluation

The framework was evaluated across three sensitive domains: medical discharge summaries (MIMIC-IV-Note), legal case notes (TAB), and user reviews (Yelp).

Efficiency and Utility

InvisibleInk demonstrates a consistent 8–16x reduction in computation cost compared to prior baselines. While previous methods required up to 100x the overhead of non-private generation to produce coherent text, InvisibleInk achieves high utility with less than 10x overhead. On the MIMIC dataset, InvisibleInk achieved MAUVE scores (a metric for human-like text) of 71-74% using only 8-16 inference calls per token. Baselines failed to reach this quality even with 128 calls.

Downstream Performance

The utility of the generated text was further validated through downstream tasks. Synthetic data generated by InvisibleInk was used to train classifiers, significantly outperforming existing methods. For instance, on a 50-class classification task, InvisibleInk achieved 33% accuracy, compared to just 7% for the AdaPMixED baseline.

Discussion and Limitations

While InvisibleInk represents a significant step forward, several considerations remain for practical deployment:

  • Model Access: The framework requires "white-box" access to model logits, meaning it is currently best suited for open-weights models (like Llama 3) or internal deployments by API providers.
  • Trust Model: It assumes a trusted central aggregator with access to the raw sensitive references to perform clipping.
  • Scope of Privacy: The current implementation focuses on example-level DP (protecting individual documents). Future work is needed to extend these guarantees to user-level privacy across multiple documents.

Conclusion

InvisibleInk bridges the gap between theoretical privacy and practical utility in LLM text generation. By recognizing that "language is mostly public" and optimizing the sampling process for coherence, it enables the safe application of LLMs to sensitive datasets without the prohibitive costs previously associated with differential privacy.

Summary updated

Abstract

As major progress in LLM-based long-form text generation enables paradigms such as retrieval-augmented generation (RAG) and inference-time scaling, safely incorporating private information into the generation remains a critical open question. We present InvisibleInk, a highly scalable long-form text generation framework satisfying rigorous differential privacy guarantees with respect to the sensitive references. It interprets sampling from the LLM's next-token-distribution as the exponential mechanism over the LLM logits with two innovations. First, we reduce the privacy cost by isolating and clipping only the sensitive information in the model logits (relative to the public logits). Second, we improve text quality by sampling from a small superset of the top-$k$ private tokens. Empirical evaluations demonstrate a consistent $8\times$ reduction in computation cost over state-of-the-art baselines to generate long-form private text of the same utility across privacy levels. In summary, InvisibleInk is able to generate private long-form text at less than $10\times$ the computation cost of non-private generation.
BibTeX
@inproceedings{vinod2025invisibleink,
  author = {Vishnu Vinod and Krishna Pillutla and Abhradeep Guha Thakurta},
  title = {{InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=B4NT8TexNS},
}
Figure