AI agents that run inside pinned software containers increasingly ship with full execution logs. These logs record every file opened, every database row queried and every command run, so that a third party can check the agent did not fake its result. That auditability is exactly what makes the logs dangerous on private data. Work on selection leakage shows that the bare record of which items an algorithm chose to look at, with no contents attached, is enough to infer which private records exist. Indices of curated samples alone support membership inference at near-perfect accuracy. We propose to measure this side channel in agent audit logs for the first time, and then to release a different kind of log. The base container image is public and reproducible, so its content hash can tell us exactly which accesses touched public software and which touched private task data. Public-layer events would be released verbatim. Private-layer events would be released only as noisy, record-capped counts with a differential privacy guarantee (a formal bound on how much any single record can change the released output).
The central empirical question is whether a third-party verifier reading this split log still catches the failures that make audit logs worth keeping: overwritten test files, fabricated tool outputs, silently degraded simulations. It must do so while the attacker's ability to tell whether a given private record exists falls to near chance. If the answer is yes, reproducibility and selection privacy stop being in direct conflict. If the answer is no, we will have mapped where they are.
Core Claim
Three claims, each testable. First, logs that are redacted in the current way still leak private-record membership through access patterns. "Redacted in the current way" means literal secrets and paths are replaced by consistent placeholders. Second, most of what an auditor needs lives in the public image layers and the evaluation harness, not in the identity of the private records read. Third, splitting the log along the image boundary and adding calibrated noise only to private-layer access counts gives a better trade-off between privacy and audit fidelity than three alternatives: syntactic redaction, outcome-only evaluation, or releasing nothing but a local verdict.
Prior Work and Positioning
The motivating evidence comes from selection leakage outside agents. Curation Leaks shows that when private data is used only to choose public samples, the chosen indices reveal private membership at roughly 98% true positive rate at 1% false positive rate. The same paper notes a "privacy onion": removing the most exposed records simply exposes the next layer. CoLa adds selection-participation inference: a record can be detected merely because it sat in the candidate pool, even if it was rejected. That is the agent analogue of a record the agent listed but never opened. Query Complexity of Bayesian Private Learning and Private Sequential Learning prove that adaptive query locations alone narrow down a hidden target, which is the formal reason a grep-then-open trajectory should leak. None of these studies tool-using agents or execution logs.
Two papers constrain the defense design. Differentially Private Active Learning finds that noising the selection itself roughly halves the gains of smart selection and collapses to random selection below $\epsilon \approx 3$. We therefore do not noise the agent's choices. We noise only the released record of them, so task success is untouched by construction. Curation Leaks needs large budgets (attack success falls from 98.4% to 5.4% only at $\epsilon=100$) when the curation output itself is privatized. Our release is coarser than that output, which is why we expect a better regime, but the experiments must show it.
The closest agent-side work defines what we do differently. SlotGuard replaces paths and credentials in transcripts with consistent, format-preserving placeholders, cutting credential leakage from 23.5% to 0.0% while keeping task success. Its authors explicitly leave out inference from execution patterns, and consistent placeholders preserve exactly the access structure we attack. SciAgent's audit harness records append-only logs and uses a verifier with no access to the agent's reasoning. It shows that a parent-only view returns INSUFFICIENT while recursive access verifies, and it treats the log as fully trusted, including on a BRCA1 variant task where the inputs could be patient data. An Auditable Harness for AI-Assisted Lean Formalization withheld transcripts because of paths and tool traces and names sanitization as future work. Agent-Diff supplies our testbed and the idea of evaluating by before/after state snapshots, but it does not consider that the snapshot difference itself is a selection signal. Establishing Best Practices for Building Rigorous Agentic Benchmarks supplies the failure cases any privacy defense must not hide: test-file overwriting in SWE-Lancer, and a do-nothing agent passing 38% of impossible τ-bench tasks.
Research Program
Threat model. The adversary sees the published artifact: the log, the verdict and the state difference. The adversary also knows the public base image and a candidate universe of possible private records, for example the plausible customers in a CRM replica or the plausible module names in a proprietary repository. The adversary wants to know whether a target record is in the private data the agent worked on. We assume the model runs locally (open-weight models) or that the model provider is trusted. Leakage to remote providers is a separate problem that SlotGuard partly addresses and that we do not claim to solve.
The leakage measurement. We build matched pairs of containers that differ in exactly one private record. This is the standard "neighboring datasets" setup of differential privacy. We run the same agent and seed in both containers and train attackers on the resulting logs. Some attackers use hand-built features, such as whether the placeholder for a record's slot is opened, listed or grepped. Others are shadow-run attackers in the style of the likelihood-ratio attack adapted in Curation Leaks. We report both membership inference (the record was read) and participation inference (the record was present but never opened), following CoLa.
The release mechanism. We compare the running container against the content hashes of the pinned image layers. Every filesystem, database and tool event is labeled public if it touches only image-layer content or the evaluation harness, and private otherwise. Public events, which include edits to test files, toolchain invocations and solver parameters read from config, are released verbatim with SlotGuard-style placeholders. Private events are clipped so that each record contributes at most $c$ events to the released counts. They are then aggregated into counts $h_b$ over coarse buckets $b$ (a directory, a table, or a hashed record bucket) and released with Laplace noise:
$$\tilde h_b = \min(h_b^{\text{clipped}}, \cdot) + \text{Lap}(c/\epsilon),$$
where $\epsilon$ is the per-trajectory privacy budget. Because one record changes the total count vector by at most $c$, the release satisfies $\epsilon$-differential privacy with respect to that record. The guarantee covers the released log, not the agent. Across $T$ released trajectories on the same private data, budgets add up, so $\epsilon_{\text{total}} = T\epsilon$ in the simple accounting.
The auditability measurement. We inject known faults into otherwise successful runs and check whether a verifier reading each released form detects them. The faults are test-file overwrites, fabricated tool outputs after a failed call, silent simulation coarsening, and doing nothing on impossible tasks. The verifier is a SciAgent-style verifier with no access to the agent's reasoning.
Baselines
The naive reference point releases the full, unredacted recursive log. This is what SciAgent assumes, and it fixes the ceiling on audit fidelity and the floor on privacy.
The strongest existing defense is SlotGuard applied to the same full log. It is the strongest competitor because it is the only deployed method that keeps logs usable while removing secrets. Its reported results are 0.0% credential leakage across 852 planted values versus 23.5% for the baseline, and it retains 100% of raw task success on Kimi and Llama, against 2.5% success for generic redaction. It has never been evaluated on selection or membership leakage, and we found no reported result on Lacuna for selection leakage from agent audit logs under any defense. We run it on identical trajectories with its default slot types. It beats us if its log yields equal fault detection with attack TPR at 1% FPR within noise of ours.
The outcome-only baseline releases only the final state difference and test results, in the style of Agent-Diff.
The local-verdict baseline runs the verifier with full recursive access inside the trusted container and publishes only VERIFIED, REFUTED or INSUFFICIENT.
Two ablations isolate our ingredients. The first is the image split without noise: public events are verbatim and private counts are exact. It tests whether the split alone stops leakage. The second applies noise to every event without the split. It tests whether the image boundary is what preserves auditability.
All conditions share the same trajectories, the same verifier model and the same prompt. The attacker gets the same shadow-run budget against every release, so the comparison measures the release format and nothing else. We count it as a win if, at some $\epsilon \le 3$, the split release reaches fault-detection recall within 5 points of the full log while pushing attack TPR at 1% FPR to near 1%. On both axes it must also beat SlotGuard, outcome-only, and the noisy-everything ablation.
Why not just…?
Why not just use SlotGuard-style redaction? Consistent placeholders are designed to preserve path geometry and cross-turn identity, which is exactly the structure a selection attacker exploits. Curation Leaks shows that indices with no content attached are enough for near-perfect membership inference. That said, it is untested whether agent access patterns are as informative as hard nearest-neighbor curation. The leakage measurement settles this. If attacks on SlotGuard logs sit near chance, the defense half of this proposal is unnecessary and we report that.
Why not just evaluate outcomes and discard the trajectory? SciAgent shows outcome-equivalent runs (both 1.000 on BRCA1) that only trajectory auditing distinguished. Establishing Best Practices documents test overwriting and a 38% do-nothing pass rate. Our fault-injection suite measures the outcome-only detection rate directly. We also measure leakage from the state difference itself, because a difference that touches private records is not leakage-free either.
Why not just verify locally and publish only the verdict? This is the real threat. It leaks roughly one adaptively chosen bit per run and catches every fault that a full-access verifier catches. It fails only when someone other than the data owner must re-audit: benchmark maintainers, reviewers, or the withheld-transcript case in the Lean harness. We will state this plainly. If the community's use case is owner-side verification only, the verdict release wins, and our contribution reduces to the leakage measurement plus a release for third-party audit. We also test whether third-party verifiers given our split log agree with the local verdict. If they do not, the published log adds nothing over the verdict.
Why not just noise the agent's selections, as in DP active learning? Differentially Private Active Learning shows that this halves selection gains and becomes random below $\epsilon \approx 3$. SWE-Bench Pro's drop from 25.9% to 8.4% when requirements are removed suggests agents depend heavily on directed search. Noising the release instead leaves task success unchanged by construction.
Proposed Experiments
The primary testbed is Agent-Diff: 224 tasks over containerized Box, Linear, Slack and Calendar replicas, with reported scores of 38.0 to 88.1 across nine models. We populate the replicas with synthetic sensitive records, for example a confidential customer or HR list, and generate one-record-different container pairs. Agents are open-weight models run locally, such as DeepSeek-v3.2 and Devstral-2512, which Agent-Diff reports as its top performers. The second testbed is code: a subset of SWE-Bench Pro in the style of its commercial split. There the private record is a held-out file or module that is present or absent in the repository, and the attacker infers its existence from access patterns. Fault detection additionally uses Terminal-Bench tasks under Harbor and the three SciAgent reproduction tasks with injected silent degradation. Those three tasks are small, so they serve as case studies rather than statistics.
We report three metrics. Privacy is attack TPR at 1% FPR and AUC for both membership and participation inference. Auditability is fault-detection recall and precision, plus agreement with the full-access verdict. Task success is unchanged by design, which we confirm.
The first decisive check is whether attacks on SlotGuard-redacted Agent-Diff logs exceed chance substantially. If they do not, the problem is weaker than the selection-leakage literature suggests. The second is whether the no-noise image split already removes most leakage. If it does, the image boundary is the contribution and differential privacy is optional. The third is whether, at $\epsilon \le 3$, a third-party verifier reading the split log still flags test overwrites and fabricated outputs at nearly full-log rates. Overwrites live in public layers and should survive. Fabrication after a failed private-data read is the hard case, and that is where we expect the trade-off curve to bend.
References
- Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learning
- CoLa: A Choice Leakage Attack Framework To Expose Privacy Risks In Subset Training
- Query Complexity of Bayesian Private Learning
- Private Sequential Learning
- Differentially Private Active Learning: Balancing Effective Data Selection and Privacy
- SlotGuard: Stop Oversharing Private Local Context in LLM Agent Transcripts
- Audit-Grade Harness for Agent-driven Scientific Computation Workflows
- An Auditable Harness for AI-Assisted Lean Formalization: Pilot Logs from Finite Differential Privacy
- Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces