When GPT Spills the Tea: Comprehensive Assessment of Knowledge File Leakage in GPTs

Xinyue Shen, Yun Shen, Michael Backes, Yang Zhang

Abstract

Knowledge files have been widely used in large language model (LLM)-powered agents, such as GPTs, to improve response quality. However, concerns over the potential leakage of knowledge files have grown significantly. Existing studies demonstrate that adversarial prompts can induce GPTs to leak knowledge file content. Yet, it remains uncertain whether additional leakage vectors exist, particularly given the complex data flows across clients, servers, and databases in GPTs. In this paper, we present a comprehensive risk assessment of knowledge file leakage, leveraging a novel workflow inspired by Data Security Posture Management (DSPM). Through the analysis of 651,022 GPT metadata, 11,820 flows, and 1,466 responses, we identify five leakage vectors: metadata, GPT initialization, retrieval, sandboxed execution environments, and prompts. These vectors enable adversaries to extract sensitive knowledge file data such as titles, content, types, and sizes. Notably, the activation of the built-in tool Code Interpreter leads to a privilege escalation vulnerability, enabling adversaries to directly download original knowledge files with a 95.95% success rate. Further analysis reveals that 28.80% of leaked files are copyrighted, including digital copies from major publishers and internal materials from a listed company. In the end, we provide actionable solutions for GPT builders and platform providers to secure the GPT data supply chain.

Introduction: Mapping the Data Supply Chain

Custom GPTs allow "builders" to upload knowledge files, such as proprietary PDFs or datasets, to provide specialized context for the agent. While previous research has focused almost exclusively on adversarial prompting—tricking the model into reciting file contents—this paper argues that such a view is too narrow. GPTs are not just models; they are web applications with complex data flows between clients, servers, and databases. By mapping this "data supply chain," researchers identified that information often "spills" through the cracks of the infrastructure itself, rather than just the chat interface.

The complex data supply chain of GPTs, illustrating flows between the user, client, server, and database.

Methodology: A DSPM-Inspired Approach

The authors adopted a framework inspired by Data Security Posture Management (DSPM), a standard industry approach for securing data across complex cloud environments. Their workflow consisted of four systematic phases to evaluate the ecosystem's vulnerabilities.

The four-phase research workflow: Data Discovery, Data Classification, Risk Assessment, and Mitigation.
  1. Data Discovery: Using a custom tool called GPTracker, the team monitored API metadata, web socket "flows" (structured messages during chat sessions), and final rendered responses.
  2. Data Classification: Leaked data was categorized into seven dimensions, ranging from basic metadata (file IDs and types) to original source files.
  3. Risk Assessment: The team tested five leakage vectors: Metadata, GPT Initialization, Retrieval (the myfiles_browser tool), Sandboxed Execution Environments (SEE), and Prompts.
  4. Mitigation: The study evaluated the effectiveness of common "defense prompts" in preventing these leaks.

Key Findings and Vulnerabilities

Infrastructure Leaks and Metadata Exposure

A critical insight is that even if a model is perfectly aligned to refuse file disclosure in chat, the underlying system may broadcast file names, sizes, and IDs in background web socket traffic (the "Initialization Flow") before a user even sends a message. In fact, 100% of GPTs with knowledge files were found to leak file IDs and types through these API or initialization flows.

Code Interpreter Privilege Escalation

The paper's most alarming finding is a privilege escalation vulnerability in the Code Interpreter tool. While OpenAI implements safeguards that return a "403 Forbidden" error when users click direct download links, researchers found a simple bypass. By asking the Code Interpreter to copy a file to a new name within the sandbox (/mnt/data), the system generates a new, unprotected download link. This exploit achieved a 95.95% success rate in downloading original, bit-for-bit knowledge files.

The scale of exposure is significant: 23.79% of all GPTs in the store (over 150,000) have knowledge files attached. Deep-dive analysis of 566 successfully extracted files revealed that nearly 29% were copyrighted, including textbooks from major publishers like Springer and Elsevier, as well as internal corporate documents valued at hundreds of millions of dollars.

Conclusion and Mitigation

The researchers have responsibly disclosed these vulnerabilities to OpenAI. They conclude that prompt-based defenses (e.g., "Do not reveal my files") are fragile and easily bypassed. The ultimate solution lies in server-side API redesigns that filter sensitive metadata and strictly control file access within execution environments, rather than relying on client-side obfuscation or model alignment.

Summary updated

Abstract

Knowledge files have been widely used in large language model (LLM)-powered agents, such as GPTs, to improve response quality. However, concerns over the potential leakage of knowledge files have grown significantly. Existing studies demonstrate that adversarial prompts can induce GPTs to leak knowledge file content. Yet, it remains uncertain whether additional leakage vectors exist, particularly given the complex data flows across clients, servers, and databases in GPTs. In this paper, we present a comprehensive risk assessment of knowledge file leakage, leveraging a novel workflow inspired by Data Security Posture Management (DSPM). Through the analysis of 651,022 GPT metadata, 11,820 flows, and 1,466 responses, we identify five leakage vectors: metadata, GPT initialization, retrieval, sandboxed execution environments, and prompts. These vectors enable adversaries to extract sensitive knowledge file data such as titles, content, types, and sizes. Notably, the activation of the built-in tool Code Interpreter leads to a privilege escalation vulnerability, enabling adversaries to directly download original knowledge files with a 95.95% success rate. Further analysis reveals that 28.80% of leaked files are copyrighted, including digital copies from major publishers and internal materials from a listed company. In the end, we provide actionable solutions for GPT builders and platform providers to secure the GPT data supply chain.
BibTeX
@inproceedings{shen2025when,
  author = {Xinyue Shen and Yun Shen and Michael Backes and Yang Zhang},
  title = {{When GPT Spills the Tea: Comprehensive Assessment of Knowledge File Leakage in GPTs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=FqXYtUgg16},
}
Figure