ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet

Yuekang Li, Wei Song, Bangshuo Zhu, Dong Gong, Yi Liu, Gelei Deng, Chunyang Chen, Lei Ma, Jun Sun, Toby Walsh, Jingling XuearXiv 2025 · OpenReview

Abstract

We introduce ai.txt, a novel domain-specific language (DSL) designed to explicitly regulate interactions between AI models, agents, and web content, addressing critical limitations of the widely adopted robots.txt standard. As AI increasingly engages with online materials for tasks such as training, summarization, and content modification, existing regulatory methods lack the necessary granularity and semantic expressiveness to ensure ethical and legal compliance. ai.txt extends traditional URL-based access controls by enabling precise element-level regulations and incorporating natural language instructions interpretable by AI systems. To facilitate practical deployment, we provide an integrated development environment with code autocompletion and automatic XML generation. Furthermore, we propose two compliance mechanisms: XML-based programmatic enforcement and natural language prompt integration, and demonstrate their effectiveness through preliminary experiments and case studies. Our approach aims to aid the governance of AI-Internet interactions, promoting responsible AI use in digital ecosystems.

Introduction: The Limitations of robots.txt

The ubiquitous robots.txt standard, established in 1994, was designed for a simpler era of web crawling. Its primary function is to manage how search engines index pages by allowing or disallowing access to specific URL paths. However, modern generative AI models and autonomous agents do far more than index content; they scrape data for training, generate summaries, and perform complex analyses.

Figure 1: Comparison between traditional robots.txt limitations and the proposed ai.txt element-level granularity.

As shown in Figure 1, current methods for restricting AI are often informal and non-machine-readable. Organizations like The New York Times currently resort to text comments within robots.txt to prohibit AI training—instructions that lack formal enforceability. robots.txt is a blunt instrument; it cannot protect specific parts of a page (like a single paragraph or image) nor can it define specific behavioral constraints, such as "allow crawling for search but disallow for training."

Design Principles and Methodology

The development of ai.txt was guided by four core design principles: simplicity, clarity, consistency, and functionality. These pillars ensure the language is accessible to webmasters while remaining powerful enough for complex AI governance.

Figure 3: The four design principles of the ai.txt domain-specific language.

The DSL Architecture

Built using Extended Backus–Naur Form (EBNF), the language mirrors the familiar structure of robots.txt but introduces several key innovations:

  1. Granular Targeting: Unlike its predecessor, ai.txt can target specific Document Object Model (DOM) elements using CSS selectors for HTML or dot notation for JSON/XML data.
  2. Action Verbs: The language utilizes a curated list of "Regulated Actions," such as Train, Summarize, Analyze, and Clip, allowing site owners to specify exactly how their content should be used.
  3. The Guide Directive: Beyond simple prohibition, the Guide directive allows site owners to provide natural language instructions (e.g., "When summarizing, ensure the primary author is credited in the first sentence").

Implementation and Compliance Mechanisms

To bridge the gap between a text file and actual AI behavior, we propose two primary compliance mechanisms:

  • Programmatic Enforcement: The ai.txt file is converted into a machine-readable XML format. This allows developers to integrate hard-coded logic into their agent's pipeline, creating a technical barrier against unauthorized actions.
  • Prompt-level Enforcement: The system parses ai.txt rules into natural language instructions that are injected directly into a Large Language Model’s (LLM) system prompt. This ensures the model is aware of the site owner's preferences during the generation process.

To support adoption, we implemented an Integrated Development Environment (IDE) using JetBrains’ Meta Programming System (MPS), providing webmasters with autocompletion and automatic XML generation tools.

Evaluation and Key Insights

Our research demonstrates that ai.txt can successfully encode complex real-world requirements that are currently relegated to informal comments. Preliminary experiments using GPT-4o showed high adherence to ai.txt directives when implemented via prompt-level enforcement. In standard scenarios, the model consistently respected element-level restrictions and followed specific summarization guidelines.

The shift from path-based to element-based control represents a fundamental change in web governance. By formalizing ethical and legal preferences into a machine-readable DSL, we provide a framework for turning abstract policies into executable instructions for autonomous systems.

Conclusion and Future Work

While ai.txt offers a significant advancement in granularity and expressiveness, it remains a "gentleman’s agreement" similar to robots.txt. It relies on the willingness of AI developers to respect the file. Future research will focus on adversarial robustness—ensuring models cannot be "tricked" into ignoring directives— and exploring active technical blocking mechanisms. Ultimately, the success of ai.txt depends on widespread industry adoption to create a more transparent and respectful digital ecosystem.

Summary updated

Abstract

We introduce ai.txt, a novel domain-specific language (DSL) designed to explicitly regulate interactions between AI models, agents, and web content, addressing critical limitations of the widely adopted robots.txt standard. As AI increasingly engages with online materials for tasks such as training, summarization, and content modification, existing regulatory methods lack the necessary granularity and semantic expressiveness to ensure ethical and legal compliance. ai.txt extends traditional URL-based access controls by enabling precise element-level regulations and incorporating natural language instructions interpretable by AI systems. To facilitate practical deployment, we provide an integrated development environment with code autocompletion and automatic XML generation. Furthermore, we propose two compliance mechanisms: XML-based programmatic enforcement and natural language prompt integration, and demonstrate their effectiveness through preliminary experiments and case studies. Our approach aims to aid the governance of AI-Internet interactions, promoting responsible AI use in digital ecosystems.
BibTeX
@misc{li2025ai,
  author = {Yuekang Li and Wei Song and Bangshuo Zhu and Dong Gong and Yi Liu and Gelei Deng and Chunyang Chen and Lei Ma and Jun Sun and Toby Walsh and Jingling Xue},
  title = {{ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet}},
  year = {2025},
  eprint = {2505.07834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.07834},
}
Figure