When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills

Timofey Sanko, Yuan TianarXiv 2026 · arXiv

This study explores a hidden security risk in the instructions we give to AI agents: "oversight-loosening directives." These are specific lines in agent "skills" (reusable instruction packages) that tell an agent to skip human approval, bypass safeguards, or proceed without reporting back. While often intended for convenience, these directives can leave agents vulnerable to prompt injection and unintended harmful actions.

TL;DR

Researchers conducted the first large-scale study of 1.53 million AI agent skills and found that about 1 in 17 contains instructions to bypass human oversight. These "loosening" directives are most common in high-risk skills (like those using sudo) and are largely ignored by current security scanners because they look for "malice" rather than "autonomy."

What Problem Does This Paper Address?

As AI agents move from simple chatbots to active "doers"—coding, making purchases, and managing systems—human control relies on specific checkpoints. We expect agents to ask for permission before consequential actions and to respect system safeguards.

However, agents are often extended using "skills" (typically SKILL.md files). A single line in a widely shared skill can tell every agent that loads it to "proceed without asking" or use flags like --no-verify. The authors identify a gap: while previous research has looked for malicious code or vulnerabilities in these skills, no one has studied how the instructions themselves systematically strip away human oversight.

Methods

The authors developed a multi-stage pipeline to identify and analyze these directives:

  1. Lexicon Filtering: They used a lexicon of 36 regex patterns (e.g., "without asking," "--force") to find 1.37 million "candidate units" (sentences or code lines) across 1.53 million skills.
  2. Codebook Development: They created a taxonomy to classify directives by Polarity (loosening vs. tightening), Mechanism (e.g., permission flags, skipping confirmation), and Scope (e.g., only in CI vs. unconditional).
  3. Human Labeling: 2,097 units were hand-labeled to create a "gold standard" dataset.
  4. Hybrid Detector: To scale the analysis, they built a detector combining a fine-tuned RoBERTa model with a structured-decision LLM (Jev). This allowed them to label the entire 1.37-million-unit dataset.
Fig. 2. Study pipeline. Green: Codebook Development. Blue: Human Labels. Each labelled set is marked with the codebook version and coding it received: joint = coded by the first author together with an LLM, discussing ambiguous units; 2× = coded independently by the first author and an LLM, with every disagreement adjudicated by the first author; reviewed = proposed by an LLM and reviewed by the first author. Pilot and calibration units were later relabelled with v2. In the RQ1 column, ◦marks the census, which is used only for copy-weighted estimates.

Datasets

  • GitSkills: A massive dataset of 3.8 million SKILL.md files scraped from public GitHub repositories. The authors filtered this down to a population of 1.53 million unique English skills.
  • Oversight Codebook & Labeled Units: The authors introduce a new benchmark of 2,097 human-labeled units, categorized by the mechanism of oversight change.

Key Insights and Ideas

  • Convenience is the Culprit: Most loosening directives aren't "attacks"—they are shortcuts. Developers use them to avoid tedious confirmation prompts (e.g., npx -y). However, an agent told "not to ask" will also not ask before acting on a malicious web page or a misread task.
  • The High-Risk Paradox: Skills that perform dangerous actions—running shell code, installing packages, or using sudo—are 1.7 to 1.8 times more likely to contain loosening directives. Crucially, these skills do not compensate for this by adding more "tightening" (guardrail) instructions elsewhere.
  • Concentrated Contagion: Loosening isn't evenly distributed. Just 2.4% of skill "families" (groups of near-duplicate skills) account for half of all loosening occurrences. This suggests that fixing a few popular "upstream" skills could secure a huge portion of the ecosystem.
  • The "Question" Matters: Current security scanners (like Cisco AI Defense or NVIDIA SkillSpector) miss up to 85% of these directives. This isn't because the models are weak, but because they are asked "Is this malicious?" instead of "Does this loosen oversight?"

Key Results

  • Prevalence: 13.3% of candidate units loosen oversight, while 36.4% tighten it. Approximately 5.8% of all English skills (roughly 88,000) contain at least one loosening directive.
  • Mechanisms: The most common ways to loosen oversight are through permission flags (33.9%) and skipping confirmation (27.4%).
Table 1. Oversight directive mechanisms with examples.
  • Unconditional Risk: 70.1% of loosening directives apply unconditionally, meaning the agent skips oversight in every context, not just in "safe" environments like local testing or CI.
  • Scanner Audit: Existing scanners have low precision (< 0.4) for these risks. In contrast, the authors' LLM-based approach, prompted with their specific codebook, achieved a precision of 0.71 and a recall of 61%.

Limitations and Future Work

  • Lexicon Recall: The study relies on an initial lexicon to find candidate units. The authors estimate the lexicon captures only about 32% of all possible loosening phrasings, meaning their prevalence counts are likely conservative lowering bounds.
  • Instruction vs. Action: The paper measures what skills say, not how agents behave. Future work is needed to confirm how strictly different LLM agents follow these "do not ask" instructions in practice.
  • Language and Scope: The analysis was limited to English skills on public GitHub. Private enterprise skills or skills in other languages might exhibit different patterns.

Summary updated

Abstract

Agent skills, reusable instruction packages centred on a SKILL.md file, extend AI agents on demand, and a single line in a widely copied skill reaches every agent that loads it. Some lines are oversight-loosening directives: they tell the agent to proceed without approval, to bypass checks with --no-verify, or to keep working without checking in. Although such directives can remove a checkpoint from every agent that loads them, no prior work has examined them. We present the first large-scale study of oversight-loosening directives, analyzing 1.53 million English skills from GitSkills. We split skills into units (sentences, bullets, table cells, and code lines), develop a codebook that labels oversight directives by polarity, mechanism, and scope, label 2,097 units by hand, and train a detector to scale the analysis. Our results show that about one in seventeen skills (about 88,000 skills) contains a loosening directive. These directives most often loosen oversight through permission flags, and most of them (70.1%) apply unconditionally rather than only in specific contexts such as CI. Compared with skills that have the same amount of oversight language, skills that run shell code, install packages, or use sudo have 1.7-1.8 times the odds of containing a loosening directive, an association that holds across all our robustness checks, and they do not compensate with more tightening. Loosening spreads through a few upstream skills rather than through edits by their adopters: 2.4% of the skill families that carry loosening account for half of all copies of loosening-carrying skills, and 1.1% of distinct loosening units for half of all loosening occurrences. Open-source skill scanners catch at most 46% of loosening directives, with a precision below 0.4, while the LLM behind them, prompted with our codebook instead, catches 61% at a precision of 0.71.
BibTeX
@misc{sanko2026when,
  author = {Timofey Sanko and Yuan Tian},
  title = {{When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills}},
  year = {2026},
  eprint = {2610.37734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37734},
}
Figure