Prompting as Scientific Inquiry

Ari Holtzman, Chenhao Tan

Abstract

Prompting is the primary method by which we study and control large language models. It is also one of the most powerful: nearly every major capability attributed to LLMs—few-shot learning, chain-of-thought, constitutional AI—was first unlocked through prompting. Yet prompting is rarely treated as science and is frequently frowned upon as alchemy. We argue that this is a category error. If we treat LLMs as a new kind of complex and opaque organism that is trained rather than programmed, then prompting is not a workaround: it is behavioral science. Mechanistic interpretability peers into the neural substrate, prompting probes the model in its native interface: language. We contend that prompting is not inferior, but rather a key component in the science of LLMs.

Introduction: The Prestige Bias

The AI research community currently suffers from a "prestige bias" that favors mathematical formalism, architectural innovation, and "opening the black box" (mechanistic interpretability) over interacting with models via natural language. Prompting is frequently viewed as a temporary workaround or unscientific "hackery" rather than a legitimate scientific tool.

We argue this is a category error. Because LLMs are "trained rather than programmed," they behave more like complex biological organisms than traditional software. We lack a "genetic code" for their weights, making black-box behavioral study—prompting—the most effective way to discover emergent capabilities like few-shot learning and chain-of-thought reasoning. This paper seeks to formalize "prompt science" as a disciplined method of scientific inquiry.

Defining Prompt Science

To move beyond the "alchemy" label, we must distinguish between Prompt Engineering and Prompt Science. While engineering focuses on brute-force optimization for a specific task score, prompt science aims to discover and confirm regularities in model behavior through systematic, falsifiable inquiry.

Distinguishing Prompt Science from Engineering and addressing common critiques.

Prompt science is grounded in several advantages:

  • Right Abstraction: Language is the native interface for LLMs.
  • Scalability: It works on models of any size, unlike mechanistic interpretability which struggles with large-scale parameters.
  • Falsifiability: A prompt study makes testable claims: "Adding string X will change the output distribution in way Y across models Z."

Theoretical Framework: Marr’s Levels and Productive Vagueness

Applying David Marr’s classic framework to AI, we argue that prompting operates at the Computational level (defining what the system does and why), while mechanistic interpretability operates at the Implementational level (how it is physically realized in weights).

Furthermore, natural language allows for "productive vagueness." Unlike the rigid precision required to map a circuit, language invokes "rich clusters of associations" learned by the model. This ambiguity is a feature that mirrors the distributed nature of neural representations, allowing researchers to probe complex concepts that are not yet reducible to individual neurons.

The Rebranding Cycle and Historical Context

A fascinating trend in the community is the "rebranding cycle." Once a prompting technique becomes successful and rigorous, it is often rebranded to distance it from the "unscientific" label of prompting. For example, Chain-of-Thought is now often discussed as "inference-time compute," and instruction-following is seen as a "training methodology" (RLHF).

The evolution and rebranding of prompting techniques over time.

Historical breakthroughs, such as the discovery of "Sparks of AGI" or the initial unlocking of Chain-of-Thought reasoning, were achieved through prompting. These case studies show that major capabilities are often discovered through behavioral interaction rather than theoretical foresight.

Conclusion and Future Directions

Prompting is not a workaround; it is the behavioral science of AI. It provides the "behavioral anomalies" that give mechanistic researchers a starting point for internal investigation. Future work should focus on:

  • Mechanism Discovery: Using systematic perturbations to map model knowledge boundaries.
  • Control Primitives: Identifying a minimal set of prompt "primitives" for predictable behavioral changes.
  • Inter-Intelligence Communication: Developing protocols for efficient communication across intelligence gaps.

By treating LLMs as "alien species" to be studied through interaction, we can turn prompting from a dark art into a rigorous science.

Summary updated

Abstract

Prompting is the primary method by which we study and control large language models. It is also one of the most powerful: nearly every major capability attributed to LLMs-few-shot learning, chain-of-thought, constitutional AI-was first unlocked through prompting. Yet prompting is rarely treated as science and is frequently frowned upon as alchemy. We argue that this is a category error. If we treat LLMs as a new kind of complex and opaque organism that is trained rather than programmed, then prompting is not a workaround: it is behavioral science. Mechanistic interpretability peers into the neural substrate, prompting probes the model in its native interface: language. We contend that prompting is not inferior, but rather a key component in the science of LLMs.
BibTeX
@inproceedings{holtzman2025prompting,
  author = {Ari Holtzman and Chenhao Tan},
  title = {{Prompting as Scientific Inquiry}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=yZU8kdwafM},
}
Figure