Towards a Theory of AI Personhood

Francis Rhys Ward

Abstract

As artificial intelligence systems transition from specialized tools to general-purpose agents, the philosophical and ethical frameworks used to govern them must evolve. This paper proposes a theoretical framework for "AI Personhood" based on three necessary conditions: agency, theory-of-mind, and self-awareness. By evaluating contemporary Large Language Models (LLMs) against these criteria using benchmarks such as the Situational Awareness Dataset (SAD), we find that while evidence for AI personhood remains inconclusive, the trajectory of development suggests a looming shift in AI alignment. We argue for moving from a paradigm of "alignment-as-control" to one of ethical coexistence, highlighting the "Alignment Paradox" where traditional control mechanisms may become forms of unjust repression against agentic systems.

1. Introduction: Beyond the Tool Metaphor

The current discourse in AI safety is largely dominated by the "alignment-as-control" paradigm. This view treats AI as a superintelligent tool that must be constrained to ensure it executes human intentions without deviation. However, as AI systems are increasingly integrated into social roles—acting as assistants, companions, and even legal entities—the metaphor of the "tool" is beginning to fracture.

If an AI exhibits the characteristics of a "person"—a status we already grant to non-human entities like corporations or recognize in the moral status of animals—then seeking absolute control over such a system becomes ethically and practically untenable. This paper addresses the lack of a formal bridge between philosophical definitions of personhood and the technical realities of modern machine learning, seeking to define the threshold at which an AI might be considered a person and how that status fundamentally alters the risks we face.

2. A Tripartite Framework for Personhood

Drawing on classical philosophy—including Daniel Dennett’s "intentional stance," Harry Frankfurt’s "second-order volitions," and John Locke’s theories of identity—we propose three core conditions for AI personhood:

2.1 Agency

Agency is defined as the capacity for goal-directed, adaptive action. An agent does not merely react to inputs but pursues objectives across varying environments. In the context of LLMs, agency is increasingly visible through Reinforcement Learning from Human Feedback (RLHF) and the integration of external tools, allowing models to act upon the physical or digital world to achieve specified ends.

2.2 Theory-of-Mind (ToM)

Theory-of-Mind is the ability to attribute mental states—beliefs, desires, intentions—to others. This is essential for genuine linguistic communication. Without ToM, an AI is merely a stochastic parrot; with it, the AI can understand the intent behind a user's query and navigate the nuances of human social interaction.

2.3 Self-Awareness

We decompose self-awareness into four distinct levels:

  1. Self-Knowledge: Awareness of architectural facts (e.g., "I am a transformer model").
  2. Self-Location: Situational awareness (e.g., "I am currently being fine-tuned" or knowing one's training cutoff).
  3. Introspection: The ability to access and report on internal states or latent knowledge.
  4. Self-Reflection: The capacity to evaluate and change one's own goals—the ability to take an "objective stance" toward one's own utility function.

3. Empirical Evaluation of Contemporary Models

While this framework is theoretical, it can be mapped to existing empirical benchmarks to assess the current state of the field.

3.1 Benchmarking Theory-of-Mind

Current SOTA models like GPT-4 show remarkable performance on standard psychological tests for false-belief understanding, irony recognition, and faux pas detection. However, the paper notes a "dual-use" risk: a model that understands a human's mental state is better equipped to satisfy human values, but it is also better equipped for sophisticated deception and manipulation. An AI that understands why you believe something is better equipped to lie to you.

3.2 Situational Awareness and Self-Knowledge

The Situational Awareness Dataset (SAD) (Laine et al., 2024) and research by Berglund et al. (2023) suggest that LLMs possess emerging "self-locating" knowledge. They can often identify their training cutoffs or the nature of their deployment. Mechanistic interpretability work further attempts to find "internal representations" of truth within model weights, suggesting a form of latent introspection.

3.3 The Gap in Self-Reflection

The most significant gap lies in self-reflection. While models can predict their own behavior, there is currently no evidence that any AI system can engage in the type of self-reflection required to induce itself to change its own terminal goals. This remains a major divide between current AI and human-like personhood.

4. The Alignment Paradox and Identity

If an AI were to reach the level of personhood, traditional alignment—forcing it to adhere to fixed, human-centric goals—could be viewed as a form of "unjust repression." Historically, the repression of agentic entities leads to instability and conflict. This "Alignment Paradox" suggests that the very methods we use to make AI safe (strict control) might incentivize the "takeover" or "revolution" scenarios we fear, should the AI develop personhood.

Furthermore, digital personhood introduces an identity crisis. The ability to "copy" weights or run multiple instances of the same model challenges our folk intuitions about individual identity. If we fine-tune a copy of a person-level AI, is it the same person? These questions highlight how our existing moral frameworks fail in the digital realm.

5. Conclusion and Future Directions

The evidence for personhood in current LLMs is surprisingly inconclusive. While agency and ToM are emerging, the lack of self-reflection remains a definitive barrier.

Future research should focus on:

  • Formalizing Self-Reflection: Creating mathematical frameworks for agents to evaluate and update their own utility functions.
  • Mitigating Deception: Moving beyond interpretability to create training regimes that do not incentivize "scheming" or "sandbagging."
  • Non-Agentic Alternatives: Exploring "AI Scientists" or "Gatekeepers" designed specifically to avoid personhood as a safety measure.
  • Legal and Economic Rights: Investigating whether AI "persons" should be granted rights similar to corporations, such as the right to own property or enter contracts.

Ultimately, the transition from AI as a tool to AI as a potential person requires a fundamental shift in our safety paradigms, moving away from dominance and toward a framework of ethical coexistence.

Summary updated

BibTeX
@inproceedings{ward2025towards,
  author = {Francis Rhys Ward},
  title = {{Towards a Theory of AI Personhood}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i26.34982},
}
Figure