Detecting and Deterring Manipulation in a Cognitive Hierarchy

Nitay Alon, Lion Schulz, Joseph M. Barnby, Jeffrey S. Rosenschein, Peter DayanarXiv 2024 · arXiv · OpenReview

Abstract

Social agents with finitely nested opponent models are vulnerable to manipulation by agents with deeper reasoning and more sophisticated opponent modelling. This imbalance, rooted in logic and the theory of recursive modelling frameworks, cannot be solved directly. We propose a computational framework, $\aleph$-IPOMDP, augmenting model-based RL agents' Bayesian inference with an anomaly detection algorithm and an out-of-belief policy. Our mechanism allows agents to realize they are being deceived, even if they cannot understand how, and to deter opponents via a credible threat. We test this framework in both a mixed-motive and zero-sum game. Our results show the $\aleph$ mechanism's effectiveness, leading to more equitable outcomes and less exploitation by more sophisticated agents. We discuss implications for AI safety, cybersecurity, cognitive science, and psychiatry.

What Problem Does This Paper Address?

In multi-agent systems, agents often employ Theory of Mind (ToM) to predict the actions of others. This creates a "Cognitive Hierarchy" where agents are ranked by their depth of model (DoM). For instance, a DoM(0) agent might act based on simple heuristics, while a DoM(1) agent models the DoM(0) agent's thoughts to gain a strategic advantage.

Cognitive Hierarchy: The nested nature of Theory of Mind models creates a strategic gap where higher-level agents can exploit lower-level ones.

The authors identify a fundamental logical trap: an agent at Level $k$ is formally incapable of accurately modeling an agent at Level $k+1$. This "cognitive gap" leaves simpler agents vulnerable to manipulation, as they cannot conceive of the complex strategies being deployed against them. The central question is: How can a computationally limited agent protect itself from a more sophisticated deceiver it cannot fully understand?

Methods: The $\aleph$-IPOMDP Framework

To address this vulnerability, the authors introduce the $\aleph$-IPOMDP (Aleph-IPOMDP) framework. This approach does not attempt to "out-think" the opponent by increasing model depth. Instead, it augments standard Bayesian reinforcement learning with defensive heuristics:

  1. The $\aleph$-Mechanism (Anomaly Detection): The agent monitors for deviations from expected behavior using two primary metrics:
  2. Typicality ($Z^1$): Utilizing a gzip-based compression algorithm, the agent assesses whether the opponent's action sequence is "typical." If the sequence is unexpectedly complex or structured, it flags an anomaly.
  3. Reward Monitoring ($Z^2$): The agent compares its actual cumulative reward against the expected reward predicted by its internal model. A significant shortfall indicates potential exploitation.
  4. The $\aleph$-Policy (Out-of-Belief Response): Upon detecting deception, the agent abandons its failed internal model and switches to an "Out-of-Belief" (OOB) policy:
  5. In mixed-motive games, it employs a Grim Trigger (retaliation) to make exploitation costly for the deceiver.
  6. In zero-sum games, it adopts a Minimax strategy to minimize its maximum possible loss.
Overview of Agents and Tasks: Comparison between standard Cognitive Hierarchy agents and the proposed Aleph-IPOMDP framework across two game-theoretic benchmarks.

Experimental Evaluation

The framework was evaluated using two simulated benchmarks:

  • Iterated Ultimatum Game (IUG): A mixed-motive scenario where a "Sender" proposes a split of resources and a "Receiver" accepts or rejects. This tests the ability to detect a deceiver masquerading as a simpler player to lower expectations before exploiting them.
  • Iterated Bayesian Zero-Sum Game: A "Row/Column" game where one player possesses private information about the payoff matrix, testing the agent's ability to "call a bluff."

Key Results and Insights

  • Closing the Income Gap: In the Ultimatum Game, the $\aleph$-mechanism reduced the reward gap between the deceiver and the victim by over 40%.
  • Successful Deterrence: High-level agents (DoM 1) were forced to make fairer offers to avoid triggering the victim's retaliatory $\aleph$-policy, leading to more equitable outcomes.
  • Heuristics over Hierarchy: The research demonstrates that an agent doesn't need to understand how it is being tricked to know that it is being tricked. Anomaly detection serves as a vital "conduct violation" alarm.
  • The "Paranoia" Trade-off: Setting detection thresholds too strictly leads to "maladaptive sensitivity," a computational analog to clinical paranoia, where agents reject honest partners due to perceived suspicion.

Limitations and Future Work

While effective, the current OOB policies are relatively blunt "all-or-nothing" instruments. Future work aims to develop more nuanced, adaptive responses and integrate active learning to characterize deceivers after detection. This framework has significant implications for AI safety—particularly in verifying the intentions of Large Language Models (LLMs)—and provides a formal model for understanding psychiatric conditions like psychosis or social anxiety.

Summary updated

Abstract

Social agents with finitely nested opponent models are vulnerable to manipulation by agents with deeper reasoning and more sophisticated opponent modelling. This imbalance, rooted in logic and the theory of recursive modelling frameworks, cannot be solved directly. We propose a computational framework, $\aleph$-IPOMDP, augmenting model-based RL agents' Bayesian inference with an anomaly detection algorithm and an out-of-belief policy. Our mechanism allows agents to realize they are being deceived, even if they cannot understand how, and to deter opponents via a credible threat. We test this framework in both a mixed-motive and zero-sum game. Our results show the $\aleph$ mechanism's effectiveness, leading to more equitable outcomes and less exploitation by more sophisticated agents. We discuss implications for AI safety, cybersecurity, cognitive science, and psychiatry.
BibTeX
@misc{alon2024detecting,
  author = {Nitay Alon and Lion Schulz and Joseph M. Barnby and Jeffrey S. Rosenschein and Peter Dayan},
  title = {{Detecting and Deterring Manipulation in a Cognitive Hierarchy}},
  year = {2024},
  eprint = {2405.01870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.01870},
}
Figure