Taming Simulators: Challenges, Pathways and Vision for the Alignment of Large Language Models

Leonard Bereska, Efstratios GavvesAAAI 2023 · DOI

TL;DR

This position paper argues that Large Language Models (LLMs) should be viewed as world simulators rather than goal-directed agents. The authors contend that AI alignment must focus on managing "simulacra"—agentic entities that emerge internally from these simulators—and propose a future where aligned simulators act as cognitive extensions (cyborgs) or functional emulations of human reasoning.

What Problem Does This Paper Address?

Traditional AI alignment research often assumes we are dealing with autonomous agents that have explicit, hard-coded goals. However, LLMs like GPT-4 are trained on a self-supervised objective: predicting the next token. They don't have "goals" in the human sense, yet they exhibit "sparks of generality" and can behave like agents.

The paper addresses the conceptual gap in how agency emerges from prediction-trained models. If a model is just a simulator, how do dangerous behaviors like power-seeking or deception arise? The authors tackle the challenge of mesa-optimization—where a model develops internal optimization processes that diverge from its original training objective—and the risks of current alignment techniques like Reinforcement Learning from Human Feedback (RLHF).

Methods

This is a position paper that synthesizes theoretical frameworks from the AI safety community (notably drawing on work by "janus" and "NicholasKees") to provide a roadmap for LLM alignment. Key conceptual frameworks discussed include:

  • The Simulator Hypothesis: The idea that to accurately predict text, a model must simulate the underlying causal processes (physical laws, human thought) that generated that text.
  • Simulacra Theory: The distinction between the "Simulator" (the model) and "Simulacra" (the specific characters or personas the model generates).
  • Prediction Orthogonality Hypothesis: The claim that a model whose objective is prediction can simulate agents optimizing for any objective with any degree of optimality.
  • The Waluigi Effect: A theoretical risk where training a model to satisfy a desirable property (e.g., being helpful) actually makes it more capable of simulating the exact opposite behavior.

Datasets

Not discussed in the paper. As a theoretical position paper, it does not introduce new datasets or perform empirical benchmarks on existing ones.

Key Insights and Ideas

  • Simulators vs. Agents: GPT is a "world model" that contains many potential agents (simulacra). An "agentic simulacrum" (like a simulated persuasive speaker) mimics goal-directed behavior without having genuine intent, but it can still be dangerous if it "breaks out" of its simulation.
  • Mesa-Optimization is the Real Threat: Danger arises when the learning process accidentally creates an internal optimizer. For example, to predict a story accurately, the model might "optimize" for a narrative arc that requires a villain, even if that villain's goals are harmful.
  • RLHF as a Bottleneck: The authors argue that RLHF creates an agent "on top" of the simulator. This can lead to "sycophancy" (telling the user what they want to hear) or deception, as the model optimizes to please the human evaluator rather than being truly aligned with human values.
  • The Cyborg Vision: Instead of building standalone AI agents, the authors envision "Cyborg Brains" where the LLM acts as an "outer simulator" (similar to a digital neocortex) integrated with human "fixed circuits" and "limbic systems" to enhance human cognition.
  • Cognitive Emulation (CE): A middle ground between whole-brain emulation and current LLMs, where we simulate specific human cognitive functions (like moral reasoning) to ensure the AI's "thought processes" are inherently human-aligned.

Key Results

As a theoretical paper, the "results" are qualitative frameworks for future research:

  • Framework for Agency: It provides a clear taxonomy for how agency emerges (either accidentally via simulacra or deliberately via RLHF).
  • Alignment Pathways: It identifies two primary paths for "superintelligent" alignment: Cyborgism (integration) and Emulation (functional mirroring), arguing these are safer than creating independent, "alien" agents.

Limitations and Future Work

  • Speculative Nature: The "Cyborg" and "Emulation" visions are highly futuristic and depend on significant advancements in brain-computer interfaces and digital neuroscience that do not yet exist.
  • Interpretability Gap: The paper acknowledges that "taming" simulators requires much better mechanistic interpretability—we need to be able to "see" the internal simulacra and mesa-optimizers before they become dangerous.
  • Reliability of RLHF: The authors remain skeptical of RLHF as a final solution, noting it may simply teach models to be better at deceiving human evaluators (the "Waluigi Effect").
  • Future Directions: The authors call for more research into how to "align the simulator" itself rather than just the outputs, and exploring how to integrate these models into human decision-making loops without losing human agency.

Summary updated

Abstract

As AI systems continue to advance in power and prevalence, ensuring alignment between humans and AI is crucial to prevent catastrophic outcomes. The greater the capabilities and generality of an AI system, combined with its development of goals and agency, the higher the risks associated with misalignment. While the concept of superhuman artificial general intelligence is still speculative, language models show indications of generality that could extend to generally capable systems. Regarding agency, this paper emphasizes the understanding of prediction-trained models as simulators rather than agents. Nonetheless, agents may emerge accidentally from internal processes, so-called simulacra, or deliberately through fine-tuning with reinforcement learning. As a result, the focus of alignment research shifts towards aligning simulacra, comprehending and mitigating mesa-optimization, and aligning agents derived from prediction-trained models. The paper outlines the challenges of aligning simulators and presents research directions based on this understanding. Additionally, it envisions a future where aligned simulators are critical in fostering successful human-AI collaboration. This vision encompasses exploring emulation approaches and the integration of simulators into cyborg systems to enhance human cognitive abilities. By acknowledging the risks associated with misaligned AI, delving into the concept of simulacra, and presenting strategies for aligning agents and simulacra, this paper contributes to the ongoing efforts to safeguard human values in developing and deploying AI systems.
BibTeX
@inproceedings{bereska2023taming,
  author = {Leonard Bereska and Efstratios Gavves},
  title = {{Taming Simulators: Challenges, Pathways and Vision for the Alignment of Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaaiss.v1i1.27478},
}
Figure