Abstract
While Large Language Models (LLMs) like ChatGPT excel at fluid conversation, their ability to maintain a consistent internal representation of a changing environment—Situational Understanding (SU)—remains a critical challenge. This paper evaluates ChatGPT's SU through a synthetic "box-moving" task designed to track state updates over time. Our results demonstrate a significant "step decay": as the number of conversational updates increases, the model's internal model of the world degrades. This suggests that ChatGPT relies heavily on pre-trained language patterns rather than robust, real-time state tracking, particularly when faced with counterintuitive logic or synthetic language.
Introduction
For an AI to achieve true "human-like" interaction, it must do more than generate plausible text; it must maintain a persistent and accurate internal representation of the situation being discussed. This capability, known as Situational Understanding (SU), is essential for tasks ranging from following complex instructions to engaging in long-term collaborative problem-solving. While previous research highlighted the struggles of non-chat-based LLMs in this area, this study investigates whether the advanced, chat-tuned ChatGPT exhibits similar limitations in tracking "truth conditions" across multiple updates.
Experimental Framework
To probe SU, researchers developed a controlled synthetic environment based on a box-moving game. The setup involves three components:
- Instructions: Defining the initial state and rules (e.g., "There are 5 boxes and 5 keys").
- Steps: Natural language descriptions of actions (e.g., "Open box-3 and retrieve key-2").
- Queries: Prompting the model to report the status of all objects after each action.
To prevent the model from relying on memorized patterns, several robustness checks were implemented:
- Synthetic Language (SL): Replacing standard terms (e.g., "Box", "Opened") with random strings to force the model to follow explicit definitions.
- Counterintuitive Instructions: Flipping standard logic (e.g., defining "True" as a box being closed) to test reliance on pre-trained biases.
- Intermediate Probing: Requiring the model to update the state after every step to pinpoint where reasoning failures occur.
The Decay of Situational Understanding
The study utilizes two primary metrics: State-EM (accuracy of individual object states) and Step-EM (accuracy of the entire state after a step). A key finding is the "Consistency Gap": while ChatGPT maintains a high State-EM (~90%), its Step-EM is significantly lower, often falling to 20-30%. This indicates that while the model gets most individual facts right, it almost always makes at least one error per update.
As shown in the probing results, performance is not limited by the complexity of the final state, but by the number of updates. As steps increase, the model's "mental map" degrades, even though the entire history remains within its context window. Interestingly, using synthetic language (SL) occasionally improved performance by mitigating the interference of pre-trained linguistic biases.
Analyzing Error Patterns
To understand the nature of these failures, the researchers categorized errors into a taxonomy of state update violations. These include "Dirty Reads" (basing an update on an incorrect previous state), "Dirty Writes" (incorrectly updating a state), and "Hallucinated Updates" (changing the status of objects not mentioned in the current step).
The analysis reveals that ChatGPT's performance crashes under counterintuitive instructions (Step-EM dropping to 0-12%), confirming a heavy reliance on standard language-to-logic mappings learned during training. Errors are frequently driven by hallucinated updates, where the model loses track of which objects were actually modified.
Conclusion
This research highlights a fundamental weakness in current LLMs: the lack of robust, persistent situational memory. Despite their conversational prowess, models like ChatGPT struggle to maintain accurate world states over extended interactions. Future work should explore whether prompt engineering techniques like Chain-of-Thought or rule-based alignment (Constitutional AI) can help bridge this gap and move LLMs closer to true situational understanding.
Summary updated