Can You Follow Me? Testing Situational Understanding in ChatGPT

Chenghao Yang, Allyson EttingerarXiv 2023 · DOI · arXiv

Abstract

While Large Language Models (LLMs) like ChatGPT excel at fluid conversation, their ability to maintain a consistent internal representation of a changing environment—Situational Understanding (SU)—remains a critical challenge. This paper evaluates ChatGPT's SU through a synthetic "box-moving" task designed to track state updates over time. Our results demonstrate a significant "step decay": as the number of conversational updates increases, the model's internal model of the world degrades. This suggests that ChatGPT relies heavily on pre-trained language patterns rather than robust, real-time state tracking, particularly when faced with counterintuitive logic or synthetic language.

Introduction

For an AI to achieve true "human-like" interaction, it must do more than generate plausible text; it must maintain a persistent and accurate internal representation of the situation being discussed. This capability, known as Situational Understanding (SU), is essential for tasks ranging from following complex instructions to engaging in long-term collaborative problem-solving. While previous research highlighted the struggles of non-chat-based LLMs in this area, this study investigates whether the advanced, chat-tuned ChatGPT exhibits similar limitations in tracking "truth conditions" across multiple updates.

Experimental Framework

To probe SU, researchers developed a controlled synthetic environment based on a box-moving game. The setup involves three components:

  1. Instructions: Defining the initial state and rules (e.g., "There are 5 boxes and 5 keys").
  2. Steps: Natural language descriptions of actions (e.g., "Open box-3 and retrieve key-2").
  3. Queries: Prompting the model to report the status of all objects after each action.

To prevent the model from relying on memorized patterns, several robustness checks were implemented:

  • Synthetic Language (SL): Replacing standard terms (e.g., "Box", "Opened") with random strings to force the model to follow explicit definitions.
  • Counterintuitive Instructions: Flipping standard logic (e.g., defining "True" as a box being closed) to test reliance on pre-trained biases.
  • Intermediate Probing: Requiring the model to update the state after every step to pinpoint where reasoning failures occur.

The Decay of Situational Understanding

The study utilizes two primary metrics: State-EM (accuracy of individual object states) and Step-EM (accuracy of the entire state after a step). A key finding is the "Consistency Gap": while ChatGPT maintains a high State-EM (~90%), its Step-EM is significantly lower, often falling to 20-30%. This indicates that while the model gets most individual facts right, it almost always makes at least one error per update.

Intermediate state probing results for 2-shot and 5-shot scenarios, illustrating the decay in Step-EM accuracy as the number of steps increases.

As shown in the probing results, performance is not limited by the complexity of the final state, but by the number of updates. As steps increase, the model's "mental map" degrades, even though the entire history remains within its context window. Interestingly, using synthetic language (SL) occasionally improved performance by mitigating the interference of pre-trained linguistic biases.

Analyzing Error Patterns

To understand the nature of these failures, the researchers categorized errors into a taxonomy of state update violations. These include "Dirty Reads" (basing an update on an incorrect previous state), "Dirty Writes" (incorrectly updating a state), and "Hallucinated Updates" (changing the status of objects not mentioned in the current step).

Taxonomy of state update errors, categorizing how models fail to maintain correct situational states over time.

The analysis reveals that ChatGPT's performance crashes under counterintuitive instructions (Step-EM dropping to 0-12%), confirming a heavy reliance on standard language-to-logic mappings learned during training. Errors are frequently driven by hallucinated updates, where the model loses track of which objects were actually modified.

Conclusion

This research highlights a fundamental weakness in current LLMs: the lack of robust, persistent situational memory. Despite their conversational prowess, models like ChatGPT struggle to maintain accurate world states over extended interactions. Future work should explore whether prompt engineering techniques like Chain-of-Thought or rule-based alignment (Constitutional AI) can help bridge this gap and move LLMs closer to true situational understanding.

Summary updated

Abstract

Understanding sentence meanings and updating information states appropriately across time -- what we call "situational understanding" (SU) -- is a critical ability for human-like AI agents. SU is essential in particular for chat models, such as ChatGPT, to enable consistent, coherent, and effective dialogue between humans and AI. Previous works have identified certain SU limitations in non-chatbot Large Language models (LLMs), but the extent and causes of these limitations are not well understood, and capabilities of current chat-based models in this domain have not been explored. In this work we tackle these questions, proposing a novel synthetic environment for SU testing which allows us to do controlled and systematic testing of SU in chat-oriented models, through assessment of models' ability to track and enumerate environment states. Our environment also allows for close analysis of dynamics of model performance, to better understand underlying causes for performance patterns. We apply our test to ChatGPT, the state-of-the-art chatbot, and find that despite the fundamental simplicity of the task, the model's performance reflects an inability to retain correct environment states across time. Our follow-up analyses suggest that performance degradation is largely because ChatGPT has non-persistent in-context memory (although it can access the full dialogue history) and it is susceptible to hallucinated updates -- including updates that artificially inflate accuracies. Our findings suggest overall that ChatGPT is not currently equipped for robust tracking of situation states, and that trust in the impressive dialogue performance of ChatGPT comes with risks. We release the codebase for reproducing our test environment, as well as all prompts and API responses from ChatGPT, at https://github.com/yangalan123/SituationalTesting.
BibTeX
@misc{yang2023can,
  author = {Chenghao Yang and Allyson Ettinger},
  title = {{Can You Follow Me? Testing Situational Understanding in ChatGPT}},
  year = {2023},
  eprint = {2310.16135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.16135},
}
Figure