Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction

Sonny George, Chris Sypherd, Dylan CashmanEMNLP 2024 · DOI

Introduction

As Large Language Model (LLM) agents are increasingly deployed as autonomous assistants, their ability to maintain a reliable "memory" of past interactions becomes critical. A key challenge for these agents is not just remembering facts, but operationalizing them—synthesizing disparate pieces of information from a long history to make correct decisions in the present. This paper investigates a specific failure mode: the "Red Herring" problem, where agents are distracted by irrelevant but recent information, leading to a significant reliability gap.

The OEDD Corpus: A Novel Testing Framework

To evaluate this capability, researchers developed the Operationalize Experience Despite Distraction (OEDD) corpus. This framework moves beyond simple instruction following by requiring agents to connect two separate premises (Fact A and Fact B) to reach a logical conclusion, often in the presence of a distracting "red herring."

Figure 1: An example scenario from the OEDD corpus involving a robot waiter at Pho79. The agent must synthesize management rules and dish ingredients while ignoring customer enjoyment feedback (the red herring).

The corpus consists of 16 human-validated scenarios expanded into 192 unique test configurations. These scenarios use "non-priors" details—specific, fictional rules—to ensure models cannot rely on pre-existing world knowledge. The difficulty is categorized by the number of premises required (1P or 2P) and the presence of a Red Herring (+RH).

Experimental Results

The study evaluated three state-of-the-art models: GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro across varying context lengths (Short, Medium, and Long).

Figure 3: Performance heatmaps for GPT-3.5, GPT-4o, and Gemini 1.5 Pro across different reasoning configurations and context lengths. Darker green indicates higher accuracy, while orange/red indicates performance near or below random chance.

Key Findings

  1. The Synthesis Bottleneck: There is a dramatic performance cliff when moving from single-premise (1P) to two-premise (2P) reasoning. While models like GPT-4o and Gemini 1.5 Pro achieve near-perfect scores (~95%) on simple 1P tasks, their ability to "connect the dots" between two separate facts is significantly lower.
  2. The Impact of Red Herrings: The introduction of a "red herring" just before the decision point causes a catastrophic drop in accuracy. In the most challenging 2P+RH configurations with long contexts, models often performed worse than a coin flip. For instance, GPT-4o's accuracy dropped to 0.41 in medium-context 2P+RH tasks, while GPT-3.5 bottomed out at 0.19.
  3. Recency Bias: The results suggest a strong recency bias; LLMs are disproportionately influenced by the most recent information in their prompt, even when it is logically irrelevant to the established historical context.

Conclusion and Future Directions

The OEDD corpus reveals that current LLM agents struggle with long-context reasoning when required to ignore distracting information. While Gemini 1.5 Pro showed slightly more robustness than GPT-4o, all models exhibited a clear failure to reliably operationalize disparate experiential context. Future research should explore whether Retrieval-Augmented Generation (RAG) or specialized fine-tuning can mitigate these biases and improve the decision-making reliability of autonomous agents.

Summary updated

BibTeX
@inproceedings{george2024probing,
  author = {Sonny George and Chris Sypherd and Dylan Cashman},
  title = {{Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.905},
}
Figure