Track the Distractors, Don't Delete Them: Entity State Tracking Under Entangled Noise

Language models are increasingly asked to follow processes written in text: a recipe, a lab protocol, a description of how rock erodes. To answer "where is the dough now?" or "what did the reagent become?", the model has to do entity state tracking, meaning it keeps a running record of what each object is, where it is, and whether it still exists as the steps go by. Real procedural text is rarely clean. A recipe blog mixes steps with anecdotes, a protocol describes two reactions in parallel, and a transcript mentions the same ingredient in a pot that doesn't matter. Work on contextual noise sensitivity shows that models fall apart when irrelevant text resembles the real target. A vision-language-action robot model, OpenVLA, drops from 77.5% to 18.9% task success on LIBERO-Goal when the command mentions other objects in the scene (Bring the Apple, Not the Sofa). The standard fix is to strip the noise before reasoning, either with a small language model filter or a summarizer, or by pruning unimportant entities.

This note argues that the standard fix has a blind spot, and it is exactly the case procedural text produces most. Some distractors can only be recognized as irrelevant once you know the entities' history. Take "stir the mixture": whether it concerns our dough or the sauce in the other pot depends on which ingredients went where three steps earlier. Deciding what is relevant then requires state tracking, so a filter that runs before tracking is solving the hard problem blind. We propose two things. The first is a benchmark that grades distractors by how much state knowledge is needed to reject them. The second is a tracking method that keeps a record of every entity's lineage, distractor entities included, and conditions each relevance decision on that record. The claim is that on history-dependent distractors, you have to track the distractor in order to ignore it.

Core Claim

We define a distractor as entangled when deciding that it does not affect the target entity requires knowing the lineage of entities so far. Lineage here means which items were combined, split, or transformed into which. For surface distractors, such as anecdotes, politeness, or unrelated topics, relevance can be judged from the sentence alone, and pre-filtering should work about as well as the 98.5% recovery reported by Bring the Apple, Not the Sofa. For entangled distractors we predict three things. Pre-filters, static salience pruning, and mention-weighted decoding will all degrade sharply. A tracker that keeps a partitioned lineage record of all entities and gates each update on it will stay close to its clean-text accuracy. The ingredient that matters is keeping and partitioning the distractor entities, as opposed to deleting them.

Positioning Against Prior Work

Several papers motivate the proposal by exposing the gap. Bring the Apple, Not the Sofa finds that similarity to real scene objects is what makes irrelevant context harmful. Its commands are single-step, though, and its fix is a filter that ignores state, so it never meets a distractor whose relevance changes over time. OpenPI2.0 shows that tracking only high-salience entities in chain-of-thought prompts shortens them by 14% and improves downstream procedural question answering. Its salience is a fixed per-entity score, and it does not handle low-salience items that merge into salient ones. Our benchmark tests that case directly. MuSiQue builds hard distractors that share surface entities with the true reasoning chain, but its facts do not change over time.

Other papers constrain the design. MEET reaches 73.1 document-level F1 on ProPara and 66.0 F1 on Recipes. It does this by weighting decoding toward steps where the entity is explicitly mentioned. That bias is a noise defense, and it is exposed whenever a distractor step mentions the target by name without acting on it, or acts on it only through "the mixture." Superposition Prompting processes context pieces in parallel and prunes the irrelevant ones. That breaks the step-to-step dependency tracking requires, so we rule out parallel pruning as a design. The ProPara authors report that implicit effects already cause 37% of errors (Tracking State Changes in Procedural Text). Any method that drops text risks adding to this category.

Other work supplies the method. From the CUTL dataset paper we reuse the distinction between coreference under identity (same entity, unchanged) and coreference under transformation (same entity after a physical change). We also reuse its "phantom" result nodes, which name unmentioned intermediate mixtures. From NCET we reuse the six-tag lifecycle (non-existent before, create, exist, move, destroy, non-existent after) and its constraint that impossible sequences, such as moving after destruction, are disallowed. From Code Prompting we take the observation that writing entities as explicit variables cuts state-tracking memory errors by 30% to 60% in GPT-3.5. Our record is serialized in that style.

Research Program

The first component is a graded distractor benchmark built on ProPara, Recipes, and the Boxes and Objects task. We keep the original gold labels and insert text that provably leaves the target entities' states unchanged. There are four tiers. Tier one is surface noise: anecdotes and filler that do not mention the target entities. Tier two is same-name, non-acting mentions, as in "garlic, which my grandmother grew, is best fresh." Tier three is a parallel process that shares entity types and uses anaphora. We interleave a second paragraph or recipe, marked in the discourse as happening elsewhere, so that "the water" or "the mixture" is resolvable only through lineage. Tier four is lineage entanglement: low-salience distractor ingredients that merge into a separate phantom mixture, which is later referred to with the same generic noun used for the target's mixture.

For Boxes and Objects (Entity Tracking in Language Models) we generate all four tiers programmatically. We add a second room of boxes holding the same object names, which gives exact gold labels and lets us sweep the number of distractor operations. For the natural-text versions, two annotators confirm that each noisy item is still answerable, and we report human accuracy per tier. This matters because a method-by-tier interaction only means something if humans stay near the ProPara human bound of about 80% on every tier. The ProPara authors report that bound (Tracking State Changes in Procedural Text).

The second component is the tracker, which we call the partitioned lineage ledger. The model reads the text one step at a time and maintains a ledger, a code-style record of every entity it has seen. Each entry holds a lifecycle tag, a location, and lineage links drawn from the CUTL relation types, including phantom result nodes. Entries are grouped into process clusters by lineage. At each step the model receives the current ledger and the new sentence. It first names which ledger entries, if any, the sentence acts on, and only then writes the update. The update applies to whichever cluster it belongs to, distractor clusters included. Nothing is deleted. At answer time, only the target entity's lineage cluster is shown to the answering step. NCET's lifecycle constraint is enforced on each entry, so the ledger cannot, for example, move a destroyed entity.

We build two versions. One is prompted, on open models of the kind used in Code Prompting, namely Mistral 7B and Mixtral 8x7B. The other is fine-tuned on a T5 backbone comparable to MEET's, trained on tier-free text plus synthetic Boxes and Objects ledgers.

Baselines

The naive reference point is the same backbone given the full noisy text with ordinary step-by-step reasoning and no ledger. Its clean-text score is the ceiling we measure degradation against, and the gap between its clean and noisy scores defines the noise penalty for each tier.

The strongest existing fine-tuned tracker is MEET (Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decoding). It reports 73.1 document-level F1 on ProPara and 66.0 F1 on Recipes on clean text; no reported result found on Lacuna for any tracker on distractor-injected procedural text. We run MEET with its released recipe and backbone size in two conditions: trained on clean data only, and trained on the same distractor-augmented training split our fine-tuned ledger sees. The tuning budget is identical across the two. To beat it, the ledger must match it within 1 F1 on clean text and exceed augmented MEET by a margin outside the paired-bootstrap confidence interval on tiers three and four.

The strongest noise-mitigation competitor is a target-aware pre-filter in the style of Bring the Apple, Not the Sofa, which restored up to 98.5% of clean performance with a 3B model. To give it every advantage, we tell the filter which entity will be queried. We run it at two sizes: a 3B model, and the same model used as the tracker. The filtered text then goes to the naive reasoner and to MEET. Beating it means recovering a larger share of clean accuracy on tiers three and four, with total tokens reported so any cost difference is visible.

We also run code prompting (Code Prompting) without our gate or partitioning, and OpenPI2.0-style salience pruning (OpenPI2.0), which keeps only entities an LLM rates as high-salience. Two ablations isolate the key ingredient. The first, a target-only ledger, records only entities judged relevant and drops distractor entities; it is our method with deletion instead of partitioning. The second is a full ledger without the relevance gate, where updates are written directly with no prior step naming which entries are acted on. If the target-only ledger performs as well as the partitioned ledger, the central claim is false.

Why not just…?

Why not just pre-filter the text with a small language model? This is the most serious threat. On tiers one and two we expect it to work, and if it also works on tiers three and four our idea is unnecessary. The reason for doubt is structural. On entangled items the filter must decide whether "stir the mixture" concerns the target, and that requires lineage the filter does not have. The summarization study found a similar ceiling: benefits shrink to about 2% on the hardest problems, where reasoning rather than noise is the bottleneck (Less is More). The target-aware, same-size filter baseline tests this directly. A tier-three or tier-four recovery within a few points of the ledger would settle the question against us.

Why not just use code prompting, which already isolates entities as variables? Code prompting makes state explicit, but it writes every entity into one shared namespace. It also has no step that asks whether a sentence acts on a given variable before updating it. Code Prompting never tested inputs containing distractor entities. Our ledger is essentially code prompting plus lineage partitioning plus a gate. The no-gate ablation and the plain code-prompting baseline show how much of the gain comes from the representation alone. If plain code prompting already holds up on tier four, we would report that as the finding.

Why not just fine-tune an existing tracker like MEET on distractor-augmented data? Augmentation may teach the model to ignore same-name mentions in tier two. Mention-guided decoding, however, rewards steps that explicitly mention the target, and tier three is built so that the acting step uses only a generic noun. The augmented-MEET baseline measures whether training data alone closes this gap. We will also report results on held-out distractor templates, because augmentation that only memorizes templates would inflate in-distribution scores.

Why not just use a frontier long-context model? Longer windows do not remove attention dilution. The Boxes and Objects study already reports that GPT-4's tracking degrades as the number of operations grows (Entity Tracking in Language Models). We include one frontier model as the naive reasoner on the synthetic tiers. Its accuracy as distractor operations increase will show whether scale alone flattens the curve.

Proposed Experiments and Decisive Checks

We report ProPara Task-1 macro-average and Task-2 F1, where NCET reported 53.93 and 62.5 (Tracking Discrete and Continuous Entity State), along with Recipes F1 and Boxes and Objects accuracy. Each is broken down by tier. Alongside them we report a robustness ratio, defined as noisy accuracy divided by clean accuracy for the same system, and total tokens per question.

The decisive check is the method-by-tier interaction. The partitioned ledger should hold a robustness ratio near one across all four tiers, while the pre-filter and the target-only ledger should drop specifically at tiers three and four. The synthetic room-of-boxes sweep gives the cleanest version of this test. Plotting accuracy against the number of distractor operations that share object names should give a flat curve for the partitioned ledger and a falling one for deletion-based methods.

A second check targets the gap left open by OpenPI2.0. On tier-four items where a low-salience distractor merges into a phantom mixture, we measure how often salience pruning and the target-only ledger misattribute that mixture to the target. A third check is error analysis on the 37% implicit-effect category from Tracking State Changes in Procedural Text, to confirm the gate does not suppress real implicit updates to the target.

The result that would falsify the proposal is a target-aware pre-filter or a target-only ledger coming within noise of the partitioned ledger on tiers three and four, with humans still accurate on those tiers.