Tracking Entity State Transformations in Procedural Text

97 concepts60 papers3 proposals

When a standard natural language processing model encounters a sentence like "John bought an apple and ate it," resolving the pronoun is straightforward. The word "it" refers strictly and identically to "the apple." This is the foundation of traditional coreference resolution: tracking static entities that maintain their identity across a narrative.

But consider a recipe instruction: "Chop the carrots, dice the onions, and toss them into the pot. Boil the mixture for twenty minutes."

What does "them" refer to? What does "the mixture" denote? A human reader intuitively understands that "them" refers to the newly modified state of the carrots and onions, and "the mixture" is an entirely new entity born from their combination.

In procedural texts—such as recipes, chemical synthesis patents, and laboratory protocols—entities are non-stationary. They are continuously sliced, heated, dissolved, and merged. This introduces a fundamental challenge for artificial intelligence: dynamic entity state tracking. To understand instructions, models cannot treat entities as static placeholders; they must track how physical and chemical transformations alter the state, composition, and very existence of objects over time.

The Near-Identity Problem

The primary reason standard language models fail on procedural text is what researchers call the "near-identity problem." Traditional coreference resolution operates under a binary assumption: two mentions either refer to the exact same entity or they do not.

As Kyeongmin Rim et al. demonstrate in The Coreference under Transformation Labeling Dataset, this assumption breaks down when actions alter an object's physical state. A whole garlic clove and a tablespoon of minced garlic are linguistically related, but they are not identical. If a model treats them as a single static entity, it loses the granular map of the procedure.

To solve this, researchers have redefined how models map discourse. In What does it take to bake a cake?, Biaoyan Fang et al. propose a state-constrained coreference framework. Under this paradigm, a strict coreference link is only allowed if the entity remains in an identical state between mentions. If an action transforms the entity, the relationship is instead classified as a bridging relation.

[Carrots] --(Action: Chop)--> [Chipped Carrots] --(Action: Boil)--> [Cooked Mass]
  |                             |                                    |
  +---- Strict Coreference -----+------------ Bridging Relation -----+

This distinction is particularly vital in specialized domains like chemistry. In ChEMU-Ref, Biaoyan Fang et al. show that modeling these state-change relations allows systems to accurately extract chemical workflows. Without tracking these transitions, generic descriptors like "the residue" or "the resulting solution" become unresolvable black boxes, breaking the chain of inputs and outputs needed to reconstruct an experiment (Sopan Khosla et al. in ChemProp).

Why Global Hidden States Fail

Standard recurrent neural networks and transformers process text by updating a single, global hidden state. While this architecture is highly proficient at capturing surface-level textual patterns, it is fundamentally unsuited for tracking individual, evolving entities.

As Yangfeng Ji et al. explain in Dynamic Entity Representations in Neural Language Models, a global hidden state dilutes entity-specific attributes. When a text introduces multiple characters or ingredients, a standard model quickly loses track of which properties belong to which variable over long contexts. This architectural limitation causes models to misassign variables or fail to infer implicit effects—such as assuming an ingredient still exists in its raw form after it has been consumed by a mixing step (Haritz Puerto et al. in Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs).

To overcome this, modern state-tracking systems explicitly simulate a "world state." Early foundational work by Robert Dale in Cooking Up Referring Expressions pioneered this by generating new knowledge base representations every time an object transitioned from an individual item to a processed mass.

In deep learning, this is achieved by maintaining dedicated, dynamically updated vector memories for each participant in a text. For example, the ProPara dataset introduced by Bhavana Dalvi et al. in Tracking State Changes in Procedural Text forced models to explicitly predict the existence and location of entities at every single step of a process. By decoupling entity tracking from the global text representation—either through relative timestamp embeddings (Hossein Rajaby Faghihi et al. in Time-Stamped Language Model) or by mapping instructions directly into structured material flow graphs (Chaitanya Kulkarni et al. in Learning Latent Structures for Cross Action Phrase Relations in Wet Lab Protocols)—systems can maintain an accurate mental model of a changing environment.

Grounding Transformations in Vision

The ultimate test of state tracking occurs when language intersects with the physical world through computer vision. In instructional videos or multimodal documents, an object's visual features do not remain constant; a whole tomato looks radically different from a pool of simmered tomato sauce. Standard image-to-label matching fails because the visual appearance shifts completely.

[Whole Tomato Image]  --> Traditional Vision Model --> "Tomato"
                                                            |
                                                      (Appearance Shift)
                                                            v
[Tomato Sauce Image]  --> Traditional Vision Model --> "Sauce" (Identity Lost!)

To maintain identity across these drastic visual deformations, models must embed the procedural history of the object into its visual representation. In Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos, De-An Huang et al. demonstrated that by jointly modeling visual state changes and linguistic cues within a latent action graph, a system can track an entity's identity through appearance shifts that would normally cause standard segmentation models to fail.

This tracking is made possible by leveraging symbolic common-sense knowledge about actions. As Te-Lin Wu et al. show in Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge, if a model understands the pre-conditions and post-conditions of a verb (e.g., knowing that the action "slice" transforms a solid object into a collection of pieces), it can anchor its visual search to those expected transformations.

By combining linguistic tracking, state-aware coreference, and visual grounding, researchers are moving away from static text processing and toward AI systems that truly understand the physics of change.

Key Foundational Work

Jetr: a Robust Machine Translation System (1987) — contributed 3 concept elements to this direction.
Language Processing Via Canonical Verbs and Semantic Models (1973) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_entity_state_tracking_b840f02f,
  title = {entity state tracking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entity-state-tracking/rdr_b840f02f2efd4b68a9b64a66a0453070},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_entity_state_tracking_b840f02f_v1,
  title = {entity state tracking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entity-state-tracking/rdr_b840f02f2efd4b68a9b64a66a0453070/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake