Zero-shot Trajectory Mapping in Holocaust Testimonies

Eitan Wagner, Renana Keydar, Omri AbendCOLING 2024

TL;DR

This paper introduces "Zero-shot Trajectory Mapping," a novel task for extracting and visualizing spatial narratives from text without prior location knowledge. They demonstrate a pipeline using large language models (LLMs) to create "maps" of locations and "trajectories" of movement within Holocaust testimonies, offering a new way to analyze and connect these historical accounts.

What Problem Does This Paper Address?

Understanding the spatial dimension of narratives – where events happen and how characters move through space – is crucial for a deep comprehension of stories. While NLP has made strides in identifying locations, extracting the progression or sequence of these locations (the "trajectory") from a narrative, especially without pre-defined lists of places, has been challenging. This is particularly true for complex, long-form texts like personal testimonies. The authors aim to bridge this gap by developing a method to automatically construct a spatial map from a collection of texts and then trace individual journeys within that map, all in a "zero-shot" manner, meaning without any human-provided labels or pre-existing ontologies.

Methods

The core of their approach is a two-stage pipeline leveraging the advanced capabilities of large language models (LLMs), specifically GPT-4 Turbo Preview, which boasts a large context window.

  1. Per-Testimony Location-Graph Extraction: For each testimony, the LLM is prompted to identify all mentioned locations (countries, cities, significant facilities like camps) and their relationships (e.g., "A is in B"). This information is structured into a graph (nodes for locations, edges for relationships) in JSON format. The LLM is instructed to categorize locations into predefined types (City, Country, Ghetto, etc.) and to infer inclusion relationships.
  2. Per-Testimony Trajectory Extraction: Using the previously generated location graph, a second LLM prompt asks for a sequence of locations visited within the testimony. Crucially, this output includes not only the sequence of locations but also the sentence number where each location is mentioned and, where inferable, the mode of transportation between locations.
  3. Combining Graphs into a Map: To create a unified "map," the individual location graphs are merged. A key step here is to identify and consolidate duplicate location names that might appear with different spellings or descriptions (e.g., "USA" and "United States"). This is also handled by the LLM. The final map is then visualized using the Networkx library.

The "zero-shot" aspect is critical: the system doesn't rely on pre-existing gazetteers or labeled datasets for location extraction or trajectory mapping.

Datasets

The primary dataset used is a collection of approximately 1000 Holocaust survivor testimonies from the Shoah Foundation (SF). These testimonies are time-stamped, transcribed texts. For their experiments, the authors arbitrarily selected a subset of 74 testimonies. No new benchmark datasets were introduced; instead, they applied their method to an existing, rich dataset.

Key Insights and Ideas

  • LLMs as Narrative Cartographers: The research demonstrates that LLMs, with their extended context windows, can effectively perform complex spatial reasoning and narrative extraction tasks without explicit supervision.
  • Unsupervised Spatial Mapping: The ability to build a location map and trace trajectories solely from raw text, without any prior knowledge of locations or their relationships, is a significant step towards more generalizable narrative analysis.
  • Holocaust Testimonies as a Rich NLP Domain: The authors highlight that Holocaust testimonies, due to their dense narrative structure and specific thematic focus, offer a unique and valuable domain for testing NLP capabilities in long-range understanding and spatial reasoning.
  • Potential for Deeper Analysis: The generated maps and trajectories are not just for visualization; they lay the groundwork for more advanced analyses like clustering testimonies by spatial patterns or aligning narratives based on shared movement paths.

Key Results

The pipeline was run on 74 Holocaust testimonies.

  • On average, each testimony yielded approximately 23 location nodes and 17 relationship edges.
  • The combined global graph comprised 883 nodes and 838 edges.
  • The average trajectory length extracted was 11 locations.
  • The authors present visualizations of the global map and example trajectories, showing how countries are represented as larger nodes and specific locations as smaller ones, with paths traced in red.

Limitations and Future Work

Limitations: * Name Consolidation: While the LLM assists in consolidating duplicate names, manual proofreading was still required for minor adjustments. * Granularity and Specificity: Zero-shot extraction is limited to locations explicitly mentioned in the text. If texts are less specific, the resulting trajectory might be shorter or less detailed. * Transportation Inference: The inference of transportation methods is based on textual cues and can be challenging.

Future Work: * Alignment and Clustering: Developing meaningful similarity measures between locations and trajectories to enable unsupervised clustering and alignment of testimonies. * Holocaust-Specific Features: Emphasizing and analyzing Holocaust-specific locations (ghettos, camps) within the spatial framework. * Dynamic Time Warping: Applying techniques like Dynamic Time Warping to derive trajectory-wise distances for optimal alignment. * Adaptation to Other Domains: Extending the zero-shot trajectory mapping task to other types of narrative texts beyond Holocaust testimonies.

Summary updated

BibTeX
@inproceedings{wagner2024zero,
  author = {Eitan Wagner and Renana Keydar and Omri Abend},
  title = {{Zero-shot Trajectory Mapping in Holocaust Testimonies}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure