Abstract
Understanding physical dynamics in natural language remains a significant challenge for modern Natural Language Understanding (NLU) models. This paper introduces LiMiT (Literal-Motion-in-Text), a large-scale dataset consisting of over 24,000 human-annotated sentences. LiMiT is designed to help models distinguish between literal physical motion and figurative expressions, while also identifying the specific entities in motion. By leveraging diverse sources such as 19th-century fiction and video captions, the authors establish strong baselines for motion classification and entity tagging, demonstrating that motion is a fundamental component of many existing NLU benchmarks.
Introduction
A core objective in NLU is grounding language in the physical world. While tasks like sentiment analysis have matured, identifying literal physical motion remains difficult due to semantic ambiguity. For example, a "voice twisting with incredulity" (figurative) is linguistically similar to a "player twisting his leg" (literal).
The LiMiT dataset is grounded in the theory of "semantic primes"—irreducible concepts like MOVE that are universal across languages. Prior to LiMiT, the NLP community lacked a large-scale resource to study how motion is expressed, with existing datasets being significantly smaller or overly focused on specific spatial roles. LiMiT addresses this by providing a dataset 13 times larger than previous efforts.
Methodology
The construction of LiMiT involved a multi-stage pipeline designed to ensure both scale and quality.
Data Sourcing and Filtering
Sentences were pulled from two distinct domains: 19th-century fiction e-books, which offer complex and descriptive language, and the Net Activity Captions (NAC) dataset, which provides direct video descriptions. To isolate potential motion, the researchers filtered these sources using a comprehensive list of over 1,200 motion verbs based on Levin’s (1993) classification.
Crowdsourcing and Bias Mitigation
A significant challenge in annotating motion is "animacy bias"—the human tendency to focus on people or animals moving while overlooking inanimate objects. To mitigate this, the researchers implemented a specialized two-design crowdsourcing strategy.
In the second design, annotators were explicitly required to identify inanimate entities (e.g., balls, cars) before labeling animate ones. This intervention successfully increased inanimate entity labels by 70%. Final labels were refined using an Expectation-Maximization (EM) strategy, which outperformed simple majority voting by approximately 4%.
Dataset Characteristics
The resulting LiMiT dataset contains 24,559 sentences, with 15,346 instances of literal motion and 9,213 instances of no motion (including figurative or imaginative motion). Annotations include both binary classification and span-based entity tagging. The dataset achieved a Fleiss Kappa of 0.71, indicating substantial agreement among human annotators.
Experimental Results
The authors established baseline performances using state-of-the-art architectures:
- Motion Classification: Identifying whether a sentence describes literal motion reached an F1-score of 77% using the Watson Natural Language Classifier.
- Entity Tagging: A BERT-based model achieved a 70.9% F1-score in identifying the "who" or "what" of the motion.
Analysis revealed that motion is pervasive in standard NLP benchmarks; over 50% of sentences in the SNLI and SICK datasets involve motion. This suggests that improving a model's understanding of physical dynamics could yield broad benefits across logic and reasoning tasks.
Conclusion and Future Work
LiMiT provides a new foundation for evaluating how models understand physical dynamics. Future work may expand the dataset to include additional motion components such as path, source, goal, and direction. Furthermore, the dataset holds potential for improving text-to-scene generation, where precise understanding of moving entities is critical for accurate animation.
Summary updated