Time-aware COMET: A Commonsense Knowledge Model with Temporal Knowledge

Eiki Murata, Daisuke KawaharaCOLING 2024

TL;DR

This paper introduces TaCOMET, a novel commonsense knowledge model that incorporates temporal information. By augmenting existing commonsense models with time intervals between events, TaCOMET demonstrates improved ability to generate contextually relevant inferences, particularly in time-sensitive applications like robot decision-making.

What Problem Does This Paper Address?

Modern language models excel at linguistic and factual knowledge but struggle with commonsense reasoning, which is often implicit and context-dependent. While commonsense knowledge graphs (like ATOMIC) and models (like COMET) have been developed, they typically overlook the crucial aspect of event granularity and temporal relationships. This omission leads to ambiguous or irrelevant inferences, especially when the timing of events significantly impacts their outcome. For instance, knowing that someone "puts on shoes" is less informative without knowing when this action occurs relative to other events, which could drastically alter the subsequent inference (e.g., "tie shoelaces" vs. "go for a run").

Methods

The authors propose a two-step approach to build TaCOMET:

  1. TimeATOMIC Dataset Creation: They leverage ChatGPT to construct a new commonsense knowledge graph called TimeATOMIC. This dataset augments existing ATOMIC triples (head, relation, tail) with temporal information, specifically the interval time between events. This allows for generating multiple, time-specific tails for the same head-relation pair.
  2. Continual Finetuning: An existing COMET model (a commonsense knowledge model) is then continually finetuned on the TimeATOMIC dataset. This process teaches the model to generate inferences that are sensitive to the input temporal interval, effectively maximizing P(tail | head, relation, time).

The key novelty lies in integrating temporal interval information into the commonsense inference generation process, moving beyond static event relationships.

Datasets

  • ATOMIC: A widely used commonsense knowledge graph focusing on event-to-event inferences and mental states. This served as the basis for creating TimeATOMIC.
  • TimeATOMIC: A newly created dataset by the authors, augmenting ATOMIC with temporal intervals between events. This dataset was generated using ChatGPT.
  • Robot Dataset (Tanaka et al., 2024): Used for evaluating TaCOMET's applicability in a downstream task for in-home robot decision-making.

Key Insights and Ideas

  • Time is Crucial for Commonsense: The paper strongly emphasizes that temporal granularity is a fundamental aspect of commonsense reasoning, and its absence leads to significant ambiguity.
  • ChatGPT as a Knowledge Augmenter: The authors demonstrate the effectiveness of using large language models like ChatGPT to efficiently create structured knowledge bases with temporal information, overcoming the limitations of manual construction.
  • Continual Finetuning for Temporal Awareness: The proposed finetuning strategy effectively imbues a pre-trained commonsense model with temporal sensitivity, allowing it to adapt its inferences based on time intervals.
  • Logarithmic Time Perception: Human evaluation suggests that language models, like humans, perceive time on a logarithmic scale, meaning smaller time differences (seconds/minutes) are more impactful than larger ones (months/years) for commonsense inference.

Key Results

  • Improved Generation Quality (BSGEN-REF): Finetuning on TimeATOMIC improved the generation performance of TaCOMET and TaCOMET SCRATCH (a baseline trained directly on TimeATOMIC) compared to the original COMET model.
  • Better Temporal Differentiation (BSINNER): TaCOMET and TaCOMET SCRATCH showed significantly lower BSINNER scores, indicating they generate more distinct inferences when the input time changes, demonstrating temporal awareness.
  • Effective in Downstream Tasks: In a robot decision-making task, TaCOMET with appropriate time inputs significantly outperformed models without time information, achieving higher accuracy and recall.
  • Model Size Matters: Larger models (XL) generally showed better performance in capturing temporal granularity and scale compared to smaller models (small).
  • Valid Generations Maintained: Despite being finetuned on temporal data, the finetuned models maintained a high percentage of valid generations, comparable to the original COMET.

Limitations and Future Work

  • Dataset Split: The evaluation used a split from a single dataset, and future work could explore more open settings where knowledge from initial training stages is leveraged.
  • Context and Relations: The authors suggest future work could incorporate broader context and explore additional types of temporal relations beyond simple intervals.
  • Multimodalization: Extending TaCOMET to handle multimodal inputs (e.g., images) is proposed as a future direction.
  • Generalizability: Applying TaCOMET to more general downstream tasks and exploring knowledge transfer to other language models are also suggested.

Summary updated

BibTeX
@inproceedings{murata2024time,
  author = {Eiki Murata and Daisuke Kawahara},
  title = {{Time-aware COMET: A Commonsense Knowledge Model with Temporal Knowledge}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure