ContraDoc: Understanding Self-Contradictions in Documents with Large Language Models
Lists
Only you see this
Abstract
In recent times, large language models (LLMs) have shown impressive performance on various document-level tasks such as document classification, summarization, and question-answering. However, research on understanding their capabilities on the task of self-contradictions in long documents has been very limited. In this work, we introduce ContraDoc, the first human-annotated dataset to study self-contradictions in long documents across multiple domains, varying document lengths, self-contradictions types, and scope. We then analyze the current capabilities of four state-of-the-art open-source and commercially available LLMs: GPT3.5, GPT4, PaLM2, and LLaMAv2 on this dataset. While GPT4 performs the best and can outperform humans on this task, we find that it is still unreliable and struggles with self-contradictions that require more nuance and context. We release the dataset and all the code associated with the experiments (https://github.com/ddhruvkr/CONTRADOC).
BibTeX
@inproceedings{li2024contradoc,
author = {Jierui Li and Vipul Raheja and Dhruv Kumar},
title = {{ContraDoc: Understanding Self-Contradictions in Documents with Large Language Models}},
booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
year = {2024},
doi = {10.18653/v1/2024.naacl-long.362},
}Loading related content...