Using Next Sentence Prediction to Test ChatGPT's Text Comprehension (Student Abstract)

Ojas M. Agarwal, Madelein Villegas, Jack MostowAAAI 2025 · DOI

TL;DR

This paper introduces the Next Sentence Prediction (NSP) task as a scalable, automated method to evaluate ChatGPT's text comprehension. By having the model distinguish the true next sentence from a distractor in children's stories, researchers can gain insights into how well LLMs understand narrative flow and coherence, with implications for educational tools.

What Problem Does This Paper Address?

Evaluating the text comprehension abilities of large language models (LLMs) like ChatGPT is crucial, especially for applications in education. Traditional methods can be labor-intensive and subjective. The authors aim to address this by developing a simple, objective, and scalable automated task that specifically tests an LLM's ability to understand the sequential nature of text and infer relationships between sentences. This is particularly relevant for educational tools like RoboTutor, which aims to teach literacy and numeracy to children. The core challenge is to measure how well an LLM can grasp the "intersentential processing" – the skill of understanding a sentence within the context of what came before it to establish coherence and draw inferences.

Methods

The paper proposes the Next Sentence Prediction (NSP) task. For a given context (a sequence of sentences from a story), the model is presented with two candidate sentences: the actual next sentence from the story and a distractor sentence from later in the same story. The task is to correctly identify which of the two is the true next sentence. This method is automated because the correct answer is inherently defined by the story's structure. The authors then use various statistical models (decision trees, gradient boosting, linear regression, logistic regression, random forest, and XGBoost) to analyze ChatGPT's performance on this task and identify which text features influence its accuracy.

Datasets

The primary dataset used is derived from 218 English-language children's stories from the African Storybook Project (ASP). From these stories, 12,674 NSP questions were automatically generated. The stories varied in length from 5 to 78 sentences. A smaller dataset of 1,010 NSP questions was generated from 15 Swahili stories from the same project to test the approach's cross-lingual generality. The English dataset was split into 80% for training and 20% for testing the predictive models.

Key Insights and Ideas

  • NSP as a Proxy for Human Comprehension: The NSP task is presented not just as a test for LLMs but as a potential proxy for human performance. Understanding what makes NSP questions easy or hard for ChatGPT could inform the design of questions that are also challenging for human readers, thus helping to identify reading difficulties.
  • Distractor Distance Matters: The further a distractor sentence is from the actual next sentence in the story, the easier it is for ChatGPT to distinguish it. This suggests that sentences closer in the narrative are more topically similar and thus harder to differentiate.
  • Distractor Length is Counter-intuitive: Surprisingly, longer distractor sentences made the task harder for ChatGPT. The authors hypothesize this might be because longer sentences provide more information, potentially making them seem more plausible or related, even if they are not the correct next sentence.
  • Context and Story Length are Not Significant Predictors: Unlike distractor characteristics, the length of the context provided or the overall story length did not significantly impact ChatGPT's performance on the NSP task.

Key Results

ChatGPT correctly answered 88% of the 12,674 NSP questions generated from English stories, which is significantly above the 50% chance level. For Swahili stories, the accuracy was 79%, though this was not statistically significant due to the smaller dataset. Logistic regression, used to predict correct answers, achieved 94% accuracy on held-out English test data. The significant predictors for correct answers were:

  • Distractor Distance: Further distractors led to higher accuracy (logit 0.063, p=0.047).
  • Distractor Length: Longer distractors led to lower accuracy (logit -0.057, p=0.001).

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Languages: The study was restricted to English and Swahili.
  • Genre Specificity: The data was limited to children's narrative fiction, so results may not generalize to other genres.
  • Independence Assumption: The analysis assumed statistical independence of NSP questions generated from the same story context, which might not hold true.

Future work could explore deeper text features (lexical overlap, sentence structure, semantics) to better understand ChatGPT's performance. Investigating ChatGPT's ability to explain its answers could also be valuable. While testing against human performance is desirable, the authors note that human performance variability requires significantly more data and is more costly to collect.

Summary updated

BibTeX
@inproceedings{agarwal2025using,
  author = {Ojas M. Agarwal and Madelein Villegas and Jack Mostow},
  title = {{Using Next Sentence Prediction to Test ChatGPT's Text Comprehension (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35238},
}
Figure