InsertGNN: A Hierarchical Graph Neural Network for the TOEFL Sentence Insertion Problem

Fang Wu, Stan Z. LiEMNLP 2024 · OpenReview

Abstract

The integration of sentences poses an intriguing challenge within the realm of NLP, but it has not garnered the attention it deserves. Existing methods that focus on sentence arrangement, textual consistency, and question answering have been shown to be inadequate in addressing this issue. To bridge this gap, we introduce InsertGNN which conceptualizes the problem as a graph and employ a hierarchical Graph Neural Network (GNN) to comprehend the interplay between sentences. Our approach was rigorously evaluated on a TOEFL dataset, and its efficacy was further validated on the expansive arXiv dataset using cross-domain learning. Thorough experimentation unequivocally establishes InsertGNN’s superiority over all comparative benchmarks, achieving an impressive 70% accuracy—a performance on par with average human test scores.

The Sentence Insertion Challenge

In standardized English proficiency tests like the TOEFL, students often face "Sentence Insertion" (SI) tasks. Given a paragraph with several marked slots and one "new" sentence that was removed, the objective is to determine the optimal placement. While humans use logic and linguistic cues to solve this, it remains a significant challenge for AI models.

Existing approaches typically treat SI as a sentence ordering problem or a question-answering task. However, these methods often fail to capture the subtle local transitions between sentences or the global logic of the entire passage. The authors argue that a linear text representation, standard in Transformers, is insufficient to comprehend the inherent logic connecting a paragraph to its designated slots.

Methodology: A Hierarchical Approach

The authors propose InsertGNN, which conceptualizes the paragraph as a directed graph where each node is a sentence and edges represent their relative order. The architecture is hierarchical, processing information at two distinct levels:

  1. Global Graph Attention Networks (GGN): This layer aggregates information from the entire paragraph to understand broad contextual semantics and the overall logic of the passage.
  2. Local Graph Convolutional Networks (LGN): Recognizing that the correct answer often depends on the sentences immediately surrounding a slot, the LGN focuses specifically on these local interactions. It uses subgraphs to extract structural "fingerprints" of the text flow.
  3. Global-Local Fusion: Finally, a fusion layer integrates the broad context from the GGN with the specific local transitions from the LGN to make a final prediction for each potential insertion slot.
The hierarchical architecture of InsertGNN, illustrating the flow from input textual graphs through GGN and LGN layers to a final fused output.

Experimental Setup and Datasets

The study utilizes two primary datasets to evaluate performance:

  • TOEFL Dataset: A high-quality, expert-curated set of 156 samples from actual TOEFL exams. This dataset includes human performance benchmarks, with average human accuracy at approximately 70.67%.
  • arXiv Abstracts: To address the small size of the TOEFL set, the authors curated 5,965 abstracts from scientific papers. These logically dense, well-edited texts are used for cross-domain learning.

Key Results and Insights

InsertGNN achieved a peak 71.43% accuracy on the TOEFL dataset, notably outperforming ChatGPT-3.5, which reached 61.52% even with optimized prompt engineering. This performance places the model on par with non-native English speakers taking high-stakes proficiency exams.

Performance comparison showing InsertGNN outperforming standard Transformer baselines like BERT and RoBERTa across different datasets.

A key insight from the study is that local cohesion is often more critical than long-distance context. The "Preceding Adjacent Vertex" (PAV) baseline, which only considers the immediate previous sentence, outperformed more complex unsupervised models, justifying the inclusion of the LGN component. Ablation studies confirmed this, showing that removing the LGN caused accuracy to drop significantly.

However, cross-domain challenges remain. When trained on arXiv and tested on TOEFL, performance dipped across all models, highlighting the stylistic differences between condensed academic abstracts and the expanded narratives found in TOEFL passages.

Conclusion

InsertGNN demonstrates that representing sentence insertion as a graph problem allows for more effective reasoning about potential sentence positions. By reaching 70% accuracy, it matches human-level performance on a complex linguistic task. Future work will focus on collecting larger high-quality datasets and testing the framework against even more powerful models like GPT-4.

Summary updated

Abstract

The integration of sentences poses an intriguing challenge within the realm of NLP, but it has not garnered the attention it deserves. Existing methods that focus on sentence arrangement, textual consistency, and question answering have been shown to be inadequate in addressing this issue. To bridge this gap, we introduce InsertGNN which conceptualizes the problem as a graph and employ a hierarchical Graph Neural Network (GNN) to comprehend the interplay between sentences. Our approach was rigorously evaluated on a TOEFL dataset, and its efficacy was further validated on the expansive arXiv dataset using cross-domain learning. Thorough experimentation unequivocally establishes InsertGNN’s superiority over all comparative benchmarks, achieving an impressive 70% accuracy—a performance on par with average human test scores.
BibTeX
@inproceedings{wu2024insertgnn,
  author = {Fang Wu and Stan Z. Li},
  title = {{InsertGNN: A Hierarchical Graph Neural Network for the TOEFL Sentence Insertion Problem}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=B0ULSHDLh4},
}
Figure