TL;DR
This paper introduces Mondrian, a framework that restructures academic abstracts into the "And, But, Therefore" (ABT) logical progression to improve clarity and coherence, especially for non-native English speakers. They also propose EB-DTW, a novel metric to better evaluate adherence to this structure.
What Problem Does This Paper Address?
Crafting a concise and logically flowing academic abstract is a significant challenge, particularly for non-native English speakers who may struggle with the nuances of academic writing conventions. While the ABT (And, But, Therefore) structure is a well-established rhetorical framework for organizing arguments, its application to abstracts can be difficult to achieve consistently. This leads to abstracts that may not effectively communicate the research's novelty and impact, hindering reader comprehension. The authors aim to bridge this gap by providing a systematic way to restructure abstracts into the ABT format, thereby enhancing their clarity and impact.
Methods
The Mondrian framework operates in two main stages:
- Deconstruction: An input abstract is first analyzed by a trained sentence classifier (using BERT or RoBERTa) to categorize each sentence into one of three ABT components: 'And' (background/context), 'But' (challenge/contradiction), or 'Therefore' (conclusion/impact). This classification is done by native English speakers.
- Restoration: The categorized sentences are then reordered to follow the ABT structure. A Large Language Model (LLM), specifically GPT-4 or Gemini-Pro, is used to naturally refine the reordered sentences, ensuring coherence and preserving as much of the original text as possible.
To quantitatively evaluate the effectiveness of ABT structuring, the authors introduce EB-DTW (Extrema-Bounded Dynamic Time Warping). This metric is an extension of traditional Dynamic Time Warping (DTW) that assigns weights to penalize deviations in key ABT segments, aiming to provide a more nuanced evaluation of structural coherence than standard DTW or Levenshtein distance.
Datasets
The authors created a dataset of approximately 1000 abstracts from papers that were under review/anonymous on OpenReview. These abstracts were manually annotated by native English speakers to label sentences as 'And', 'But', or 'Therefore' for training the sentence classifier.
Key Insights and Ideas
- ABT Structure Enhances Clarity: The core idea is that adhering to the ABT structure significantly improves the logical flow and comprehensibility of academic abstracts.
- Mondrian as a Tool for Equity: The framework is particularly beneficial for non-native English speakers, offering an equitable tool to improve their academic writing.
- EB-DTW's Nuance: The proposed EB-DTW metric is more sensitive to structural adherence than traditional metrics, especially for sequences with inherent ordinality and structure.
- LLMs as Refinement Tools: LLMs are crucial for the "restoration" phase, not just for reordering but for ensuring the final abstract reads naturally and cohesively.
Key Results
- Quantitative Improvements: Mondrian and Mondrian-ST (the self-training version) significantly reduced the DTW and EB-DTW scores, indicating better adherence to the ABT structure. On average, Mondrian-ST lowered the EB/DTW score by approximately 2.00.
- Qualitative Improvements: Human evaluations showed substantial improvements in content, organization, language, and overall coherence. For example, Mondrian-ST achieved an average improvement of 27.71% in organization and 24.71% in overall coherence.
- Non-Native Speaker Advantage: Improvements for abstracts written by non-native English speakers were even greater than the overall average improvements, particularly in organization and overall coherence.
- Abstract Preference: Human evaluators increasingly preferred abstracts processed by Mondrian and Mondrian-ST over the originals, especially in the Mondrian-ST setting.
Limitations and Future Work
- Data Bias: The dataset was limited to English abstracts from OpenReview, potentially introducing domain and language biases.
- Anonymity Assumption: While human evaluators identified abstracts likely written by non-native speakers, there's a possibility of misclassification.
- Generalizability: The authors suggest that the Mondrian framework could be extended to cross-linguistic text alignment by aligning larger segments rather than just sentences, to account for syntactic differences and contextual nuances.
- Plagiarism Detection: The EB-DTW metric's focus on structure and flow could be beneficial for detecting more sophisticated forms of plagiarism that involve reordering or rewording content.
Summary updated