Abstract
While Large Language Models (LLMs) are increasingly utilized as writing assistants, current benchmarks often fail to capture the iterative and knowledge-intensive nature of professional research writing. This paper introduces KIWI, a dataset designed to evaluate LLMs as iterative writing assistants for complex scientific tasks. Through a multi-turn interaction pipeline involving expert annotators, the study reveals that even state-of-the-art models like GPT-4 struggle with precise instruction following, frequently making unrequested changes or failing to integrate new information effectively.
Introduction
Large Language Models have become ubiquitous in writing workflows, with recent estimates suggesting that 10-20% of real-world LLM queries are related to writing assistance. However, most existing evaluation frameworks focus on single-turn tasks or creative writing prompts that do not reflect the rigor of scientific discourse. Research writing is inherently iterative; it requires grounding claims in specific literature and refining text through multiple rounds of feedback.
The authors identify a critical gap in our understanding of how LLMs handle specific revision instructions—such as adding evidence from a specific source or reorganizing a paragraph—while maintaining the integrity of the existing text. To address this, they present the KIWI dataset, focusing on the challenges of knowledge integration and constraint satisfaction in a research context.
The KIWI Dataset and Methodology
The researchers developed an interactive data collection pipeline that pairs expert human annotators (NLP researchers) with three prominent LLMs: GPT-4, GPT-3.5-turbo, and LLaMA-2-70b-chat. The process is structured as a multi-turn loop:
- Initial Drafting: The model generates a long-form answer to a research question based on retrieved passages from ACL 2023 papers. These papers were selected to ensure the content was published after the models' training cutoffs, preventing data leakage.
- Iterative Instruction: The human expert issues specific revision instructions, categorized as either Information-seeking (e.g., "add more evidence") or Stylistic (e.g., "change the tone" or "shorten the length").
- Model Revision: The model attempts to update the text based on the instruction.
- Expert Evaluation: The researcher rates the response as Good, Neutral, or Bad, providing a detailed explanation for the rating. If a model fails, the human can manually correct the text to ensure the conversation history remains high-quality for subsequent turns.
The resulting KIWI dataset comprises 1,260 interaction turns across 234 sessions, covering 78 unique research questions. Instructions are further categorized into fine-grained labels such as "Expand coverage," "Clarification," "Remove," and "Reorganize."
Key Results and Model Performance
The study highlights a significant performance gap between current LLM capabilities and the requirements of professional writing assistance.
- Success Rates: GPT-4 emerged as the most capable model, yet it only successfully followed instructions in 59% of cases. GPT-3.5 followed instructions 37% of the time, while LLaMA-2-70b-chat struggled significantly with a 27% success rate.
- Instruction Type Sensitivity: Models generally performed better on stylistic instructions than on information-seeking ones. For LLaMA-2, this gap was particularly pronounced: a 41% success rate on style versus only 15% on information integration.
- The Retrieval-Integration Gap: Interestingly, the study found that in 50% of information-seeking failures, the retrieval system had successfully provided the correct evidence. The failure lay not in finding the information, but in the LLM's inability to "surgically" integrate that information into the existing draft.
Analysis of Failure Modes
The researchers identified several recurring patterns in model failures that provide insight into the limitations of current LLMs:
The "Unrequested Change" Trap
The most frequent error, accounting for 31% of failures, was the model making changes the user did not request. For example, while attempting to shorten a paragraph, a model might inadvertently delete a crucial definition or a cited fact that was intended to remain.
Precision and Hard Constraints
LLMs struggle with precise structural constraints, such as placing a specific sentence at the very beginning of a text or adhering to strict sentence counts. This suggests that while models are proficient at general generation, they lack the fine-grained control required for professional editing.
Integration vs. Generation
The findings suggest that models are much more effective at writing a fresh draft from scratch than they are at modifying an existing one. Integrating new facts often resulted in "awkward flow" or imbalanced structures, indicating a lack of awareness regarding the existing text's rhetorical structure.
Limitations of Self-Evaluation
The study also tested GPT-4’s ability to act as an automated judge of its own performance. GPT-4 was found to be overly optimistic, with its agreement with human annotators lagging by 12%. It frequently failed to notice when it had missed subtle constraints or introduced unrequested deletions.
Conclusion and Future Directions
The KIWI dataset provides a rigorous benchmark for the next generation of writing assistants. While currently limited to the English language and the NLP research domain, the findings have broad implications for the development of more reliable iterative models.
Future work involves using the human-corrected versions of failed turns in KIWI as training data to fine-tune models for better iterative editing. Additionally, the researchers suggest that KIWI can be used to develop more "critically-minded" reward models that are better equipped to identify when an LLM has ignored a user's specific constraints or compromised the integrity of a document.
Summary updated