Linguistic Features of Helpfulness in Automated Support for Creative Writing

Melissa Roemmele, Andrew GordonNAACL 2018

Sure, here's a blog-style summary of the paper "Linguistic Features of Helpfulness in Automated Support for Creative Writing":

TL;DR

This paper explores how to automatically identify what makes AI-generated story continuations truly "helpful" to human writers. By analyzing user interactions with a creative writing assistant, they found that features like sentence coherence and semantic similarity to the story context are more important than simple sentence length or grammatical correctness. This work offers a way to evaluate and improve AI writing tools by focusing on features that implicitly guide human creativity.

What Problem Does This Paper Address?

Developing AI that can effectively assist human creativity, particularly in writing, is a significant challenge. While AI can generate text, it's hard to know if these suggestions actually help writers move their stories forward. Traditional evaluation metrics like BLEU scores, which measure similarity to "gold standard" answers, fall short because creative writing has many valid continuations. Simply asking users to rate suggestions is also limited, as it doesn't reveal why a suggestion was helpful. This paper tackles the problem of objectively measuring the "helpfulness" of AI-generated text in a creative writing context by analyzing how users interact with and modify these suggestions.

Methods

The core of this research lies in analyzing user interactions with an AI writing assistant called "Creative Help." This system allows users to request sentence suggestions and then freely edit them. The researchers leverage the system's ability to track these edits, creating pairs of original suggestions and their modified versions. They then quantify the "helpfulness" of a suggestion by measuring the similarity between the original suggestion and its modified form using Levenshtein edit distance. A higher similarity score indicates that the author found the suggestion helpful and made fewer changes.

To understand what drives this helpfulness, they extracted a set of linguistic features from the suggestions and their preceding story context. These features fall into two categories:

  • Story-independent features: These analyze the suggestion itself, including:
  • Sentence length
    • Grammaticality (using LanguageTool)
    • Lexical frequency
    • Syntactic complexity (number and length of noun and verb phrases)
  • Story-dependent features: These consider the suggestion in relation to the story context, including:
  • Lexical cohesion (Jaccard similarity, GloVe embeddings, skip-thought vectors)
    • Style consistency (part-of-speech tag similarity)
    • Sentiment similarity
    • Entity coreference

They then used Spearman correlation to measure how well these linguistic features predicted the similarity between the original suggestion and its modification, essentially quantifying how much each feature contributed to the suggestion being perceived as helpful.

Datasets

The study utilized a dataset collected from users interacting with the "Creative Help" system. This dataset consists of suggestion-modification pairs derived from 940 help requests made by 139 participants. The Creative Help system itself was trained on a subset of the BookCorpus, a large collection of fiction books.

Key Insights and Ideas

  • Helpfulness is more than just grammar: While grammatical correctness is important, it wasn't the strongest predictor of helpfulness. This suggests that AI suggestions don't need to be perfectly polished to be useful; they need to advance the narrative.
  • Context is king: Features that consider the relationship between the suggestion and the preceding story context were generally more predictive of helpfulness. This highlights the importance of semantic and stylistic coherence.
  • Semantic similarity matters most: Skip-thought vectors, which capture deep semantic meaning, showed the strongest correlation with helpfulness. This implies that suggestions that "understand" the story's meaning and continue it logically are highly valued by writers.
  • Noun phrases are good, verb phrases are neutral: Suggestions with more noun phrases were edited less, suggesting they contribute positively to story structure. Verb complexity, however, didn't seem to impact helpfulness.
  • Sentiment alignment is key: When the sentiment of a suggestion matched the sentiment of the story context, authors were more likely to accept it with fewer edits.
  • Entity coreference builds coherence: Suggestions that correctly linked entities (like characters or objects) to the existing story context were also found to be more helpful.

Key Results

The paper reports Spearman correlation coefficients (ρ) between various linguistic features and the similarity between suggestions and their modifications. Key findings include:

  • Skip-thought similarity had the highest correlation (ρ = 0.258), indicating it's a strong predictor of helpfulness.
  • Entity coreference (ρ = 0.134) and noun phrase count (ρ = 0.112) also showed significant positive correlations.
  • Sentiment similarity (ρ = 0.107) was also a helpful feature.
  • Features like sentence length (ρ = -0.082) and verb phrase complexity (ρ = -0.022) showed little to no predictive power.
  • Grammaticality (ρ = 0.097) was moderately helpful.

The average Levenshtein similarity score between suggestions and modifications was 0.695, suggesting authors generally retained a significant portion of the AI's suggestions.

Limitations and Future Work

The authors acknowledge that this analysis is based on a specific set of features and a particular creative writing application ("Creative Help"). They suggest that this approach can be scaled to evaluate any feature in an automated writing support framework where user interactions are logged. Future work could involve exploring a wider range of linguistic features, different AI generation models, and applying this methodology to other creative domains. The ultimate goal is to use this data-driven approach to build better AI tools that genuinely enhance human creativity.

Summary updated

BibTeX
@inproceedings{roemmele2018linguistic,
  author = {Melissa Roemmele and Andrew Gordon},
  title = {{Linguistic Features of Helpfulness in Automated Support for Creative Writing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}

Loading related content...

Figure