TL;DR
This paper introduces novel, unsupervised metrics to automatically evaluate the "knowledge depth" and "narration flow" of non-fiction narrative texts, specifically "Contextual Master (CM) stories." By identifying key knowledge markers and sentence categories, these metrics aim to provide actionable feedback for writers, improving the efficiency and quality of knowledge-rich document creation.
What Problem Does This Paper Address?
Creating high-quality non-fiction narrative texts, like those used to document technical solutions or business insights (termed CM stories), is crucial but challenging. While readability metrics exist, they don't capture the core value: the depth and clarity of the knowledge conveyed. Human review is time-consuming and subjective. This paper addresses the need for automated, objective evaluation of knowledge content and the logical flow of information within these texts, aiming to assist writers in producing better documents faster.
Methods
The authors propose a two-phase approach:
- Learning Phase: They leverage a corpus of "ideal" CM stories (those already revised and approved by human reviewers) to learn statistical norms for their proposed metrics. This establishes a benchmark for what constitutes good knowledge depth and narration flow.
- Operating Phase: For a new CM story, the system evaluates its metrics against the learned corpus statistics. Crucially, it also generates specific, actionable suggestions for improvement.
Key methodological components include:
- Knowledge Markers: Identifying entities like "Skills" (e.g., "SAP S4 HANA"), "Tasks" (e.g., "analyzed the configuration"), "Roles" (e.g., "IT Manager"), and "Concepts" (e.g., "unsupervised learning").
- Sentence Categories: Classifying sentences into types like "Background," "Problem," "Expert_Knowledge," "Solution," "Benefit," and "Client_Appreciation" using a DistilBERT-based multi-label classifier.
- Knowledge Quality Metrics: Quantifying the density of knowledge markers and sentence categories within a story.
- Narration Flow Metric (SCF): Measuring how well sentence categories are positioned relative to their typical occurrence in ideal stories, flagging sentences that appear in "unusual" positions.
- Aggregated Metrics: Combining individual metrics into overall scores using Mahalanobis distance (Distmean) and a scaled sum (Zsum) to represent overall knowledge quality.
Datasets
- Training Corpus (Dtrain): A large corpus of 53,675 final, human-reviewed CM stories (1.4 million sentences, 28.8 million words). This dataset is used to learn corpus statistics for the proposed metrics.
- Evaluation Dataset (Deval and Deval): A smaller set of 67 CM stories, each with an initial version (written by the author) and a final version (revised based on human feedback). This paired dataset allows for evaluating how well the proposed metrics align with human judgment of improvement.
Key Insights and Ideas
- Beyond Readability: The paper highlights that traditional readability scores (like FRES) are insufficient for evaluating knowledge-rich texts, as optimal knowledge density might sometimes slightly decrease readability.
- Structure Matters: The "flow" of information, specifically the sequence and placement of different sentence types (e.g., problem, solution, benefit), is a critical indicator of a well-structured narrative.
- Unsupervised Learning: The proposed metrics are unsupervised, meaning they don't require manually annotated "quality scores" for training, making them scalable.
- Actionable Feedback: The system doesn't just score; it provides concrete suggestions (e.g., "Please add more details about the specific benefits achieved") that directly address metric deficiencies.
- Alignment with Human Judgment: The evaluation strategy, using paired initial/final versions and t-tests, aims to show that the proposed metrics correlate with human reviewers' assessments of story improvement.
Key Results
The paper demonstrates that their aggregated knowledge quality metrics (Distmean and Zsum) and the narration flow metric (SCFall) show statistically significant improvements when comparing final CM stories to their initial versions. This indicates that these metrics effectively capture aspects that human reviewers deem important for story quality. In contrast, baseline metrics like FRES and Essay Grading (EG) did not show significant improvement for final versions, and in some cases, even assigned better scores to initial versions, suggesting they don't capture the same quality aspects. The SCF metric, unlike perplexity which focuses on local sentence sequences, effectively identifies unusual sentence placements across the entire story.
Limitations and Future Work
- Malicious Intent: The current knowledge quality metrics are dependent on the density of markers but don't explicitly check for relevance or potential "gaming" of the system by adding irrelevant entities. This is acknowledged as a future work area.
- Adaptability: The authors plan to adapt these metrics to other types of non-fiction narrative texts (e.g., security incident reports).
- Discovering Categories: A future direction is to automatically discover key sentence categories for new document types.
- New Flow Metrics: Exploring new narration flow quality metrics, such as those based on sequence entropy, is also planned.
- User Studies: While initial feedback is positive, detailed user studies are planned to further validate the system's utility.
Summary updated