Automatic Prediction of Aesthetics and Interestingness of Text Passages

Debasis Ganguly, Johannes Leveling, Gareth JonesCOLING 2014

TL;DR

This paper tackles the challenge of automatically predicting whether a text passage is aesthetically pleasing or interesting, a problem largely unexplored compared to image aesthetics. They introduce a novel approach using a "mapping convergence" algorithm that learns from positive examples only, overcoming the difficulty of obtaining reliably negative text samples. Their method achieves promising results, highlighting the importance of features like topic diversity and sentiment.

What Problem Does This Paper Address?

Predicting text aesthetics is a fascinating but difficult problem. Unlike images, where subjective beauty can be somewhat captured by visual features, text aesthetics are deeply tied to human interpretation and context. The authors are motivated by the availability of "popular highlights" from Amazon Kindle, which represent passages that a large number of readers found pleasing or interesting. The core challenge is to automatically identify these passages without relying on human annotators for negative examples, as simply labeling non-highlighted text as "unpleasant" is inaccurate. This is crucial for applications like recommending content or providing authors with feedback on their writing.

Methods

The authors propose a supervised classification approach but face a challenge: they have abundant "positive" examples (Kindle highlights) but struggle to reliably obtain "negative" examples (unpleasant text). To address this, they employ the Mapping Convergence (MC) algorithm. This algorithm is designed to learn from a mixture of positive and unlabeled data. It works in two stages:

  1. Mapping Stage: This stage uses one-class classifiers (like One-Class SVM or SVDD) to identify "strong negative" samples from the unlabeled data – those that are distinctly different from the positive examples.
  2. Convergence Stage: This is an iterative process where a binary classifier (like SVM) is trained using the positive samples and the identified strong negative samples. In each iteration, more strong negative samples are selected from the remaining unlabeled data, refining the classifier.

The paper also introduces a rich set of features to represent text passages:

  • Word-based: Word repetition (to capture emphasis) and average word length (longer words potentially indicating emphasis).
  • Topic-based: Topic diversity (using LDA to measure how many different topics a passage touches upon) and topic abstractness (measuring how representative words are of their topics).
  • Part-of-Speech (POS): Average number of adjectives and adverbs.
  • Sentiment: Aggregated absolute differences in sentiment polarity between words, weighted by proximity.
  • Inter-word Semantic Distance: Measures of semantic similarity between word pairs using the DISCO tool.

Datasets

The authors construct a novel dataset using Amazon Kindle "Popular Highlights." This dataset consists of:

  • Positive samples: Passages that were popularly highlighted by Kindle users.
  • Unlabeled samples: Paragraphs immediately preceding and succeeding the highlighted passages. These are used by the MC algorithm to identify potential negative samples.

The dataset was compiled by crawling the "Popular Highlights" web page and extracting text from books available on Project Gutenberg, focusing on 50 popular classic English fictions. The final dataset comprised 337 highlighted (positive) and 624 non-highlighted (unlabeled) passages.

Key Insights and Ideas

  • The difficulty of negative sampling: Simply treating all non-highlighted text as negative is flawed because these passages might be neutral or even interesting to some readers. The MC algorithm elegantly sidesteps this by learning to identify "strong negatives" from unlabeled data.
  • Topic diversity as a key aesthetic indicator: Passages that explore a broader range of topics, often philosophical or opinion-based, are more likely to be aesthetically pleasing. This suggests that a richer, more varied conceptual landscape contributes to interestingness.
  • Sentiment contrast matters: The presence of opposing sentiment words in close proximity (e.g., "best" and "worst") is a strong indicator of aesthetic appeal, likely due to the use of antithesis for emphasis.
  • MC algorithm improves separability: Visualizations show that the MC algorithm effectively refines the training set by retaining only strong negative samples, leading to better class separation compared to using all unlabeled data as negatives.

Key Results

The MC algorithm, particularly with a Gaussian kernel for mapping and a linear kernel for convergence, achieved the best results. This combination yielded:

  • Accuracy: Approximately 56%
  • Recall: Around 74%
  • Precision: Around 42%
  • F-score: Around 54%

The paper also highlights that topic diversity, sentiment, and word repetition were the most distinguishing features for identifying text aesthetics.

Limitations and Future Work

  • Limited dataset size: While novel, the dataset is relatively small (337 positive samples). The authors acknowledge that the OSVM and SVDD used in the mapping stage can be prone to over-fitting or under-fitting with small positive sample sets.
  • Accuracy vs. Recall: The authors emphasize that recall is more important than precision for this task, as the goal is to identify as many aesthetically pleasing passages as possible. However, the achieved precision is relatively low (42%), meaning a significant portion of identified passages might not be truly aesthetic.
  • Future work could explore:
    • More sophisticated feature engineering.
  • Larger and more diverse datasets.
  • Investigating the interplay between different features more deeply.
  • Exploring other algorithms for learning from positive and unlabeled data.

Summary updated

BibTeX
@inproceedings{ganguly2014automatic,
  author = {Debasis Ganguly and Johannes Leveling and Gareth Jones},
  title = {{Automatic Prediction of Aesthetics and Interestingness of Text Passages}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2014},
}
Figure