TL;DR
This paper introduces a novel method for predicting book success by analyzing semantic word associations within a novel's text, leveraging word embeddings and Roget's Thesaurus. Their approach significantly outperforms previous methods, achieving a state-of-the-art weighted F1-score of 0.89, offering a more nuanced understanding of what makes a book resonate with readers.
What Problem Does This Paper Address?
The publishing industry faces a significant challenge in predicting which books will become popular. With a vast number of books published annually, only a fraction achieve widespread success. While factors like plot, character development, author reputation, and marketing play a role, this paper focuses on a less explored area: the intrinsic semantic content of the book itself. The authors aim to quantify the "literary content" by analyzing how words within a book associate with broader semantic concepts, and how these associations correlate with a book's success. This addresses the need for a more objective, data-driven approach to predicting literary success beyond traditional qualitative assessments.
Methods
The core of this research lies in using semantic word associations derived from word embeddings. Here's a breakdown of their approach:
- Word Embeddings: They generate word embeddings for each individual book using the
fastTextmodel, which is adept at handling rare words and out-of-vocabulary terms due to its use of character n-grams. - Global Embedding: A global
fastTextembedding model, trained on Wikipedia, serves as a reference. - Embedding Alignment: Crucially, they align the book-specific embeddings to the global embedding space. This allows for a consistent comparison of word meanings across different books and the global corpus. They experimented with several alignment methods and settled on
VecMapfor its effectiveness. - Association Score Calculation: For each book, they compute an association score between its content and a predefined set of Roget's Thesaurus concepts. This involves calculating the average Euclidean distance between the book's aligned word vectors and the global vectors for words within each Roget concept.
- Feature Representation: Each book is then represented as a vector of these association scores, with each dimension corresponding to a Roget concept.
- Classification: A weighted linear Support Vector Machine (SVM) is used as the classifier to predict book success (successful/unsuccessful).
- Feature Selection: To optimize performance and identify the most influential concepts, they employ a filter-based feature selection method, ranking features by their relevance and selecting the top ones.
What's novel here is the systematic application of word embedding association tests, specifically adapted to represent entire books as concept association spectra, and then using these spectra to predict literary success.
Datasets
The primary dataset used in this study is the one introduced by Maharjan et al. (2017). It comprises 1,003 books downloaded from Project Gutenberg. Each book is labeled as either successful or unsuccessful based on Goodreads ratings (successful if rated by at least 10 users with a rating ≥ 3.5 out of 5). The dataset is divided into eight genres.
Key Insights and Ideas
- Semantic Associations Trump Frequency: The study strongly suggests that the semantic relationships between words and broader concepts are far more predictive of book success than simple word frequency counts (e.g., using PoS tags, LIWC categories, or NRC sentiment).
- Roget's Thesaurus as a Powerful Lexicon: The structured semantic categories within Roget's Thesaurus, when combined with word embeddings, provide a rich feature set for understanding literary content.
- Genre-Specific Success Factors: Different genres exhibit distinct patterns of semantic associations that contribute to success. For instance, "Detective and Mystery" novels might favor themes related to character choices and feelings over purely logical deduction.
- The Power of "Less is More" (Sometimes): Feature selection reveals that while a large number of concepts can be used, performance often plateaus or declines after a certain point, indicating that a curated set of the most discriminative features is highly effective.
- Beyond Surface-Level Features: This approach moves beyond surface-level linguistic features (like word counts or sentiment scores) to capture deeper thematic and conceptual underpinnings that resonate with readers.
Key Results
- State-of-the-Art Performance: The proposed method, using semantic word associations from Roget's Thesaurus, achieved a weighted F1-score of 0.89, significantly outperforming previous baseline methods (best baseline at 0.75) and other frequency-based approaches.
- Outperforming Other Methods: Roget's Association Score method achieved an average weighted F1 score of 0.88, beating the baseline results and the state-of-the-art result obtained using Roget's word group frequency.
- High Precision-Recall AUC: For most genres, the precision-recall curves showed an Area Under the Curve (AUC) above 0.90, indicating robust performance even on the imbalanced dataset.
- Discriminative Features Identified: The study identified specific Roget concepts that are highly discriminative for success in different genres, providing qualitative insights into reader preferences. For example, "Intellectual Faculties" was found to be important for Science Fiction, while "Voluntary Powers" and "Sentiment and Moral Powers" were associated with successful "Detective and Mystery" stories.
Limitations and Future Work
- Dataset Size and Scope: While 1,003 books is a substantial dataset, the authors acknowledge that experiments on a larger dataset would be beneficial.
- Genre-Specific Embeddings: The authors hypothesize that retraining global embeddings using genre-specific data could lead to even better results by capturing genre-specific contexts more explicitly.
- Further Exploration of Concepts: The study focused on Roget's Thesaurus; future work could explore other semantic lexicons or develop custom concept sets.
- Deeper Analysis of Success Factors: While discriminative features were identified, a deeper dive into why these specific associations lead to success could be explored.
- Predicting Other Literary Qualities: The methodology could potentially be extended to predict other literary qualities beyond just success, such as reader engagement, emotional impact, or thematic depth.
Summary updated