Abstract
This paper addresses the generation of stylized texts in a multilingual setup. A long short-term memory (LSTM) language model with extended phonetic and semantic embeddings is shown to capture poetic style when trained end-to-end without any expert knowledge. Phonetics seems to have a comparable contribution to the overall model performance as the information on the target author. The quality of the generated texts is estimated through bilingual evaluation understudy (BLEU), a new cross-entropy based metric, and a survey of human peers. When a style of target author is recognized by the humans, they do not seem to distinguish generated texts and originals.
Introduction
Most modern language models are trained on massive, diverse datasets. While this makes them versatile, it also makes them "average"—they tend to produce text that lacks a specific personality or "voice." In the realm of creative writing, particularly poetry, style is everything.
The challenge is that "literary style" is notoriously difficult to quantify. Experts can argue for hours about what makes Edgar Allan Poe sound like Poe, but translating those nuances into code is a hurdle. Furthermore, standard models often "forget" the intended style as a sentence gets longer. This paper tackles the problem of generating stylized text without relying on expert-defined rules, ensuring the model maintains a consistent "authorial persona" from the first word to the last.
Methodology
The authors employ a Long Short-Term Memory (LSTM) network with several architectural innovations designed to preserve stylistic integrity:
- Continuous Conditioning: Unlike standard models that might only be told which author to mimic at the beginning, this model is fed author and document embeddings at every single step of the generation process. This prevents the "forgetting" problem common in long sequences.
- Phonetic Embeddings: To capture the "euphony" (the pleasing sound) of poetry, the authors use two bidirectional LSTMs to create word representations. One processes the word character-by-character, while the other processes it phoneme-by-phoneme using International Phonetic Alphabet heuristics.
- Concatenated Representations: The final input for each word is a massive 1152-dimensional vector that combines the standard word embedding with the character-based state, the phoneme-based state, and the author/document metadata.
To measure stylization quality, the authors introduced a new metric: Sample Cross-Entropy. They calculate the cross-entropy between a set of generated texts and a set of original texts from the target author using a 3-gram language model with smoothing.
Datasets
The study utilized two large datasets of poetry to test the model's multilingual capabilities. The English corpus included approximately 110,000 documents from 19,000 authors, featuring diverse styles from Shakespeare to modern lyrics by bands like Nirvana and Muse. The Russian corpus was even larger in terms of document count, focusing on 1,700 authors including masters like Alexander Pushkin.
Key Insights and Results
The Importance of Phonetics
One of the most striking findings was that phonetic information is nearly as important as the author's identity itself. In ablation studies, adding phonetic embeddings improved the model's performance (measured by BLEU score) significantly more than just adding author metadata alone. Adding author information provided a +1% boost over a vanilla LSTM, but adding phonetics provided an additional +1.7% boost.
The "Umbrella" Advantage
The researchers found that training one large model on the entire multi-author corpus and then "restricting" it to a specific author worked better than training a small model on just one author's work. The model learns the general rules of language from the crowd but takes its stylistic cues from the individual.
Human Recognition and the "Turing" Moment
In a survey of 140 fluent English speakers, participants recognized Shakespeare’s style in the AI-generated text 37% of the time. Most impressively, when humans did recognize an author's style, they were often unable to distinguish between the AI and the original. For Shakespeare, 37% of respondents thought the AI-generated quatrain was the original, while 46% correctly identified the actual Shakespearean text—a remarkably narrow gap.
Conclusion and Future Work
While the model excels at syntax and phonetics, it does not yet incorporate high-level semantic standards or thematic constraints. For example, it cannot yet adhere to the specific topical requirements found in classical Japanese poetry. Additionally, performance drops for authors underrepresented in the training set, leading to occasional syntax errors.
Future research will focus on adding "pseudo-semantic rules" or external classifiers to monitor stylistic saliency, potentially moving from generating short quatrains to full-length, thematically consistent poems.
Summary updated