You Don't Have Time to Read This: An Exploration of Document Reading Time Prediction

Orion Weller, Jordan Hildebrandt, Ilya Reznik, Christopher Challis, E. Shannon Tass, Quinn Snell, Kevin D. SeppiACL 2020 · OpenReview

Abstract: Orion Weller, Jordan Hildebrandt, Ilya Reznik, Christopher Challis, E. Shannon Tass, Quinn Snell, Kevin Seppi. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

2020.

Introduction

In the landscape of online publishing, "reading time" has become a critical metric for user engagement. Platforms like Medium often target a "sweet spot" of 3–7 minutes to maximize reader retention. However, most linguistic research has historically focused on the word level—analyzing how specific features, such as lexical surprisal, affect cognitive processing. This paper addresses a significant gap: whether these word-level insights scale to full-length documents. The authors investigate if machine learning can provide more nuanced document-level predictions than the standard "240 words per minute" (WPM) rule of thumb, or if factors like font size, genre, and readability significantly alter reading speed at scale.

Methodology

The researchers employed a two-pronged approach involving a large-scale human study and a multi-model machine learning comparison.

Experimental Design

To efficiently test multiple variables, the study utilized a Fractional Factorial Design (FFD). This statistical method allowed the team to test 8 different factors (including font size, genre, and readability) at two levels each using only 32 unique survey combinations, rather than the 256 required for a full factorial study. Data was collected from over 1,000 participants via Amazon Mechanical Turk, with strict controls such as demographic sanity checks, comprehension questions, and time thresholds to filter out outliers.

Modeling and Datasets

The study introduced the Document Reading Time Dataset, comprising 1,130 experimental trials across 32 different articles. The researchers compared four categories of models:

  1. Feature-only: Linear Regression (LR), Random Forests, and KNN using extracted features (e.g., font, readability scores).
  2. Text-only: Modern neural embeddings (XLNet, ELMo, roBERTa) to capture semantic meaning.
  3. Stacked: Combinations of neural embeddings and extracted features.
  4. Baselines: The industry-standard 240 WPM and a "Surprisal-Sum" model based on word-level cognitive load.

Results and Analysis

The performance of these models was evaluated using Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) in seconds.

Performance comparison of various models across feature-only, text-only, and stacked architectures.

As shown in the results, the Random Forest model slightly outperformed other methods, but the difference compared to the simple 240 WPM baseline and Linear Regression (word count only) was marginal. Notably, the "Surprisal-Sum" model—highly effective for word-level prediction—performed the worst at the document level, highlighting a disconnect between micro-level cognitive load and macro-level reading behavior.

Key Insights: The "Evening Out" Effect

The most significant finding is that complex neural models, which typically excel at NLP tasks, failed to beat simple baselines. The authors hypothesize an "evening out" effect: while difficult words or complex syntax cause tiny fluctuations in reading speed at the word level, these variations average out over the course of a 1,000-word article.

Scatter plot showing the strong linear correlation between article length (words) and reading time (seconds).

As illustrated in the scatter plot above, the volume of text is the primary driver of reading time. A simple Linear Regression model using only word count explained approximately 40% of the variance ($R^2 = 0.40$). Adding complex features like font style or subject matter provided negligible improvements, suggesting that for document-level reading, the "volume" of text is a far more critical predictor than semantic complexity.

Discussion and Limitations

While the 240 WPM rule remains a robust heuristic, the study acknowledges several limitations. The participant pool consisted of Mechanical Turk workers, who may be younger and more tech-savvy than the general population. Furthermore, the study focused on reading speed rather than comprehension depth; future work could explore the "efficiency" of reading—the threshold where speed begins to compromise understanding. Finally, while simple models currently lead, the authors suggest that significantly larger datasets might eventually allow neural networks to identify subtle patterns that linear models miss.

Summary updated

BibTeX
@inproceedings{weller2020you,
  author = {Orion Weller and Jordan Hildebrandt and Ilya Reznik and Christopher Challis and E. Shannon Tass and Quinn Snell and Kevin D. Seppi},
  title = {{You Don't Have Time to Read This: An Exploration of Document Reading Time Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=I9af-OhulNC},
}
Figure