Dataset Distillation for Handwritten Text Recognition: An Empirical Study

Félix Fuentes-Hurtado, Aric Ayllón, Juan C. AlfaroarXiv 2026 · arXiv

TL;DR

This paper presents the first empirical study of dataset distillation (DD) for line-level Handwritten Text Recognition (HTR). Despite adapting state-of-the-art distillation methods to handle sequence-based labels, the authors found that no distilled synthetic set outperformed the real images used for its initialization—suggesting that for HTR, a small set of random real lines is currently a stronger training signal than any "optimized" synthetic alternative.

What Problem Does This Paper Address?

Handwritten Text Recognition (HTR) models are computationally expensive to train and frequently require retraining to adapt to new handwriting styles or historical periods. Dataset Distillation (DD) offers a potential solution: it aims to condense massive datasets into tiny, highly informative synthetic sets that allow models to reach near-peak performance at a fraction of the training cost.

While DD has shown impressive results in image classification (e.g., turning thousands of CIFAR-10 images into just 10 "prototypes"), it hasn't been applied to HTR. HTR presents a unique challenge because it doesn't use discrete class labels. Instead, it deals with unaligned sequences (text) from a virtually infinite output space. This paper investigates whether standard DD techniques can be adapted to this complex, sequence-heavy domain.

Methods

The authors adapted three representative DD methods to a Convolutional Recurrent Neural Network (CRNN) architecture trained with Connectionist Temporal Classification (CTC) loss:

  1. Gradient Matching (GM): Matches the gradients of the model when trained on real vs. synthetic data.
  2. Distribution Matching (DM): Matches the mean feature embeddings of real and synthetic images.
  3. Matching Training Trajectories (MTT): Optimizes synthetic data so that a model trained on it follows the same parameter trajectory as a model trained on the full dataset.

To handle the lack of discrete classes, the authors introduced text pseudo-classes. They clustered transcriptions based on character n-grams (using TF-IDF weights and k-means) to group similar text lines together. They also established a rigorous "0-iter" control: comparing each distilled set against the exact same real images it started from, ensuring that any observed gains were truly due to pixel optimization and not just a "lucky" initial selection.

Datasets

The study evaluated four diverse line-level HTR datasets:

  • IAM: Modern English, hundreds of writers.
  • George Washington (GW): 18th-century English, only two writers.
  • READ 2016: Early Modern German, several hands.
  • LAM: 17th-18th century Italian letters from a single writer.

Experiments were conducted across three training budgets: 1%, 5%, and 10% of the original training lines.

Key Insights and Ideas

  • The "Starting Line" Dominance: The most significant finding is that performance is almost entirely driven by the initial selection of images. Distillation—the process of "optimizing" those pixels—provided no measurable benefit.
  • Random is Hard to Beat: Surprisingly, choosing a subset of random lines proved to be a stronger initialization than selecting "text medoids" (representative lines from the pseudo-classes).
  • The Readability Constraint: In standard image classification, a distilled "dog" image can become an abstract texture that a neural network recognizes as "dog-like." In HTR, a line must remain a specific, readable character sequence. Moving pixels too far from their original state quickly degrades the model's ability to recognize the text, as seen in Figure 4.
Figure 4. An IAM line (K = 308) at its starting point and after DMmed in batch mode with increasing optimisation strength (left), and its difference to the starting line (right). Drift and validation CER (one seed) refer to the whole buffer.
  • Methodological Rigor Matters: The authors highlight that without their "0-iter" control, one might mistakenly conclude that distillation works simply because the starting subset was already high-quality.

Key Results

The quantitative results were consistently "null":

  • No Improvement: Across all 12 experimental cells (4 datasets × 3 budgets), none of the five distillation methods significantly outperformed its own starting lines.
  • Performance Gap: Even at a 10% budget, all methods fell significantly short of the performance achieved by the full training pool. For example, on the IAM dataset, the best distilled sets hovered around 24% Character Error Rate (CER), while the full dataset reached 13.6% (see Figure 1).
Figure 1. Test CER against budget (mean, 95 % CI; points offset for legibility). Random: one subset per seed; DM, GM and MTT start from the same random lines. Dashed: full training pool.
  • Optimization Failure: As shown in Figure 2, the difference in CER between distilled sets and their 0-iteration controls is statistically zero, centered around the vertical axis.
Figure 2. Difference in test CER between each distilled set and its starting lines trained without optimisation (paired by seed; mean and 95 % CI). Filled: DM, GM and MTT from random lines; open: DMmed and GMmed from the medoids. Negative: distil- lation helps.

Limitations and Future Work

The authors acknowledge several limitations:

  • Fixed Transcriptions: They only optimized pixels, keeping the text labels fixed. Future work might explore "learning" the transcriptions or alignments alongside the pixels.
  • Embedding Simplicity: The Distribution Matching (DM) method used a simple averaging of features that might ignore the crucial temporal order of characters in a line.
  • Model Size: The study used a relatively small CRNN; larger models might potentially extract more information from optimized pixels.

The paper concludes with a set of guidelines for future HTR data compression research, urging researchers to always compare against 0-iter controls and to report pixel drift to ensure that "optimization" isn't just adding destructive noise.

Summary updated

Abstract

Dataset distillation (DD) replaces large training datasets with smaller synthetic ones to reduce computational costs. While traditionally used for classification, we present, to our knowledge, the first DD study for line-level handwritten text recognition (HTR). Unlike classification tasks, which use discrete labels, HTR employs unaligned sequences from a virtually infinite output space as labels. To adapt classification-based DD methods to this scenario, we define text pseudo-classes by clustering transcriptions based on character n-grams. We adapt gradient matching (GM), distribution matching (DM) and trajectory matching (MTT) to a Convolutional Recurrent Neural Network (CRNN) trained with Connectionist Temporal Classification (CTC). In this process, synthetic line images are generated by optimising only the pixels of real starting lines (either randomly chosen or selected as text medoids), while keeping their original transcriptions fixed. Across four datasets (IAM, George Washington, READ\,2016, LAM) and three budgets (1--10\,\% of the training lines), no distilled set outperforms its real initialisation lines. Random lines prove better than medoids, and further optimisation only degrades recognition. Methodologically, DM lowers its objective without benefit, while GM and MTT barely move theirs. We turn these findings into guidelines for evaluating data compression in HTR.
BibTeX
@misc{fuenteshurtado2026dataset,
  author = {Félix Fuentes-Hurtado and Aric Ayllón and Juan C. Alfaro},
  title = {{Dataset Distillation for Handwritten Text Recognition: An Empirical Study}},
  year = {2026},
  eprint = {2611.21054},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21054},
}
Figure