CT-SRCNN: Cascade Trained and Trimmed Deep Convolutional Neural Networks for Image Super Resolution

Haoyu Ren, Mostafa El‐Khamy, Jungwon Lee

Abstract

We propose methodologies to train highly accurate and efficient deep convolutional neural networks (CNNs) for image super resolution (SR). A cascade training approach to deep learning is proposed to improve the accuracy of the neural networks while gradually increasing the number of network layers. Next, we explore how to improve the SR efficiency by making the network slimmer. Two methodologies, the one-shot trimming and the cascade trimming, are proposed. With the cascade trimming, the network's size is gradually reduced layer by layer, without significant loss on its discriminative ability. Experiments on benchmark image datasets show that our proposed SR network achieves the state-of-the-art super resolution accuracy, while being more than 4 times faster compared to existing deep super resolution networks.

Introduction: The "Goldilocks" Problem in Super-Resolution

Single-image super-resolution (SR) is the task of recovering a high-resolution (HR) image from a low-resolution (LR) counterpart. While Convolutional Neural Networks (CNNs) have revolutionized this field, they often present a difficult trade-off. Shallow models, like the original 3-layer SRCNN, are computationally efficient but lack the receptive field necessary to reconstruct complex textures and sharp edges. Conversely, deep models like VDSR or DRCN offer superior image quality but are computationally expensive, slow on mobile hardware, and notoriously difficult to optimize due to vanishing gradients and sensitivity to hyperparameters.

CT-SRCNN aims to bridge this gap by introducing a "cascade" framework that simplifies the training of deep networks and optimizes them for real-time performance through structured trimming.

Methodology

1. Cascade Training

To overcome the optimization hurdles of deep networks, the authors propose Cascade Training. Instead of training a 19-layer network from scratch, the process begins with a simple 3-layer model. Training proceeds until the loss plateaus—specifically, when the error decrease falls below 3% per epoch. At this point, two new randomly initialized layers are inserted into the middle of the network.

Crucially, the existing weights are "inherited" from the previous stage. This incremental growth acts as a form of partial-supervised initialization, stabilizing the optimization process and eliminating the need for complex learning rate schedules or heavy weight decay tuning.

The Cascade Training process showing incremental layer addition and weight inheritance.

2. Cascade Trimming

Efficiency is achieved through Filter Trimming. Unlike standard weight pruning, which creates sparse matrices that are difficult for hardware to accelerate, filter trimming removes entire feature maps. This results in a "slimmer" but still standard dense CNN that runs efficiently on GPUs and mobile DSPs.

The authors introduce Cascade Trimming to prevent the "accuracy cliff" often seen in aggressive model compression. Rather than trimming the entire network at once, filters are removed from two adjacent layers at a time, followed by a fine-tuning phase. This gradual reduction allows the network to adapt its remaining weights to compensate for the lost capacity.

The Cascade Trimming process, illustrating the layer-by-layer reduction of filters to maintain accuracy.

Performance and Key Insights

Training at Scale

The authors utilized a subset of the Open Image Dataset (Thumbnail300KURL), generating over 30 million patches for training. This provides significantly more diversity than the standard 291-image datasets used in earlier research, contributing to the model's robust generalization.

Key Insights

  • The Deeper the Better (with a Strategy): Accuracy consistently improves with depth, provided the cascade training strategy is used. Without it, deeper networks often fail to converge to optimal solutions.
  • Incremental Growth as Regularization: Starting small allows the network to learn fundamental mappings first. Subsequent layers then refine high-frequency details without losing the global structure.
  • Train-Trim > Train-Thin: A vital finding is that training a high-capacity "fat" network and then trimming it yields better results than training a thin network from the start. The initial capacity allows the model to discover superior feature representations.

Benchmarks

The 19-layer CT-SRCNN achieved state-of-the-art results across standard benchmarks (Set5, Set14, B100):

  • Speed: It is 4x faster than VDSR and over 15x faster than recursive models like DRCN.
  • Efficiency: The model uses only 200K parameters, roughly 30% of VDSR's footprint.
  • Visual Quality: Qualitative results demonstrate sharper edges and significantly fewer artifacts compared to bicubic interpolation.

Conclusion

CT-SRCNN provides a practical blueprint for deploying high-quality image enhancement on resource-constrained devices. By combining incremental growth with structured, hardware-friendly trimming, the authors demonstrate that deep learning models can be both highly accurate and exceptionally fast. Future work may involve automating the growth and trimming schedules via Neural Architecture Search (NAS) to further optimize these "cascade" processes.

Summary updated

Abstract

We propose methodologies to train highly accurate and efficient deep convolutional neural networks (CNNs) for image super resolution (SR). A cascade training approach to deep learning is proposed to improve the accuracy of the neural networks while gradually increasing the number of network layers. Next, we explore how to improve the SR efficiency by making the network slimmer. Two methodologies, the one-shot trimming and the cascade trimming, are proposed. With the cascade trimming, the network's size is gradually reduced layer by layer, without significant loss on its discriminative ability. Experiments on benchmark image datasets show that our proposed SR network achieves the state-of-the-art super resolution accuracy, while being more than 4 times faster compared to existing deep super resolution networks.
BibTeX
@misc{ren2017ct,
  author = {Haoyu Ren and Mostafa El‐Khamy and Jungwon Lee},
  title = {{CT-SRCNN: Cascade Trained and Trimmed Deep Convolutional Neural Networks for Image Super Resolution}},
  year = {2017},
  eprint = {1711.04048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.04048},
}
Figure