Computer-Aided Colorectal Tumor Classification in NBI Endoscopy Using CNN Features

Toru Tamaki, Shoji Sonoyama, Tsubasa Hirakawa, Bisser Raytchev, Kazufumi Kaneda, Tetsushi Koide, Shigeto Yoshida, Hiroshi Mieno, Shinji TanakaarXiv 2016 · arXiv

TL;DR

Pre-trained CNNs can act as highly effective "off-the-shelf" feature extractors for specialized medical image classification without any fine-tuning. Surprisingly, features extracted from early convolutional layers achieve up to 96.9% accuracy in classifying colorectal tumors from Narrow-band Imaging (NBI) endoscopy, matching the performance of complex, hand-crafted texture pipelines.

What Problem Does This Paper Address?

Colorectal cancer screening relies heavily on endoscopic evaluations. Technologies like Narrow-band Imaging (NBI) enhance the visibility of microvessel texture patterns on mucosal surfaces, helping doctors distinguish between benign and malignant tumors. However, accurately interpreting these NBI magnification findings is challenging for non-experts and trainees.

While Computer-Aided Diagnosis (CAD) systems can support decision-making, traditional approaches rely on complex, hand-crafted texture descriptors (such as SIFT, Bag-of-Visual-Words, Fisher Vectors, or VLAD). Deep learning offers an alternative, but training a convolutional neural network (CNN) from scratch requires massive, expertly labeled medical datasets that are rarely available. The authors investigate whether features from standard CNNs pre-trained on everyday objects (ImageNet) can be repurposed for this highly specialized medical task without any fine-tuning.

NBI endoscopic images and medical categories

Methods

The authors evaluate three popular pre-trained CNN architectures: AlexNet, Caffe Net, and GoogLeNet. Instead of using these networks for direct classification, they extract raw feature vectors from various intermediate layers (convolutional, pooling, and fully connected).

Fig. 3. CNNs used for experiments: (left) AlexNet, (middle) Caffe Net, and (right) GoogLeNet. Gray polygons (and white ellipses) represent data, or CNN features, and color rectangles represent data processing in each layer. Note that numbers of layers are counted differently in different literature. Here a layer is data after a processing (such as convolution, pooling, normalization, etc.). Numbers in names of layers show a pack of these layers (for example, conv5 and pool5 are 5th layer of the CNN model in the literature).

These extracted features are then fed directly into a linear Support Vector Machine (SVM) classifier. The authors systematically compare the classification accuracy of each network layer to determine where the most useful diagnostic information resides. They also benchmark these results against traditional hand-crafted feature pipelines (BoVW, Fisher Vector, and VLAD).

Datasets

The study utilizes a clinical dataset consisting of 908 NBI image patches collected during endoscopic examinations at Hiroshima University Hospital. Following standard clinical sub-classification frameworks, the patches are labeled into three distinct categories based on their microvessel structures: Type A, Type B, and Type C3.

Key Insights and Ideas

  • Early Layers Rule: The highest classification accuracies are achieved using features from the first few convolutional or pooling layers, rather than the final fully connected layers.
  • Deep Layers Blur Texture: As an image progresses deeper into an ImageNet-trained network, the features become highly specialized for semantic object recognition (e.g., distinguishing a dog from a car). For medical texture classification, these deep layers actually discard or "blur out" the fine-grained microvessel patterns essential for diagnosis.
  • CNNs as Advanced Filter Banks: Early CNN layers function similarly to traditional texture analysis tools (like Gabor or wavelet filters). This makes them naturally suited for endoscopic image patches without requiring any domain-specific fine-tuning.
  • Resizing is Safe: Intuitively, resizing texture patches to a fixed input size might distort the scale and aspect ratio of the microvessels, harming performance. However, the experiments counter-intuitively show that simple image resizing works remarkably well and does not degrade accuracy.

Key Results

  • GoogLeNet achieved the highest performance among the deep models, reaching a maximum accuracy of 96.9 ± 1.3% using features from the pool3/3x3_s2 layer.
  • Caffe Net peaked at 96.8 ± 2.0% at the conv3 layer.
  • AlexNet reached 94.1 ± 3.7% at the pool2 layer.

These off-the-shelf CNN features perform on par with the best hand-crafted pipeline (VLAD, which achieved 97.2 ± 3.2%). Crucially, the CNN approach achieves this competitive accuracy while bypassing the tedious multi-stage encoding processes required by traditional hand-crafted methods.

Fig. 6. Recognition results with GoogLeNet layers.

Limitations and Future Work

A major drawback of using early convolutional layers is their high dimensionality, which significantly increases the computational overhead and memory requirements for training the SVM classifier. While fully connected layers help reduce dimensionality, they suffer from a slight drop in accuracy because they are too specialized for ImageNet classes.

To bridge this gap, the authors propose that future work should focus on designing a custom, shallow CNN architecture specifically tailored for NBI colorectal polyp recognition. This ideal network would feature only two or three convolutional layers to capture fine texture details, followed by a compact fully connected layer designed strictly for dimensionality reduction rather than semantic classification.

Summary updated

Abstract

In this paper we report results for recognizing colorectal NBI endoscopic images by using features extracted from convolutional neural network (CNN). In this comparative study, we extract features from different layers from different CNN models, and then train linear SVM classifiers. Experimental results with 10-fold cross validations show that features from first few convolution layers are enough to achieve similar performance (i.e., recognition rate of 95%) with non-CNN local features such as Bag-of-Visual words, Fisher vector, and VLAD.
BibTeX
@misc{tamaki2016computer,
  author = {Toru Tamaki and Shoji Sonoyama and Tsubasa Hirakawa and Bisser Raytchev and Kazufumi Kaneda and Tetsushi Koide and Shigeto Yoshida and Hiroshi Mieno and Shinji Tanaka},
  title = {{Computer-Aided Colorectal Tumor Classification in NBI Endoscopy Using CNN Features}},
  year = {2016},
  eprint = {1608.06709},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1608.06709},
}
Figure