Late fusion of deep learning and hand-crafted features for Achilles tendon healing monitoring

Norbert Kapinski, Jedrzej M. Nowosielski, Maciej E. Marchwiany, Jakub Zielinski, Beata Ciszkowska-Lyson, Bartosz A. Borucki, Tomasz Trzcinski, Krzysztof S. NowinskiarXiv 2019 · OpenReview

Abstract

Healing process assessment of the Achilles tendon is usually a complex procedure that relies on a combination of biomechanical and medical imaging tests. As a result, diagnostics remains a tedious and long-lasting task. Recently, a novel method for the automatic assessment of tendon healing based on Magnetic Resonance Imaging and deep learning was introduced. The method assesses six parameters related to the treatment progress utilizing a modified pre-trained network, PCA-reduced space, and linear regression. In this paper, we propose to improve this approach by incorporating hand-crafted features. We first perform a feature selection in order to obtain optimal sets of mixed hand-crafted and deep learning predictors. With the use of approx. 20,000 MRI slices, we then train a meta-regression algorithm that performs the tendon healing assessment. Finally, we evaluate the method against scores given by an experienced radiologist. In comparison with the previous baseline method, our approach significantly improves correlation in all of the six parameters assessed. Furthermore, our method uses only one MRI protocol and saves up to 60% of the time needed for data acquisition.

Introduction

Recovering from an Achilles tendon rupture is a long, arduous process. To ensure rehabilitation is effective, clinicians must monitor tissue morphology—specifically how the tendon is regenerating. Currently, this involves a battery of biomechanical tests and complex MRI evaluations that are both time-consuming for patients and labor-intensive for radiologists. While deep learning has recently been applied to automate this assessment, previous models often required multiple MRI protocols to achieve sufficient clarity. This paper introduces a "late fusion" approach that combines deep learning features with traditional hand-crafted texture analysis to make automated monitoring more accurate and significantly faster by utilizing a single, optimized MRI sequence.

Methodology

The proposed framework utilizes a late fusion architecture, integrating two distinct types of information to capture a holistic view of tendon health.

Proposed late fusion framework combining deep learning and hand-crafted features.

Feature Extraction and Selection

  1. Deep Learning (DL) Features: A truncated AlexNet, pre-trained on tendon health classification, extracts 4,096 features from the fc6 layer. These are reduced to the 200 most significant components via Principal Component Analysis (PCA).
  2. Hand-Crafted (HC) Features: These include Region of Interest (ROI) statistics (area, mean, skewness) and Haralick’s textural features, which describe the spatial relationship between pixels to capture the "grain" of the tissue.

To prevent overfitting on clinical datasets, the researchers used LASSO (Least Absolute Shrinkage and Selection Operator) to select only the most predictive features for each clinical parameter. This resulted in sparse models typically using fewer than 20 features per parameter.

Protocol Optimization

A key contribution of this work is identifying that the $T2^*GRE\ TE\_MIN$ protocol is the most informative. Unlike other sequences that may appear healthy (black) prematurely, this protocol remains sensitive to pathological changes even after 12 weeks of healing.

Results

The study utilized a longitudinal dataset of approximately 20,000 MRI slices from 60 subjects, tracked up to 52 weeks post-operation. The late fusion method demonstrated superior performance across all six clinical parameters: Structural changes (SCT), Thickening (TT), Edge sharpness (STE), Edema (TE), Uniformity (TU), and Tissue edema (TisE).

Comparison of predicted healing scores (SVR) vs. ground truth over 52 weeks for selected parameters.

Key Findings

  • Improved Accuracy: The proposed method significantly outperformed the baseline in terms of correlation with expert radiologist scores. Support Vector Regression (SVR) emerged as the most stable meta-regression algorithm.
  • Clinical Efficiency: By optimizing for a single MRI protocol, the method reduces the time a patient spends in the scanner by approximately 60% (from ~12 minutes down to ~5 minutes).
  • The Value of Texture: Haralick features were particularly significant for assessing structural changes and edema, proving that micro-texture in the MRI signal carries vital diagnostic weight.

Discussion and Future Work

The integration of deep learning's abstract patterns with the specific structural descriptions of hand-crafted features provides a more robust tool for clinical monitoring. However, a current limitation is the requirement for manual segmentation of the tendon (ROI). Future iterations of this work aim to employ fully convolutional networks to automate the segmentation task, creating a fully end-to-end automated pipeline for Achilles tendon rehabilitation monitoring.

Summary updated

Abstract

Healing process assessment of the Achilles tendon is usually a complex procedure that relies on a combination of biomechanical and medical imaging tests. As a result, diagnostics remains a tedious and long-lasting task. Recently, a novel method for the automatic assessment of tendon healing based on Magnetic Resonance Imaging and deep learning was introduced. The method assesses six parameters related to the treatment progress utilizing a modified pre-trained network, PCA-reduced space, and linear regression. In this paper, we propose to improve this approach by incorporating hand-crafted features. We first perform a feature selection in order to obtain optimal sets of mixed hand-crafted and deep learning predictors. With the use of approx. 20,000 MRI slices, we then train a meta-regression algorithm that performs the tendon healing assessment. Finally, we evaluate the method against scores given by an experienced radiologist. In comparison with the previous baseline method, our approach significantly improves correlation in all of the six parameters assessed. Furthermore, our method uses only one MRI protocol and saves up to 60\% of the time needed for data acquisition.
BibTeX
@misc{kapinski2019late,
  author = {Norbert Kapinski and Jedrzej M. Nowosielski and Maciej E. Marchwiany and Jakub Zielinski and Beata Ciszkowska-Lyson and Bartosz A. Borucki and Tomasz Trzcinski and Krzysztof S. Nowinski},
  title = {{Late fusion of deep learning and hand-crafted features for Achilles tendon healing monitoring}},
  year = {2019},
  eprint = {1909.05687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.05687},
}
Figure