TL;DR
This paper tackles the challenge of inconsistent scoring in pediatric visual-motor integration (VMI) assessments by proposing a novel deep learning architecture. Their improved DenseNet model significantly boosts scoring accuracy, especially for a larger number of VMI questions, offering a more reliable and efficient tool for clinical use.
What Problem Does This Paper Address?
Assessing visual-motor integration (VMI) in children is crucial for identifying developmental delays that can impact learning, particularly in writing. The standard method, the Beery-Buktenica VMI test, relies on manual scoring by professionals. However, this manual approach suffers from subjectivity, leading to inconsistent scoring standards and a significant workload, especially for large-scale assessments. While AI has been explored to automate this, existing deep learning models struggle with accuracy as the number of VMI questions increases, limiting their practical application. This paper aims to bridge this gap by developing a more accurate and robust AI system for VMI scoring.
Methods
The core of this research is a novel deep learning architecture designed to improve VMI scoring accuracy. Key innovations include:
- Optimized Input Features: Instead of directly concatenating raw question images with answer images, the authors propose concatenating question numbers with answer features. This reduces data complexity and computational load while maintaining performance. They also experimented with different channel combinations to find the optimal input structure.
- Improved DenseNet Architecture: The study leverages the pre-trained DenseNet201 model as a backbone. They then add new fully connected layers for enhanced feature extraction and classification. Crucially, they switch from a tanh activation function to softmax for the output layer, enabling a multi-class classification approach. This design aims to reduce model complexity and prevent overfitting.
- Multi-Class Classification: The task is reframed as a multi-class classification problem, where each class represents a specific combination of rules satisfied for a given question. This approach is more efficient than multi-label classification used in prior work.
- AI Scoring Application: A user-friendly software application was developed to integrate the AI model, allowing for automated scoring, raw score calculation, norm-based standard score conversion, and achievement level classification. It also provides explanations for unmet scoring rules.
Datasets
The dataset used in this study was provided by the Occupational Therapy Department of National Taiwan University. It consists of:
- Hand-drawn sketch data: 8610 images of children's (aged 3-6 years) drawings for 12 types of VMI test questions.
- Data Augmentation: To address class imbalance and increase dataset size, techniques like rotation, stretching, scaling, and skewing were applied. The final augmented dataset comprised 20,365 images.
- Data Split: A 1/10 split was used for validation, ensuring no overlap with the training set. Augmented data was exclusively used for training.
Key Insights and Ideas
- Simplicity Can Be Powerful: Concatenating question numbers with answer features, rather than full images, proved to be a more effective strategy for reducing complexity and improving accuracy. This highlights the importance of feature engineering in deep learning.
- DenseNet201 is a Strong Backbone: The study found DenseNet201 to be the most suitable pre-trained model among ResNet50, ResNet101, and InceptionResNetV2 for this specific task, demonstrating its superior generalization capabilities.
- Multi-Class Classification Outperforms Multi-Label: Reframing the VMI scoring as a multi-class problem, rather than a multi-label one, led to a more efficient and accurate output representation.
- Fine-tuning is Key, But Not Excessive: While unfreezing some layers of the pre-trained DenseNet201 improved performance, excessive unfreezing led to overfitting and decreased accuracy, suggesting a careful balance is needed.
Key Results
The proposed architecture and improved DenseNet model demonstrated significant improvements in scoring accuracy:
- 6-Question Graphics: Accuracy increased by 12.8% compared to the most relevant literature.
- 12-Question Graphics: Accuracy increased by 20.14% compared to the most relevant literature.
- Improved DenseNet Accuracy: Achieved 95.13% accuracy for 6-question graphics and 89.84% for 12-question graphics.
- Stability: The proposed model maintained high accuracy even when the number of questions increased from 6 to 12, unlike previous methods where accuracy significantly dropped.
Limitations and Future Work
- Limitations:
- The application currently cannot accurately identify individual rule errors for questions 14 and 15 due to limitations in data collection.
- While the SGD classifier offered efficiency, it resulted in a slight decrease in final accuracy, which the authors deemed acceptable for certain applications but not for the primary scoring system.
- Future Work:
- Improving the balance between SGD classification efficiency and accuracy.
- Expanding the number of assessment tool questions covered by the system.
- Further enhancing model accuracy.
Summary updated