Real-Time Visual Perception and Explainable Fault Diagnosis for Railway Point Machines at the Edge

Yu Feng Zhai, Lili WeiElectronics 2026 · DOI

TL;DR

This paper tackles the challenge of real-time, interpretable fault diagnosis for railway point machines on resource-constrained edge devices. They introduce a novel framework that combines optimized visual perception with a large language model (LLM) for semantic reasoning, achieving high accuracy and speed while providing actionable maintenance insights.

What Problem Does This Paper Address?

Railway point machines are critical for safe and efficient train operations. Failures can lead to significant disruptions, including derailments and cascading delays. Traditional inspection methods are often manual, discontinuous, or rely on sensors susceptible to environmental noise and aging. While computer vision offers a promising non-contact solution, existing methods struggle with the "accuracy-latency paradox" on edge devices and often lack interpretability, providing raw data rather than actionable insights. This gap hinders timely and effective maintenance, especially in complex, dynamic railway environments.

Methods

The authors propose a multi-stage "Edge Perception—Cloud Diagnosis—Client Interaction" framework.

  • Hardware-Aware Optimization: To run complex models on the RK3588S edge platform, they developed a Sensitivity-Aware Mixed-Precision Quantization and Heterogeneous Scheduling (SMPQ-HS) strategy. This involves:
  • Mixed-Precision Quantization: Quantizing the model's backbone to INT8 for speed while keeping sensitive segmentation heads in FP16 to maintain accuracy.
  • Heterogeneous Task Scheduling: Assigning different computational tasks (image acquisition, feature extraction, post-processing) to the most suitable hardware units on the RK3588S (A55 cores, NPU, A76 cores) to maximize efficiency.
  • Geometric Prior-Based Depth Estimation: A novel approach using fixed reference points (bolts) on the point machine and perspective rectification to accurately estimate engagement depth, compensating for varying camera angles.
  • Multimodal Semantic Diagnostic Framework: They integrate the estimated depth with visual features and feed them into the Qwen2.5-VL-7B visual language model. A Hard-Constrained Knowledge Embedding Paradigm (using a structured prompt template) guides the VLM to produce standardized, interpretable diagnostic reports (JSON format) with detection status, maintenance advice, and risk levels.

Datasets

The study utilized a self-constructed dataset of 800 original images of ZD6 electric point machines. This dataset was augmented to include diverse environmental conditions (standard illumination, low-light, dust, vibration) and fault types (Normal, Shallow, Deep, No Contact), resulting in a total of 1600 samples. This dataset was divided into training, validation, and test sets with a 7:1:2 ratio.

Key Insights and Ideas

  • The "Accuracy-Latency Paradox" is Solvable on Edge: By carefully optimizing models with mixed-precision quantization and heterogeneous scheduling, high-accuracy segmentation can be achieved on resource-constrained edge devices.
  • Geometric Priors Enhance Vision: Using fixed geometric features (bolts) and perspective rectification provides a robust way to estimate critical physical parameters like engagement depth, crucial for diagnosis.
  • VLMs Bridge the Gap to Interpretability: Large language models, when guided by structured prompts, can translate raw visual and geometric data into human-readable, actionable maintenance recommendations, moving beyond simple detection.
  • Structured Prompts are Key for VLM Reliability: A three-layer prompt structure ("role definition + rule constraints + format locking") is essential for guiding VLMs to produce consistent, standardized, and reliable diagnostic outputs, especially in safety-critical applications.
  • "Graceful Degradation" is Crucial: The decoupled edge-cloud architecture ensures that even if cloud-based semantic diagnosis fails due to network issues, basic geometric monitoring and safety supervision can continue at the edge.

Key Results

  • Optimized Segmentation: The proposed YOLO11-seg model on the RK3588S achieved an inference speed of 38.5 FPS and an mIoU of 0.849, significantly outperforming standard segmentation models in speed while maintaining high precision.
  • Accurate Depth Estimation: The cubic polynomial model for depth estimation achieved an average MAE of 0.21 mm and a maximum relative error below 3.1%, meeting engineering tolerances.
  • Effective VLM Diagnosis: The VLM-based diagnostic module achieved a classification accuracy of 88.0% and a Macro-F1 score of 0.842, outperforming a numerical baseline.
  • End-to-End Latency: The total diagnostic latency was approximately 1.35 seconds, with the cloud reasoning stage being the primary contributor (~85%). This is considered acceptable for maintenance tasks.
  • Network Robustness: The system maintained a diagnostic success rate of 91.2% even with 10% packet loss, demonstrating reasonable robustness.

Limitations and Future Work

  • Generalization: The current evaluation is based on a specific dataset (ZD6 point machines) and constrained setting. Transferring the system to other point machine types or different installation geometries might require recalibration of geometric references and potential model adaptation.
  • Hardware Generalization: While the heterogeneous scheduling strategy is theoretically transferable, full hardware-level acceleration on other edge platforms might be limited by NPU toolchain support for specialized operators.
  • Environmental Robustness: The dataset covers common environmental variations, but extreme conditions like heavy rain, snow, fog, or severe lens contamination are not fully explored. The authors suggest a conservative strategy of rejecting uncertain frames in such cases.
  • VLM Stochasticity: The generative nature of VLMs can lead to a ~7.5% format error rate, requiring additional checks. While a "retry" mechanism mitigates this, further research into VLM reliability for safety-critical applications is warranted.
  • Future Work: The authors plan to expand the benchmark to other edge platforms and explore emerging lightweight backbones as toolchain support matures. They also suggest incorporating temporal smoothing for more robust depth estimation.

Summary updated

Abstract

Existing inspection systems for railway point machines often suffer from high latency and poor interpretability, which impedes the real-time detection of critical mechanical anomalies, thereby increasing the risks of derailment and leading to cascading schedule delays. Addressing these challenges, this study proposes a lightweight computer vision-based detection framework deployed on the RK3588S edge platform. First, to overcome the accuracy degradation of segmentation networks on constrained edge NPUs, a Sensitivity-Aware Mixed-Precision Quantization and Heterogeneous Scheduling (SMPQ-HS) strategy is proposed. Second, a Multimodal Semantic Diagnostic Framework is constructed. By integrating geometric engagement depths—calculated via perspective rectification—with visual features, a Hard-Constrained Knowledge Embedding Paradigm is designed for the Qwen2.5-VL model. This approach constrains the stochastic reasoning of the Qwen2.5-VL model into standardized diagnostic conclusions. Experimental results demonstrate that the optimized model achieves an inference speed of 38.5 FPS and an mIoU of 0.849 on the RK3588S, significantly outperforming standard segmentation models in inference speed while maintaining high precision. Furthermore, the average depth-estimation error remains approximately 3%, and the VLM-based fault identification accuracy reaches 88%. Overall, this work provides a low-cost, deployable, and interpretable solution for intelligent point machine maintenance under edge-computing constraints.
BibTeX
@inproceedings{zhai2026real,
  author = {Yu Feng Zhai and Lili Wei},
  title = {{Real-Time Visual Perception and Explainable Fault Diagnosis for Railway Point Machines at the Edge}},
  booktitle = {Electronics},
  year = {2026},
  doi = {10.3390/electronics15010230},
}
Figure