TL;DR
This paper introduces DT-LSD, a novel end-to-end transformer-based model for line segment detection that significantly outperforms existing methods in accuracy and training speed. By incorporating a deformable attention mechanism for cross-scale interactions and a new training technique called Line Contrastive DeNoising (LCDN), DT-LSD achieves state-of-the-art results with substantially fewer training epochs.
What Problem Does This Paper Address?
Line segment detection is a crucial low-level computer vision task with applications in 3D reconstruction, camera calibration, and scene understanding. While Convolutional Neural Networks (CNNs) have dominated this field, they often require complex post-processing steps to extract final line predictions. Transformer-based models, like LETR, offer an end-to-end approach but have faced challenges with slow convergence and high computational complexity, hindering their widespread adoption. This paper aims to bridge this gap by developing a more efficient and accurate transformer-based line segment detector.
Methods
The core of DT-LSD lies in two key innovations:
- Deformable Attention Encoder: To address the quadratic complexity of standard global attention in transformers, DT-LSD employs a deformable attention mechanism. This module allows the model to attend to a fixed number of sampling points per query, significantly reducing computational cost. Crucially, it enables both intra-scale (within a feature map) and cross-scale (across different feature maps) interactions, a capability lacking in previous transformer models like LETR. This is achieved by using a multi-scale deformable attention variant that integrates features from hierarchical backbone outputs.
- Line Contrastive DeNoising (LCDN): To accelerate training, the authors introduce LCDN. This technique stabilizes the one-to-one matching process between predicted line segments and ground truth, inspired by contrastive denoising methods. LCDN works by generating "positive" and "negative" queries through controlled line length scaling and rotation. This guides the Hungarian matcher to better distinguish between correct and incorrect line predictions, leading to faster convergence. Notably, LCDN is a training-time technique and does not impact inference speed.
Datasets
The paper evaluates DT-LSD on two established datasets:
- Wireframe Dataset: Used for both training and testing.
- YorkUrban Dataset: Used for testing.
The authors also mention the use of the ShangaiTech Wireframe Dataset for comparisons.
Key Insights and Ideas
- Deformable Attention is Key for Efficiency and Effectiveness: The paper strongly suggests that the deformable attention mechanism is crucial for achieving both computational efficiency and improved performance in transformer-based line segment detection. Its ability to handle cross-scale interactions without the quadratic complexity of global attention is a significant advantage.
- End-to-End Transformers Can Outperform CNNs: DT-LSD demonstrates that transformer-based models, when designed and trained effectively, can surpass traditional CNN-based methods in line segment detection, eliminating the need for hand-crafted post-processing.
- Targeted Training Techniques Accelerate Convergence: LCDN proves to be a highly effective training strategy, drastically reducing the number of epochs required for convergence without sacrificing accuracy. This makes transformer models more practical for real-world applications.
- Pre-training is Crucial: The study highlights the importance of pre-training the entire network (not just the backbone) on large datasets like COCO for achieving state-of-the-art results.
Key Results
- Accuracy: DT-LSD achieves state-of-the-art results on both the Wireframe and YorkUrban datasets, outperforming previous transformer-based models (LETR) and leading CNN-based methods. For instance, on the Wireframe dataset, it achieves 71.7 for sAP10 and 73.9 for sAP15.
- Training Speed: DT-LSD trains significantly faster than LETR, requiring only 24 epochs compared to LETR's 825 epochs.
- Inference Speed: The model achieves a respectable inference speed of nearly 9 frames per second.
- Qualitative Improvements: Qualitative results show DT-LSD producing cleaner line detections, avoiding some of the noisy outputs seen in LETR, and generally performing better than CNN-based approaches like LCNN and HAWP.
Limitations and Future Work
- Pre-training Dependency: The authors note that achieving state-of-the-art results heavily relies on pre-training the network with COCO weights. Training from scratch is identified as a future research direction.
- Specialized Backbones: The paper suggests developing specialized backbones tailored for transformer-based line segment detection models as a promising avenue for future work.
- Further Optimization: While DT-LSD offers a good balance, further exploration into optimizing the trade-off between inference speed and accuracy through different configurations (e.g., feature map usage, backbone choice) is ongoing.
Summary updated