Semantic Line Combination Detector

Jinwon Ko, Dongkwon Jin, Chang-Su KimCVPR 2024 · DOI · arXiv

Introduction

In computer vision, a semantic line is a meaningful boundary that defines the structural layout of a scene, such as a horizon, a road's edge, or an architectural axis. Traditional detection methods typically evaluate these lines individually or in pairs, often relying on Non-Maximum Suppression (NMS) to prune overlaps. However, these local approaches frequently fail to capture the global harmony of a scene, leading to cluttered compositions or sets of lines that do not logically partition the image into meaningful regions.

This paper introduces the Semantic Line Combination Detector (SLCD), a framework that shifts the paradigm from local detection to holistic optimization. By evaluating entire groups of lines simultaneously, SLCD identifies the most harmonious global composition, ensuring that the detected lines work together to define the image's structural layout.

Comparison of traditional local methods (NMS, Pairwise) versus the proposed global SLCD approach, which evaluates line combinations for composition analysis.

Methodology

The SLCD framework treats line detection as a combinatorial optimization problem. The process involves four primary stages:

  1. Reliable Line Selection: The model identifies a small set of $K$ (typically 8) "reliable" candidate lines using a modified S-Net architecture.
  2. Combination Generation: All $2^K$ possible combinations of these reliable lines are generated for evaluation.
  3. The SLCD Architecture: For each combination, the model extracts semantic and compositional features. A key innovation is Semantic Feature Grouping, which uses cross-attention with "region queries" to partition the image into $M$ regions. This is achieved without explicit segmentation labels through a novel Semantic Region Separation (SRS) loss based on Kullback-Leibler divergence.
  4. Score Regression: A regressor predicts a "composition score" for each group. The combination with the highest score is selected as the final output.
The detailed architecture of the SLCD framework, illustrating the flow from image encoding and semantic grouping to compositional feature extraction and final score regression.

Datasets and Evaluation

The researchers evaluated SLCD on standard benchmarks including SEL (landscapes) and NKL (indoor/outdoor). To address the limitations of existing datasets, they introduced CDL (Compositionally Diverse Lines), a new benchmark containing 7,100 images categorized into seven distinct composition classes such as Symmetric, Diagonal, and Triangle. Performance was measured using Harmonious Intersection over Union (HIoU), a metric specifically designed to assess how well a set of lines matches the structural layout of a scene.

Key Results and Applications

SLCD consistently outperformed existing state-of-the-art models across all datasets:

  • SEL Dataset: Achieved an HIoU of 84.09%, a significant improvement over the previous best of 81.03%.
  • CDL Dataset: Led with 68.85% HIoU, demonstrating robustness across diverse and complex layouts.

Beyond line detection, the authors demonstrated the utility of SLCD in downstream tasks. It improved Vanishing Point Detection accuracy to 78.3% (at a 10° threshold) and enabled a novel Composition-Based Image Retrieval system that finds photos with similar structural layouts rather than just matching colors or objects.

Conclusion

By prioritizing global composition over local edge detection, the Semantic Line Combination Detector provides a more sophisticated understanding of image geometry. While challenges remain in detecting lines implied by complex organic objects (such as humans or animals), the model's ability to implicitly learn semantic regions and evaluate line harmony marks a significant step forward in scene layout analysis.

Summary updated

BibTeX
@inproceedings{ko2024semantic,
  author = {Jinwon Ko and Dongkwon Jin and Chang-Su Kim},
  title = {{Semantic Line Combination Detector}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02651},
}

Loading related content...

Figure