Multi-Scale Bidirectional FCN for Object Skeleton Extraction

Fan Yang, Xin Li, Hong Cheng, Yuxiao Guo, Leiting Chen, Jianping Li

Abstract

Object skeleton extraction is a critical computer vision task that identifies the medial axes of objects. Unlike edge detection, which relies on local contrast, skeletonization requires high-level semantic understanding. This paper introduces the Multi-Scale Bidirectional Fully Convolutional Network (MSB-FCN), an architecture designed to leverage deep semantic features while discarding noisy low-level information. By employing a bidirectional, densely connected structure across multiple scales, MSB-FCN achieves state-of-the-art performance, notably improving accuracy by 28.7% on the challenging Sym-PASCAL dataset.

Introduction

Extracting the "inner bones" or skeletons of objects is fundamental for shape analysis and object recognition. While superficially similar to edge detection, skeleton extraction is distinct: edges are defined by local changes in color or texture, whereas skeletons represent the global geometric structure of an object.

Conceptual difference between edge detection (low-level) and skeleton extraction (high-level semantic).

Previous approaches often adapted edge detection architectures like Holistically-nested Edge Detection (HED). However, these models typically utilize "skip-layer" structures that mix shallow, low-level features with deep, semantic ones. The authors of this work argue that for skeletonization, shallow features act as "messy" noise, particularly in complex scenes. MSB-FCN addresses this by focusing exclusively on high-level features processed through a novel multi-scale bidirectional framework.

Methodology: The MSB-FCN Architecture

The MSB-FCN consists of three primary modules designed to capture multi-scale semantic information without the interference of low-level noise.

The MSB-FCN architecture featuring a feature pyramid, multi-scale bidirectional network, and intermediate supervision.
  1. Feature Pyramid Construction: Instead of aggregating features from all network layers, MSB-FCN extracts the final, most semantic layer from a pre-trained ResNet-101. Pyramid pooling then generates four distinct scales (10x10, 20x20, 30x30, and 60x60 bins). This allows the model to analyze object structure at various resolutions using only "smart" high-level features.
  2. Multi-Scale Bidirectional Network: Features are processed in two directions: coarse-to-fine (global context to local detail) and fine-to-coarse (local detail to global context). This bidirectional flow ensures that predictions at any scale are informed by both the overall object shape and its specific parts.
  3. Dense Connections and Fusion: Inspired by DenseNets, every scale is linked to every other scale in a feed-forward manner. This dense connectivity prevents information loss during multi-scale processing. Finally, a Fusion Module learns the optimal weights to combine these multi-scale predictions into a single, high-resolution skeleton map.

Experimental Evaluation

The model was rigorously tested on four major benchmarks:

  • SK-SMALL: Known for high variance in object structure.
  • WH-SYMMAX: Focused on the Weizmann Horse dataset.
  • SK-LARGE: A comprehensive, large-scale benchmark.
  • Sym-PASCAL: The most demanding dataset, featuring 20 object classes in complex, "in-the-wild" environments.

Results and Discussion

MSB-FCN consistently outperformed both traditional and deep-learning-based predecessors. On the Sym-PASCAL dataset, it achieved an F-measure of 0.570, representing a 28.7% improvement over the previous state-of-the-art (SRN at 0.443).

Key insights from the study include:

  • Efficacy of Deep Features: Shallow CNN layers are not only unnecessary but can be detrimental to skeleton detection. By relying on the final convolutional block, the model avoids confusion from background textures.
  • Bidirectional Context: Skeletons are intrinsically linked to object boundaries. The bidirectional flow allows the network to better model the relationship between global shape and local symmetry.
  • Efficiency: The model processes images in approximately 0.13 seconds on a modern GPU, making it viable for near-real-time applications.

Conclusion

The MSB-FCN demonstrates that a focused, multi-scale approach using high-level semantic features is superior for object skeleton extraction. While the model excels in cluttered scenes, it still faces challenges in extreme cases where objects are nearly indistinguishable from their backgrounds. Future work will explore applying this bidirectional architecture to other high-level vision tasks such as 3D structure estimation.

Summary updated

Abstract

Object skeleton detection is a challenging problem with wide application. Recently, deep Convolutional Neural Networks (CNNs) have substantially improved the performance of the state-of-the-art in this task. However, most of the existing CNN-Based methods are based on a skip-layer structure where low-level and high-level features are combined and learned so as to gather multi-level contextual information. As shallow features are too messy and lack semantic knowledge, they may cause errors and inaccuracy. Therefore, we propose a novel network architecture, Multi-Scale Bidirectional Fully Convolutional Network (MSB-FCN), to better capture and consolidate multi-scale high-level context information for object skeleton detection. Our network uses only deep features to build multi-scale feature representations, and employs a bidirectional structure to collect contextual knowledge. Hence the proposed MSB-FCN has the ability to learn the semantic-level information from different sub-regions. Furthermore, we introduce dense connections into the bidirectional structure of our MSB-FCN to ensure that the learning process at each scale can directly encode information from all other scales. Extensive experiments on various commonly used benchmarks demonstrate that the proposed MSB-FCN has achieved significant improvements over the state-of-the-art algorithms.
BibTeX
@inproceedings{yang2018multi,
  author = {Fan Yang and Xin Li and Hong Cheng and Yuxiao Guo and Leiting Chen and Jianping Li},
  title = {{Multi-Scale Bidirectional FCN for Object Skeleton Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12288},
}
Figure