Automatic Gating of Attributes in Deep Structure

Xiaoming Jin, Tao He, Cheng Wan, Lan Yi, Guiguang Ding, Dou ShenIJCAI 2018 · DOI

Introduction

Deep learning models have revolutionized computer vision by learning hierarchical representations of data. In these structures, lower layers typically capture simple edges and textures, while higher layers synthesize these into complex parts and holistic objects. Parallel to this, human-defined attributes also exist at varying levels of abstraction: "round" is a low-level geometric property, whereas "engine" represents a high-level, complex concept.

Despite this shared hierarchical nature, most existing models treat attributes as "flat" inputs, feeding them indiscriminately into a single layer (often the first or last). This paper addresses the fundamental "mismatch" in abstraction that occurs when high-level attributes are connected to low-level layers, or vice versa, which can lead to incomplete or over-abstracted features and ultimately degrade performance.

Conceptual mapping of low-level and high-level attributes to corresponding hierarchical layers in a deep network.

The Attribute Gated Deep Belief Network (AG-DBN)

To solve the alignment problem, the authors propose the Attribute Gated Deep Belief Network (AG-DBN). This architecture introduces a learnable gating mechanism that automatically discovers the optimal "home" for each attribute within the network's deep structure.

The Gating Mechanism

The core innovation is a "gate matrix" ($G$) that acts as a switchboard. It determines the connectivity between an attribute $a_j$ and a hidden layer $h_l$. Rather than hard-coding these connections based on human intuition, the model treats the gates as part of a probabilistic distribution within a Restricted Boltzmann Machine (RBM) and Deep Belief Network (DBN) framework.

The AG-DBN architecture featuring a learnable gate matrix G that dynamically routes attributes to appropriate hidden layers.

Iterative Learning Process

The AG-DBN is trained using an alternating two-phase process:

  1. Parameter Learning: The weights and biases of the network are updated while the gate distribution remains fixed.
  2. Gate Learning: The gate matrix is updated by observing which connections exhibit the strongest interactions between attributes and hidden units.

Because the model is generative, it can also "fill in the blanks" for missing attributes during inference. It uses learned hidden representations to infer missing labels, maintaining high classification accuracy even when data is incomplete.

Experimental Evaluation

The researchers evaluated the AG-DBN across three diverse datasets to test its robustness and accuracy:

  • a-ImageNet: A subset of ImageNet (10 synsets, ~8,300 images) with 25 attributes covering color, pattern, shape, and texture.
  • a-Yahoo: 12 object categories (e.g., "centaur," "jet ski") with 64 attributes.
  • a-Pascal: Used to benchmark the AG-DBN against state-of-the-art CNNs like VGG16 and VGG19.

Key Results

The AG-DBN consistently outperformed standard DBNs and high-performing CNN baselines. Most notably, on the a-Pascal dataset, the model reduced classification error from 26.70% (VGG19) to 13.56%—effectively cutting the error rate by nearly half.

Dataset Baseline Error AG-DBN Error
a-Pascal 26.70% (VGG19) 13.56%
a-ImageNet 0.8188 (DBN) 0.6845
a-Yahoo 0.2344 (Random AG-DBN) 0.2015

Discussion and Insights

Abstraction Alignment is Critical

Preliminary tests confirmed that "wrong" connections actively hurt the model. Forcing a high-level concept like "engine" into a low-level layer increased test error from ~73% to ~82%. This validates the necessity of the gating mechanism.

Machines vs. Human Intuition

Interestingly, the learned gating often outperformed manual groupings. While humans might group attributes by semantic categories (e.g., all shapes together), the AG-DBN found optimal connections that suggest the "abstraction level" in a neural network does not always perfectly align with human semantic logic.

Limitations and Future Work

While powerful, the AG-DBN faces challenges in computational complexity, as the search space for optimal connections grows exponentially with the number of layers and attributes ($L^{N_a}$). Future research may focus on scaling this mechanism to extremely deep architectures like ResNets or Transformers and extending the gating technique to sequential data models like LSTMs.

Summary updated

BibTeX
@inproceedings{jin2018automatic,
  author = {Xiaoming Jin and Tao He and Cheng Wan and Lan Yi and Guiguang Ding and Dou Shen},
  title = {{Automatic Gating of Attributes in Deep Structure}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/319},
}
Figure