Differentiable Kernel Evolution

Yu Liu, Jihao Liu, Xiaogang Wang, Ailing ZengICCV 2019 · OpenReview

Abstract

This paper proposes a differentiable kernel evolution (DKE) algorithm to find a better layer-operator for the convolutional neural network. Unlike most of the other neural architecture searching (NAS) technologies, we consider the searching space in a fundamental scope: kernel space, which encodes the assembly of basic multiply-accumulate (MAC) operations into a conv-kernel. We first deduce a strict form of the generalized convolutional operator by some necessary constraints and construct a continuous searching space for its extra freedom-of-degree, namely, the connection of each MAC. Then a novel unsupervised greedy evolution algorithm called gradient agreement guided searching (GAGS) is proposed to learn the optimal location for each MAC in the spatially continuous searching space. We leverage DKE on multiple kinds of tasks such as object classification, face/object detection, large-scale fine-grained recognition, with various kinds of backbone architecture. Not to mention the consistent performance gain, we found the proposed DKE can further act as an auto-dilated operator, which makes it easy to boost the performance of miniaturized neural networks in multiple tasks.

Introduction

Standard Convolutional Neural Networks (CNNs) rely on kernels defined on a fixed, rectangular grid (e.g., $3 \times 3$). While effective, this rigid structure doesn't necessarily match the diverse shapes of real-world objects. Previous attempts to solve this include Dilated Convolutions, which expand the grid but keep it rigid, and Deformable Convolutions (DCN). DCN uses a sub-network to predict offsets for every input pixel, making the kernel shape "data-dependent." However, DCN can introduce unnecessary uncertainty and noise for tasks where data is well-aligned (like face recognition) and adds significant computational overhead. Differentiable Kernel Evolution (DKE) addresses this by creating a kernel that can learn an optimal, irregular shape that is data-independent and stable to train via standard backpropagation.

Methodology

DKE treats a convolution as a set of Multiply-Accumulate (MAC) operations located at continuous coordinates $y \in \mathbb{R}^2$ rather than discrete integer points.

1. Generalized Convolution

The convolution operator is redefined to be independent of the input data location, maintaining translation invariance, but flexible in its internal MAC connections. Each channel can have its own unique set of learned offsets.

2. Gradient Agreement Guided Searching (GAGS)

Searching for the best coordinates in a continuous space using discrete image data is challenging. Standard interpolation methods like bilinear or nearest-neighbor have a major flaw: their gradients don't always point toward the optimal coordinate, leading to "unsteady" or "curved" searching paths during optimization.

To fix this, the authors introduce three constraints for an ideal interpolation function: Continuity, Monotonicity, and Gradient Agreement. The partial derivative of the interpolation function must point directly toward the reference point. They propose a negative-exponential interpolation kernel that satisfies all three, allowing the kernel offsets to "evolve" smoothly toward their optimal positions using standard Stochastic Gradient Descent (SGD).

Comparison of interpolation kernels and their gradient fields. Bilinear interpolation (top) shows biased gradients, while the proposed negative-exponential family (bottom) ensures gradient agreement.

Experimental Results

The authors evaluated DKE across a wide range of benchmarks including general classification (ImageNet), face recognition (MegaFace), and detection (FDDB, MALF, MS COCO).

Key Insights

  • The Interpolation Trap: Bilinear interpolation is poor for greedy coordinate searching because its gradient field is biased, preventing kernels from finding true optima.
  • Data-Independent Flexibility: DKE learns a fixed irregular shape, making it robust for aligned tasks like face recognition where data-dependent offsets (DCN) can degrade performance.
  • Auto-Dilation: DKE naturally pushes MAC operations further apart in shallow networks to increase the receptive field, significantly boosting miniaturized models.

Performance Highlights

On ImageNet, DKE-ResNet-101 achieved a 21.1% Top-1 error, outperforming both the baseline (22.7%) and DCN (21.6%). For face recognition on MegaFace, DKE reached 98.0% accuracy, whereas DCN dropped below the baseline. In small models, DKE reduced the Top-1 error of a 6-layer ResNet from 51.7% to 42.9%.

Performance comparison on face detection benchmarks (a) FDDB and (b) MALF, showing consistent gains with DKE.

Conclusion

Differentiable Kernel Evolution provides a robust, data-independent way to evolve kernel shapes. By overcoming the "interpolation trap" with a novel gradient-agreement-guided search, DKE allows CNNs to learn optimal spatial structures that significantly improve performance across various computer vision tasks, especially for small-scale architectures.

Summary updated

Abstract

This paper proposes a differentiable kernel evolution (DKE) algorithm to find a better layer-operator for the convolutional neural network. Unlike most of the other neural architecture searching (NAS) technologies, we consider the searching space in a fundamental scope: kernel space, which encodes the assembly of basic multiplyaccumulate (MAC) operations into a conv-kernel. We first deduce a strict form of the generalized convolutional operator by some necessary constraints and construct a continuous searching space for its extra freedom-of-degree, namely, the connection of each MAC. Then a novel unsupervised greedy evolution algorithm called gradient agreement guided searching (GAGS) is proposed to learn the optimal location for each MAC in the spatially continuous searching space. We leverage DKE on multiple kinds of tasks such as object classification, face/object detection, large-scale finegrained and recognition, with various kinds of backbone architecture. Not to mention the consistent performance gain, we found the proposed DKE can further act as an autodilated operator, which makes it easy to boost the performance of miniaturized neural networks in multiple tasks.
BibTeX
@inproceedings{liu2019differentiable,
  author = {Yu Liu and Jihao Liu and Xiaogang Wang and Ailing Zeng},
  title = {{Differentiable Kernel Evolution}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=WySfLUs-Ro3},
}
Figure