Kornia: an Open Source Differentiable Computer Vision Library for PyTorch

Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, Gary BradskiWACV 2020

TL;DR

Kornia is a new open-source library for PyTorch that brings traditional computer vision algorithms into the deep learning ecosystem. By making these algorithms differentiable and GPU-accelerated, Kornia allows for seamless integration into neural networks, enabling end-to-end training of complex vision tasks.

What Problem Does This Paper Address?

The field of computer vision has seen a massive shift towards deep learning, with frameworks like PyTorch becoming the de facto standard for training models. While PyTorch excels at automatic differentiation and GPU acceleration, it traditionally lacks comprehensive, differentiable implementations of many classical computer vision algorithms. Existing libraries like OpenCV are highly optimized for CPU-based, non-differentiable operations, making it difficult to integrate them directly into deep learning pipelines for tasks like image registration, depth estimation, or feature detection. This creates a gap where researchers and engineers need to either reimplement these algorithms in a differentiable way or use separate, non-differentiable tools, hindering the development of end-to-end trainable vision systems.

Methods

Kornia bridges this gap by providing a collection of differentiable routines and modules that operate on PyTorch tensors. The library leverages PyTorch's reverse-mode automatic differentiation to compute gradients for complex vision functions, allowing them to be treated as layers within neural networks. Key features include:

  • Differentiability: All operators are designed to be differentiable, enabling backpropagation through vision pipelines.
  • GPU Acceleration: Algorithms are implemented to run efficiently on GPUs, taking advantage of PyTorch's parallel processing capabilities.
  • Transparent API: Kornia's API is designed to be device-agnostic, allowing users to switch between CPU and GPU with minimal code changes.
  • Batch Processing: Operators are built to handle batches of images, crucial for efficient GPU utilization and modern deep learning workflows.
  • Modularity: The library is organized into submodules covering common computer vision tasks like color space conversions, filtering, geometry transformations, and feature detection.

Datasets

The paper mentions using the SceneNet RGB-D dataset for depth estimation experiments. No new datasets or benchmarks are introduced.

Key Insights and Ideas

  • Bridging the Gap: The most significant insight is the successful integration of classical computer vision algorithms into the deep learning paradigm. By making them differentiable and GPU-friendly, Kornia unlocks new possibilities for end-to-end trainable vision models.
  • Differentiable Warping: The concept of differentiable image warping (e.g., using kornia.geometry.HomographyWarper) is powerful. It allows for optimizing geometric transformations directly by minimizing photometric errors, eliminating the need for manual gradient calculations.
  • Unified Framework: Kornia provides a unified framework where traditional vision tasks can be seamlessly combined with deep learning components, simplifying complex pipelines.
  • Performance on GPU: While Kornia's CPU performance might be comparable to other libraries for small batches, its GPU performance significantly outperforms others when using larger batch sizes, highlighting its suitability for deep learning.

Key Results

  • Image Registration: The paper demonstrates image registration using gradient descent, achieving convergence and refining solutions across different pyramid levels.
  • Depth Estimation: A fully differentiable multi-view depth estimation pipeline is presented, showcasing the ability to optimize depth maps using various loss functions.
  • Adversarial Attack on SIFT Matching: Kornia's differentiability is used to perform a targeted adversarial attack on SIFT feature matching, demonstrating the ability to manipulate image features to achieve specific matching outcomes.
  • Performance Benchmark: A benchmark comparing Kornia to OpenCV, PIL, skimage, and scipy shows that Kornia achieves comparable results, especially on GPU with larger batch sizes. For Sobel edge detection, Kornia on GPU was the fastest for larger batch sizes.

Limitations and Future Work

  • CPU Performance: The paper acknowledges that Kornia's CPU implementation is not the fastest, especially for larger batch sizes, compared to highly optimized libraries like OpenCV.
  • Not Explicitly Optimized for Speed: The authors state that the primary goal is not to provide the absolute fastest implementations but rather differentiable and GPU-accelerated ones that integrate well with PyTorch.
  • Future Work: The authors anticipate that researchers and companies will contribute to expanding the library's functionality, suggesting a community-driven growth model. The paper also hints at the potential for Kornia to become a standard component for pre- and post-processing in deep learning pipelines.

Summary updated

BibTeX
@inproceedings{riba2020kornia,
  author = {Edgar Riba and Dmytro Mishkin and Daniel Ponsa and Ethan Rublee and Gary Bradski},
  title = {{Kornia: an Open Source Differentiable Computer Vision Library for PyTorch}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure