Dragon-Alpha&cu32: A Java-based Tensor Computing Framework With its High-Performance CUDA Library

Zhiyi Zhang, Pengfei Zhang, Qi Wang

Abstract

Java is very powerful, but in the Deep Learning field, its capabilities have probably not been sufficiently exploited. Compared to Java-based deep-learning frameworks, the Python-based ones (PyTorch, TensorFlow, etc.) are undoubtedly the mainstream due to their ease of use, flexibility, and better ecosystem. Dragon-Alpha is a Java-based Tensor Computing Framework designed to be easy to use, highly scalable, and high-performance, aiming to break Java's dilemma in the deep learning field. Dragon-Alpha supports different levels of APIs and can be used as a deep-learning framework through its user-friendly high-level APIs. It has the potential to aggregate computing power across heterogeneous platforms and devices, based on its multi-layer architecture and Java's big-data ecosystem. Dragon-Alpha features asynchronized APIs to improve parallelism and a highly-optimized CUDA library, cu32, which adopts unique convolution/deconvolution operators for small feature maps. Experiments show that, compared to PyTorch&cuDNN, Dragon-Alpha&cu32 costs less time and memory (75.38% to 97.32%, 29.2% to 66.4%) to train typical neural networks like AlexNet, VGG, GoogleNet, and ResNet on Cifar-10.

Introduction

While Python is the dominant language for deep learning research, Java remains the backbone of enterprise software and big-data infrastructure, including systems like Hadoop, Spark, and Flink. The potential for Java in deep learning is currently under-exploited because of a lack of high-performance, accessible frameworks. Existing Java options often suffer from complexity or performance lags compared to Python counterparts. Additionally, most frameworks rely on NVIDIA’s closed-source cuDNN library. Dragon-Alpha aims to bridge this gap by providing a framework that is as easy to use as Python while being more memory-efficient and better integrated with distributed big-data environments.

Methods

The solution consists of two primary components: the Dragon-Alpha framework and the cu32 library.

Dragon-Alpha Architecture

The framework is built on a 6-layer stack that abstracts complexity, moving from low-level native code to high-level user APIs.

The 6-layer hierarchical architecture of the Dragon-Alpha framework.
  • Layer 1-3: Handle low-level operations, memory pooling, and 64-bit address-oriented operators.
  • Layer 4-5: Manage tensors, computational graphs, loss functions, and optimizers.
  • Layer 6: Provides a user-friendly high-level API supporting "method chaining" and "factory patterns," making it intuitive for those familiar with modern Python libraries.

cu32 Library

A custom, open-source CUDA library, cu32, was developed to replace generic libraries. It features specialized convolution and deconvolution operators optimized for "small feature maps"—a common performance bottleneck in deep networks where spatial dimensions decrease in deeper layers.

Asynchronous Execution

Dragon-Alpha utilizes non-blocking APIs to maximize hardware utilization. This allows the CPU to handle tasks like data preprocessing while the GPU is active. Furthermore, a single CPU thread can launch multiple kernels across different GPU streams simultaneously.

The asynchronous execution model and stream synchronization in Dragon-Alpha.

Memory Optimization

To achieve its high efficiency, the framework employs several strategies:

  • Memory Pooling: Custom management of memory blocks reduces the overhead of frequent system calls.
  • In-place Operators: Aggressive use of in-place operations saves space by calculating gradients directly in the memory of precursor tensors.
  • Int8 Transmission: Images are transferred as int8 and converted to float32 only at the destination, minimizing data transfer bottlenecks.

Results

The framework was evaluated using the Cifar-10 dataset (60,000 32x32 color images) across several classic architectures: AlexNet, VGG (16 & 19), GoogleNet, and ResNet (18 & 34).

Performance Benchmarks

In head-to-head comparisons with PyTorch 1.7.1 (using cuDNN), Dragon-Alpha demonstrated superior performance:

  • Training Speed: Dragon-Alpha completed training epochs in 75.38% to 97.32% of the time required by PyTorch.
  • Memory Usage: Memory savings were significant, with Dragon-Alpha consuming only 29.2% to 66.4% of the GPU memory used by PyTorch. For instance, AlexNet required only 885MB on Dragon-Alpha compared to 3021MB on PyTorch.
  • Accuracy: Convergence and accuracy were generally comparable to PyTorch, validating the framework's effectiveness.

Discussion and Future Work

Dragon-Alpha demonstrates that Java can be a first-class citizen in the deep learning world, especially when integrated with its existing big-data ecosystem. The "small feature map" optimization in cu32 provides a distinct advantage for deep architectures.

However, limitations remain. The framework currently lacks support for sequence models like LSTMs and is primarily optimized for NVIDIA GPUs. Future work will focus on expanding operator support (including fused operators), enhancing the memory pool, and conducting larger-scale experiments within distributed Java big-data frameworks like Spark.

Summary updated

Abstract

Java is very powerful, but in Deep Learning field, its capabilities probably has not been sufficiently exploited. Compared to the Java-based deep-learning-frameworks, the Python-based (PyTorch, TensorFlow, etc) are undoubtedly the mainstream, due to their easy-to-use, flexibility and better ecosystem. Dragon-Alpha is a Java-based Tensor Computing Framework, with easy-to-use, high-scalability and high-performance, trying to break Java's dilemma in deep learning field and make it more effective. Dragon-Alpha supports different levels of APIs, and can be used as a deep-learning-framework through its user-friendly high-level APIs. Dragon-Alpha has potential to aggregate computing-power across heterogeneous platforms and devices, based on its multi-layer architecture and Java's big-data ecosystem. Dragon-Alpha has its asynchronized APIs to improve parallelism, and highly-optimized CUDA library cu32 which adopts unique convolution\deconvolution operators for small feature maps. The experiments show that, compared to PyTorch&cuDNN, Dragon-Alpha&cu32 costs less time and memory (75.38% to 97.32%, 29.2% to 66.4%), to train some typical neural networks (AlexNet, VGG, GoogleNet, ResNet) on Cifar-10.
BibTeX
@misc{zhang2023dragon,
  author = {Zhiyi Zhang and Pengfei Zhang and Qi Wang},
  title = {{Dragon-Alpha\&cu32: A Java-based Tensor Computing Framework With its High-Performance CUDA Library}},
  year = {2023},
  eprint = {2305.08819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.08819},
}
Figure