SynJax: Structured Probability Distributions for JAX

Miloš Stanojević, Laurent SartranEMNLP 2023

TL;DR

SynJax is a new library for JAX that makes it easy to build and train deep learning models with complex, structured outputs like trees and sequences. It achieves this by leveraging automatic differentiation to efficiently compute probabilities and other quantities for these structures, enabling faster and more scalable research in areas like natural language processing and computational biology.

What Problem Does This Paper Address?

Deep learning has revolutionized many fields, but its success has largely been driven by models with simple, vectorized computations, like Transformers. However, many important problems involve data with inherent structure – think of the hierarchical relationships in language (syntax trees), the sequential nature of DNA, or the alignments between different languages. Building deep learning models that explicitly account for these structures has been challenging. Existing approaches often require custom, hard-to-vectorize algorithms, making them slow and difficult to scale on modern hardware accelerators like GPUs and TPUs. This gap limits our ability to build more powerful and interpretable models for a wide range of applications.

Methods

SynJax tackles this by providing a unified interface for various structured probability distributions within the JAX ecosystem. The core idea is to exploit the connection between automatic differentiation and probabilistic inference. Instead of writing separate, complex inference algorithms for each specific structure and task (like sampling, finding the most likely structure, or computing marginal probabilities), SynJax allows users to define the structure and its potentials. The library then automatically derives the necessary inference computations using techniques like semirings and backpropagation.

Key aspects of SynJax's approach include:

  • Vectorized Implementations: SynJax provides efficient, vectorized implementations of inference algorithms for structures like linear-chain CRFs, semi-Markov CRFs, tree-CRFs, PCFGs, and various types of spanning trees.
  • Automatic Differentiation: By treating structured distributions as differentiable functions, SynJax can compute gradients and other quantities needed for training deep learning models. This means users don't need to manually derive these complex relationships.
  • Unified Interface: SynJax offers a consistent API for a wide range of structured distributions, abstracting away the underlying algorithmic complexities. Users can switch between different structure types or constraints (e.g., projective vs. non-projective spanning trees) by simply changing a few parameters.
  • Leveraging JAX: Built on JAX, SynJax benefits from its automatic differentiation, JIT compilation, and ability to run on accelerators. It also integrates well with other JAX libraries like Equinox, Haiku, and Flax.

Datasets

Not discussed in the paper. The paper focuses on the library's capabilities and performance rather than specific model training on particular datasets.

Key Insights and Ideas

  • Structure is Key for Generalization: Explicitly modeling structure in data can provide strong inductive biases, leading to better generalization and sample efficiency in deep learning models.
  • The Power of Differentiable Inference: By making inference algorithms differentiable, we can seamlessly integrate structured models into end-to-end deep learning pipelines and train them using gradient-based methods.
  • Unified Algorithms for Diverse Structures: The paper demonstrates that many inference tasks (sampling, argmax, marginals, entropy) can be handled by a common framework (semirings, backpropagation) across different structured distributions, reducing the need for highly specialized code for each.
  • Vectorization is Crucial for Performance: The significant speedups observed (e.g., 71x for Non-Projective CRF) highlight the importance of efficient, vectorized implementations for structured inference on modern hardware.

Key Results

  • Significant Speedups: SynJax achieves substantial speedups compared to existing libraries like Torch-Struct (up to 71x for Non-Projective CRF) and custom implementations for spanning trees (e.g., 3x faster for Projective Maximum Spanning Tree algorithms).
  • Concise Code: SynJax implementations are significantly more concise than alternatives, as demonstrated by the line-of-code comparisons.
  • Broad Distribution Support: The library supports a wide array of structured distributions relevant to NLP and other fields.
  • Unified Interface: SynJax provides a single, user-friendly interface for 8 different types of spanning tree structures, a feature not found in other libraries.

Limitations and Future Work

  • Performance Bottlenecks: The paper acknowledges that large temporary tensors in dynamic programming algorithms can be a bottleneck for speed and memory, especially on TPUs. Optimizations using custom kernels (like Pallas) are suggested.
  • Hardware Acceleration: While SynJax uses JAX for acceleration, some operations (like matrix multiplication with semirings) are not yet fully leveraging specialized hardware like TensorCores on GPUs, relying instead on regular CUDA cores.
  • Vectorizing Complex Algorithms: Some algorithms, like maximum spanning tree, are still executed as optimized Numba CPU code rather than being fully vectorized, limiting their parallelization potential.
  • Future Work: The authors hope SynJax will encourage further research into structured models and inspire new alternatives to auto-regressive approaches. They also mention the potential for further optimization by leveraging specialized hardware more effectively.

Summary updated

BibTeX
@inproceedings{stanojevic2023synjax,
  author = {Miloš Stanojević and Laurent Sartran},
  title = {{SynJax: Structured Probability Distributions for JAX}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure