Deep Learning Without Neural Networks: Fractal-nets for Rare Event Modeling

Ishanu Chattopadhyay, Yi Huang, James A. EvansResearch 2020 · DOI

Abstract

Complex phenomena of societal interest such as weather, seismic activity and urban crime, are often punctuated by rare and extreme events, which are difficult to model and predict. Evidence of long-range persistence of such events has underscored the need to learn deep stochastic structures in data for effective forecasts. Recently neural networks (NN) have emerged as a defacto standard for deep learning. However, key problems remain with NN inference, including a high sample complexity, a general lack of transparency, and a limited ability to directly model stochastic phenomena. In this study we suggest that deep learning and the NN paradigm are conceptually distinct -- and that it is possible to learn "deep" associations without invoking the ubiquitous NN strategy of global optimization via back-propagation. We show that deep learning of stochastic phenomena is related to uncovering the emergent self-similarities in data, which avoids the NN pitfalls offering crucial insights into underlying mechanisms. Using the Fractal Net (FN) architecture introduced here, we actionably forecast various categories of rare weather and seismic events, and property and violent crimes in major US cities. Compared to carefully tuned NNs, we boost recall at 90% precision by 161.9% for extreme weather events, 191.3% for light-to-severe seismic events with magnitudes above the local third quartile, and 50.8% - 404.9% for urban crime, demonstrating applicability in diverse systems of societal interest. This study opens the door to precise prediction of rare events in spatio-temporal phenomena, adding a new tool to the data science revolution.

Introduction

Predicting rare events—those occurring with a frequency of less than 10%—presents a significant challenge for modern data science. Whether forecasting sudden market fluctuations, violent crimes, or major seismic shifts, these events are often characterized by "long-range persistence," where critical patterns are buried deep within historical data. While Recurrent Neural Networks (RNNs) like LSTMs are currently the standard for sequence modeling, they struggle with high sample complexity, a lack of transparency, and the "vanishing gradient" problem, which prevents them from learning long-term dependencies effectively.

This research challenges the assumption that deep learning must rely on neural networks and backpropagation. Instead, the authors introduce Fractal Nets (FN), an architecture rooted in the mathematics of self-similar stochastic processes, designed specifically for high-stakes, rare-event prediction.

Methodology: Fractal Nets and Self-Similar Compression

Fractal Nets depart from the traditional layer-of-neurons approach, utilizing Probabilistic Finite State Automata (PFSA). The core of the system is the Self-Similar Compression (SSC) algorithm. In ergodic systems, future paths of a process are often scaled copies of the overall process. SSC identifies these fractal structures and "stitches" them into a state machine.

Key innovations include:

  • Local Model Generation: Unlike NNs that use global optimization, FNs infer local state machines (elemental units) to capture dependencies between data streams.
  • Linear Combination of Non-linearities: FNs combine the outputs of many local non-linear models linearly, allowing for on-the-fly updates without full retraining.
  • Coefficient of Causality ($\gamma$): A stable measure of information flow between streams that does not decay over long time horizons, effectively solving the vanishing gradient problem.

Experimental Results

The researchers evaluated Fractal Nets across diverse domains, including US weather patterns, global seismic activity, and urban crime logs from major cities like Chicago, Atlanta, and Philadelphia.

Extreme Weather and Seismic Activity

Fractal Nets demonstrated a remarkable ability to capture extreme weather events. For US weather data, FNs achieved a 161.9% boost in recall at 90% precision compared to LSTMs, while using significantly fewer parameters (~4.8 million vs. up to 66 million for LSTMs).

Predictive risk maps for extreme weather events across the US, showing high precision and recall for precipitation and cold/snow risks.

In seismic modeling, FNs improved recall by 191.3% for events above the local third quartile, providing actionable lead times of up to 120 days.

Urban Crime Forecasting

In the domain of urban crime, FNs revealed that cities operate with "long-ranged coherence," where crime patterns across a city are interconnected. The model achieved recall boosts between 50.8% and 404.9% across different crime categories and cities.

Urban crime risk modeling in Chicago, demonstrating the Fractal Net's ability to capture spatial coherence in violent and property crimes.

Discussion and Conclusion

The success of Fractal Nets suggests that the "depth" in deep learning can be achieved through temporal complexity in state machines rather than just stacking neural layers. By avoiding backpropagation, FNs provide a more transparent and computationally efficient framework for modeling stochastic phenomena.

While currently limited by the need for categorical data quantization, future work aims to extend this architecture to continuous data and even rarer "Black Swan" events. This study underscores that the future of AI may lie in diverse mathematical frameworks that better reflect the self-similar nature of the physical and social world.

Summary updated

Abstract

Abstract Complex phenomena of societal interest such as weather, seismic activity and urban crime, are often punctuated by rare and extreme events, which are difficult to model and predict. Evidence of long-range persistence of such events has underscored the need to learn deep stochastic structures in data for effective forecasts. Recently neural networks (NN) have emerged as a defacto standard for deep learning. However, key problems remain with NN inference, including a high sample complexity, a general lack of transparency, and a limited ability to directly model stochastic phenomena. In this study we suggest that deep learning and the NN paradigm are conceptually distinct -- and that it is possible to learn ``deep' associations without invoking the ubiquitous NN strategy of global optimization via back-propagation. We show that deep learning of stochastic phenomena is related to uncovering the emergent self-similarities in data, which avoids the NN pitfalls offering crucial insights into underlying mechanisms. Using the Fractal Net (FN) architecture introduced here, we actionably forecast various categories of rare weather and seismic events, and property and violent crimes in major US cities. Compared to carefully tuned NNs, we boost recall at 90% precision by 161.9% for extreme weather events, 191.3% for light-to-severe seismic events with magnitudes above the local third quartile, and 50.8% - 404.9% for urban crime, demonstrating applicability in diverse systems of societal interest. This study opens the door to precise prediction of rare events in spatio-temporal phenomena, adding a new tool to the data science revolution.
BibTeX
@inproceedings{chattopadhyay2020deep,
  author = {Ishanu Chattopadhyay and Yi Huang and James A. Evans},
  title = {{Deep Learning Without Neural Networks: Fractal-nets for Rare Event Modeling}},
  booktitle = {Research},
  year = {2020},
  doi = {10.21203/rs.3.rs-86045/v1},
}
Figure