Logarithmic Compression for Scale Invariant Temporal Memory

This direction is merged into another direction: logarithmic temporal compression. Its last version stays readable and citable here.

Version 1 of 127 papers52 conceptsfrozen 2026-10-04

In the physical world, the meaning of a signal is often independent of its speed. A spoken sentence remains intelligible whether spoken quickly or slowly; a melody is recognizable regardless of its tempo. Yet, traditional artificial neural networks struggle with this "scale invariance." If a model is trained on data at one sampling rate, it often fails when that rate changes.

A growing body of research suggests that the solution lies in how we represent time itself. By shifting from a linear representation of time to a logarithmic one, we can transform the complex problem of temporal rescaling into a simple problem of translation.

The Mathematical Magic of Logarithms

The core intuition behind this research direction is a fundamental property of logarithms: the logarithm of a product is the sum of the logarithms.

$$\log(a \cdot t) = \log(a) + \log(t)$$

In the context of time-series data, $t$ represents time and $a$ represents a scaling factor (speeding up or slowing down). In a linear time domain, changing the speed $a$ stretches or compresses the signal. But in a logarithmic time domain, changing the speed simply shifts the entire signal by a constant amount ($\log a$).

As Brandon G. Jacques, Zoran Tiganj, and Marc W. Howard demonstrate in their work on SITHCon, this transformation is incredibly powerful. Because convolutional neural networks (CNNs) are already designed to be invariant to translations (shifts), applying a CNN to a log-compressed temporal memory makes the model naturally invariant to time rescaling. The network treats a "stretched" signal as the same underlying pattern, just shifted along the logarithmic axis.

Biological Inspiration and the Weber-Fechner Law

This mathematical trick isn't just a computational convenience; it appears to be how the brain handles time. Human perception follows the Weber-Fechner law, which states that the perceived change in a stimulus is proportional to the initial stimulus. We are much more sensitive to a one-second difference between two short intervals than a one-second difference between two hour-long intervals.

Feng Pan and Jerry R. Hobbs noted that learning event durations from text is most effective when using a logarithmic scale, aligning machine models with human temporal intuition. Similarly, Billy Dickson and Zoran Tiganj used this principle in Gradual Forgetting to extend Transformer context windows. By using log-spaced filters, they allow the model to maintain high resolution for recent events while "blurring" the distant past, mimicking the way biological "time cells" represent history.

Implementing Scale Invariance Across Architectures

The shift toward logarithmic temporal representations has been applied across several distinct neural architectures:

  • Recurrent Networks: M. Humayun Kabir and colleagues introduced CogRNN, which uses logarithmically spaced decay constants. This allows reinforcement learning agents to generalize to "slowed down" environments without retraining.
  • Convolutional Networks: Beyond SITHCon, David W. Romero developed Wavelet Networks, which use scale-translation equivariant layers to ensure consistent feature detection across different signal speeds.
  • State-Space Models: The HiPPO framework by Albert Gu and Christopher Re uses orthogonal polynomials to compress history. Their "scaled" measure (HiPPO-LegS) ensures that the importance of past data remains consistent regardless of the input's sampling rate.
  • Spiking Neural Networks: Even earlier, J. J. Hopfield and Sam T. Roweis proposed computing with action potentials where information is encoded as the logarithm of time relative to an internal rhythm, achieving scale invariance for odors and speech.

Why It Works: The Decomposition of What and When

A key advantage of this approach, highlighted in the DeepSITH architecture, is the decomposition of "what" happened from "when" it happened across multiple timescales.

In a linear system, a pattern occurring over 100ms looks entirely different to a neuron than the same pattern occurring over 200ms. In a logarithmic system with multi-scale representations—similar to the Multi-scale Successor Representations proposed by Marc W. Howard—the "what" (the pattern of activity) remains the same, while the "when" (the scale) is simply a shift in which units are active.

This leads to zero-shot generalization. Models trained on a specific speed can immediately process data that is an order of magnitude faster or slower. As Zhongyi Yu shows in TAMI, even in messy real-world data like temporal graphs, applying a logarithmic transformation to time intervals reduces the skewness of the data, making it significantly easier for neural networks to optimize and learn.

By embracing the logarithmic nature of time, researchers are moving away from rigid, sampling-rate-dependent models toward fluid, scale-invariant systems that perceive the world more like we do.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_logarithmic_compression_for_scale_invari_ebdfce94,
  title = {Logarithmic Compression for Scale Invariant Temporal Memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logarithmic-compression-for-scale-invariant-temporal-memory/rdr_ebdfce941ec14d7f8354754c3e0a151b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_logarithmic_compression_for_scale_invari_ebdfce94_v1,
  title = {Logarithmic Compression for Scale Invariant Temporal Memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logarithmic-compression-for-scale-invariant-temporal-memory/rdr_ebdfce941ec14d7f8354754c3e0a151b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}