Abstract
This paper presents ENHAnCE, an algorithm that simultaneously learns a predictive model of the input stream and generates representations of the concepts being observed. Following cognitively-inspired models of event segmentation, ENHAnCE uses expectation violations to identify boundaries between temporally extended patterns. It applies its expectation-driven process at multiple levels of temporal granularity to produce a hierarchy of predictive models that enable it to identify concepts at multiple levels of temporal abstraction. Evaluations show that the temporal abstraction hierarchies generated by ENHAnCE closely match hand-coded hierarchies for the test data streams. Given language data streams, ENHAnCE learns a hierarchy of predictive models that capture basic units of both spoken and written language: morphemes, lexemes, phonemes, syllables, and words.
Introduction
For an AI to understand the world, it must make sense of continuous sensory input. Consider the task of speech recognition: a robot does not perceive discrete spaces between words; it receives a continuous stream of acoustic signals. Humans naturally segment such streams into discrete units—events or concepts—across multiple levels of abstraction.
Most current machine learning approaches rely on pre-segmented or human-labeled data. The core challenge addressed here is Unsupervised Hierarchical Temporal Abstraction: the autonomous discovery of multi-layered structures (e.g., characters to syllables to words) from raw, unsegmented data.
The ENHAnCE Algorithm
The authors introduce ENHAnCE (Expectation driveN HierarhicAl Concept lEarning), a framework grounded in Event Segmentation Theory (EST). EST posits that humans identify boundaries between events when internal predictions about the immediate future fail.
The architecture operates in an iterative, bottom-up manner:
- Prediction (SRNN): A base layer utilizing a Simple Recurrent Neural Network (SRNN) attempts to predict the next observation in the stream.
- Boundary Detection (RL + Gating): When prediction error (Sum Squared Error) spikes, a Reinforcement Learning policy learns to trigger a "gate," signaling the end of one concept and the start of another.
- Representation (GSOM): A Growing Self-Organizing Map (GSOM) clusters the network's internal states at these boundary points, creating "prototypes" for discrete concepts.
- Stacking the Hierarchy: Once a layer stabilizes, a new layer is added. This higher layer takes the discrete concepts from the lower layer as its input, repeating the process to discover more abstract patterns over longer time scales.
Experimental Setup
ENHAnCE was evaluated on language data due to its well-defined, objective hierarchies (phonemes, syllables, words) which serve as ground truth.
- Saffran Infant Data: A sequence of 90 words from developmental psychology experiments studying word boundary learning.
- Orwell’s 1984: The first 30,000 words of the novel, offering a complex vocabulary.
- Encodings: Two representations were tested: Morse code (binary) and phonemes (numeric codes for speech sounds).
Results and Discussion
ENHAnCE successfully identified concept boundaries matching human-defined linguistic units. On the 1984 phoneme dataset, it achieved a hit rate of over 90% across multiple levels. It outperformed baselines like Hierarchical Voting Experts (HVE), which often struggles with sub-sequences—short patterns embedded within longer ones. ENHAnCE’s predictive model handles these overlaps more gracefully.
Key insights include:
- Error as a Signal: Rather than just a metric to minimize, "expectation failure" is a vital signal defining concept boundaries.
- Stability-Driven Growth: The model masters lower levels (e.g., characters) before attempting higher-level patterns (e.g., words), preventing noise propagation.
- Recurrent Memory: Unlike frequency-based methods, ENHAnCE uses recurrent memory to distinguish patterns based on prior context.
Conclusion and Future Work
While successful, ENHAnCE currently relies on a strictly bottom-up flow. Future work may incorporate "top-down" feedback, where higher-level understanding (e.g., a "word") helps resolve lower-level ambiguities (e.g., a "phoneme"). Additionally, the authors aim to address computational complexity at higher abstraction levels and move toward "one-shot" learning for instant concept recognition in new contexts.
Summary updated