Human conversation is inherently nested: words form utterances, and utterances form a dialogue. While standard language models often treat text as a flat sequence of tokens, a growing body of research argues that modeling this natural hierarchy explicitly is essential for capturing long-range context and discourse structure.
The Two-Tier Architecture
The core intuition behind hierarchical modeling is to separate local semantic extraction from global context tracking. Instead of processing a conversation as one long string of words, these models employ a dual-stage encoding process.
In the first stage, a lower-level encoder—often a Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), or Transformer—processes individual tokens to generate a fixed-length representation for each utterance. In the second stage, a higher-level encoder treats these utterance representations as a new sequence, modeling the dependencies and flow across the entire conversation.
Sordoni et al. pioneered this approach with the Hierarchical Recurrent Encoder-Decoder (HRED), which uses one recurrent layer for words and a second for the sequence of sentences. This structure allows the model to maintain context across multiple turns by separating local word dependencies from global dialogue dynamics. Similarly, Jiao introduced HiGRU, which utilizes Gated Recurrent Units at both levels to capture the natural hierarchy of conversation.
Capturing Long-Range Dependencies
Flat models often suffer from "vanishing context," where information from early in a conversation is lost by the time the model generates a response. Hierarchical architectures mitigate this by providing a "shortcut" for information flow. By summarizing an entire sentence into a single vector, the higher-level encoder can process dozens of turns in a fraction of the steps required by a token-level model.
Research by Noseworthy et al. and Noseworthy et al. demonstrates that this two-tier approach is particularly effective for predicting the success of goal-driven dialogues, as it captures both the internal semantics of a turn and the broader problem-solving trajectory. Tanaka et al. and Arase further show that this structure enables the prediction of future "Dialogue Acts"—the functional intent of a speaker—by tracking the progression of the conversation over time Tanaka et al..
Integrating Structural and Speaker Metadata
Hierarchical models provide natural "hooks" to integrate high-level metadata that flat sequences struggle to incorporate. Because the model has a dedicated layer for utterances, it can explicitly account for speaker identity, turn order, and topic boundaries.
- Speaker Awareness: Zhang et al. and Zhang et al. developed the SCFA architecture, which uses speaker-aware transformers to capture how an individual's emotional state is influenced by their own history versus their partner's reactions.
- Structural Positions: To help models understand the "where" of a word, Belainine et al. proposed a dual-axis position coding scheme. This assigns two vectors to each token: one for its position within the utterance and another for the utterance's position within the dialogue.
- Topic Evolution: Sun et al. introduced ConvNTM, a neural topic model that uses a hierarchical encoder to track how topics shift across turns and how different roles influence that evolution.
Beyond Text: Multi-Modal and Asynchronous Hierarchies
The hierarchical principle extends beyond standard text-based chatbots. In spoken language understanding, Sunder et al. demonstrated that audio segments can be encoded into utterance-level vectors and then aggregated to capture context directly from speech signals, bypassing the need for text transcripts.
The architecture also adapts to asynchronous environments like email or forums. Mohiuddin showed that hierarchical LSTMs can bridge the gap between synchronous meetings and asynchronous threads for speech act recognition. In the realm of visual dialogue, Siddharth N treated dialogue history as a multi-channel "image" where each turn is a separate channel, allowing convolutional layers to capture dependencies across both words and turns simultaneously.
Evolution Toward Hierarchical Transformers
While early work focused on RNNs, recent research has shifted toward hierarchical Transformer architectures. Chapuis et al. and Chapuis et al. argue that a dual-stage Transformer—where a lower encoder generates utterance representations and a higher encoder models their relationships—more accurately reflects the nested structure of human conversation than a flat Transformer.
This evolution has led to specialized pre-training tasks. Li proposed turn-level masked language modeling to encourage models to generate discrete summary embeddings for each speaker turn. Similarly, Paranjape et al. and Neubig used hierarchical encoders to predict the content of surrounding utterances, capturing deep conversational context in an unsupervised manner Paranjape et al..
Key Foundational Work
Automatic social role recognition in professional meetings using conditional random fields (2013) — contributed 1 concept elements to this direction.
Finding Problem Solving Threads in Online Forum (2011) — contributed 1 concept elements to this direction.