In traditional deep learning architectures, feature aggregation is largely passive. Convolutional neural networks (CNNs) rely on operations like max-pooling or average-pooling to reduce spatial dimensions and create translation invariance. While effective, max-pooling is a blunt instrument: it simply selects the most intense activation within a local window, discarding critical information about the spatial relationships, orientations, and structural hierarchies between features.
To overcome this limitation, Sara Sabour et al. introduced a paradigm shift with Capsule Networks (CapsNets) in their seminal paper, Dynamic Routing Between Capsules. Instead of using scalar-valued neurons, they proposed grouping neurons into vector-valued "capsules." To pass information between layers of capsules, they replaced static pooling with an iterative routing-by-agreement mechanism. This approach allows a network to dynamically construct a flexible, input-specific parse tree, ensuring that lower-level features are only aggregated into a higher-level concept if they collectively agree on its properties.
The Intuition: Vector Activations and Part-Whole Relationships
To understand why dynamic routing works, we must first look at how capsules represent information. A standard neuron outputs a single scalar representing the probability that a feature exists. A capsule, by contrast, outputs a vector.
- Vector Length: Represents the probability that the entity or feature is present in the input.
- Vector Orientation: Encodes the specific instantiation parameters of that feature, such as its precise position, scale, rotation, texture, or tone.
Because features are represented as vectors, the network can evaluate structural compatibility. Imagine trying to detect a human face. A traditional CNN looks for the presence of eyes, a nose, and a mouth. If all three are present—even if the mouth is above the eyes—max-pooling layers may still trigger a "face" classification because they only track the presence of the strongest signals, not their relative geometry.
Dynamic routing solves this "part-whole" problem through an active negotiation process between layers.
How the Routing Loop Works
The core mechanism of routing-by-agreement is an iterative expectation-maximization-like loop executed during the forward pass. Rather than relying on fixed weights learned solely during backpropagation, the network dynamically computes coupling coefficients ($c_{ij}$) for each specific input.
[Lower-Level Capsules (i)]
│
├─► Multiply by Transformation Matrices (W_ij) ──► [Predicted Parent Vectors (u_j|i)]
│ │
▼ ▼
[Compute Coupling Coefficients (c_ij)] ◄─── Update via Agreement ─── [Aggregate Parent Capsules (s_j)]
- Prediction Generation: Every lower-level capsule $i$ multiplies its own output vector $u_i$ by a learned transformation matrix $W_{ij}$ to generate a prediction vector, $\hat{u}_{j|i}$. This vector represents what the lower-level capsule thinks the higher-level capsule $j$'s state should be. For instance, an "eye" capsule uses $W_{ij}$ to predict the position and orientation of the entire "face" capsule based on its own geometry.
- Initial Allocation: Initially, the coupling coefficients $c_{ij}$—which determine how much information capsule $i$ sends to capsule $j$—are initialized uniformly. They are passed through a softmax function to ensure that all outputs from a single lower-level capsule sum to one.
- Aggregation and Squashing: The target capsule $j$ computes a weighted sum of all incoming predictions: $s_j = \sum_i c_{ij} \hat{u}_{j|i}$. To ensure its length remains a valid probability between 0 and 1, $s_j$ is passed through a non-linear "squashing" function to produce the final capsule output $v_j$.
- Agreement Update: This is where the "agreement" happens. The network calculates the scalar dot product between the actual output of the parent capsule ($v_j$) and the initial prediction made by the child capsule ($\hat{u}_{j|i}$).
$$\text{Agreement} = v_j \cdot \hat{u}_{j|i}$$
If the child’s prediction aligns closely with the final consensus vector, the dot product is large and positive. The network leverages this feedback to increase the coupling coefficient $c_{ij}$ for the next iteration. If the prediction diverges, the coefficient shrinks.
Over a few iterations (typically 3), this loop converges. The network effectively routes clusters of mutually compatible features to the correct higher-level parent, while naturally filtering out noise and irrelevant background clutter.
Generalizing Beyond Computer Vision
While originally designed to preserve spatial hierarchies in images, the intuition behind dynamic routing—replacing passive pooling with an intelligent, content-dependent clustering mechanism—has proven highly effective across diverse domains.
Natural Language Processing
In text processing, sentences exhibit complex hierarchical syntactic structures rather than simple linear sequences. Jingjing Gong et al. adapted this mechanism for sequence encoding in Information Aggregation via Dynamic Routing for Sequence Encoding. Instead of squashing a sequence of word vectors into a single vector via max-pooling, their model uses dynamic routing to let source and target nodes negotiate information transfer, preserving semantic structures.
Similarly, Congying Xia et al. applied routing-by-agreement to Zero-shot User Intent Detection via Capsule Neural Networks. By ensuring that a high-level intent capsule (e.g., "Book a Flight") is only activated when there is a strict consensus among its constituent semantic parts, the network can accurately identify entirely unseen user commands based purely on structural composition. For identifying complex, overlapping relationships within sentences, Ningyu Zhang et al. demonstrated that iterative agreement routing outperforms static pooling by dynamically determining information flow based on the compatibility between local word features and global relation structures (Attention-Based Capsule Network with Dynamic Routing for Relation Extraction).
Audio and Speech Processing
Audio spectrograms contain intricate temporal and spatial patterns. Jaesung Bae et al. utilized dynamic routing to replace max-pooling in End-to-End Speech Command Recognition with Capsule Network. The iterative voting process allows the network to capture the precise spatial-temporal relationships of speech features, resulting in an 11% error rate reduction compared to standard convolutional baselines.
Multi-Instance Learning
In Multi-Instance Learning (MIL), data is organized into "bags" of instances, where a bag is labeled positive if at least one instance within it is positive. Xinggang Wang et al. developed a framework that replaces static pooling functions with an iterative dynamic pooling mechanism (Deep Multi-instance Learning with Dynamic Pooling). By iteratively updating instance weights based on their agreement with the aggregated bag embedding, the model adaptively isolates key instances and filters out uninformative noise.
The Scaling Bottleneck
Despite its conceptual elegance, dynamic routing faces significant practical challenges. The iterative nature of the routing loop requires calculating large numbers of dot products and executing multiple forward passes per layer during both training and inference. This introduces a substantial computational bottleneck that prevents standard capsule networks from scaling efficiently to large-scale datasets like ImageNet.
In Why Capsule Neural Networks Do Not Scale: Challenging the Dynamic Parse-Tree Assumption, Matthias Mitterreiter investigated these structural limitations. The author demonstrated that the current implementation of the routing-by-agreement mechanism often suffers from a vanishing gradient problem that starves capsules during training. As a result, the network frequently fails to form the diverse, input-specific dynamic parse-trees promised by the theory, leaving many capsules underutilized.
Summary
Dynamic routing by agreement replaces the rigid, lossy operations of traditional pooling layers with an intelligent, iterative voting mechanism. By representing features as multi-dimensional vectors and routing them based on geometric and semantic compatibility, it provides a mathematically rigorous framework for capturing part-whole relationships. While scaling issues remain an active area of research, the core philosophy of dynamic routing continues to inspire architectures that prioritize structural configuration over simple feature presence.
Key Foundational Work
Attention-Based Capsule Network with Dynamic Routing for Relation Extraction (2018) — contributed 2 concept elements to this direction.
Attention-Based Capsule Networks with Dynamic Routing for Relation Extraction (2018) — contributed 2 concept elements to this direction.
Dynamic Routing Between Capsules (2017) — contributed 1 concept elements to this direction.