Hierarchical Feature-level Reverse Propagation for Post-Training Neural Networks

Introduction

Deep neural networks (DNNs), particularly end-to-end (E2E) models, have become the cornerstone of modern artificial intelligence, achieving state-of-the-art performance in complex domains like autonomous driving. However, these models are often criticized as "black boxes" due to their highly entangled internal representations. This lack of transparency makes it difficult to interpret specific decisions or to update individual components without a full, computationally expensive retraining of the entire system.

While modular architectures offer better interpretability by separating tasks like detection and planning, they frequently fall short of the raw performance seen in E2E systems. This paper proposes a middle ground: Hierarchical Feature-level Reverse Propagation (FR-PT). FR-PT is a post-training framework designed to decouple network layers, allowing for independent optimization and improved model interpretability without sacrificing the performance benefits of E2E architectures.

Expand

Source, p. 1

Methodology: Decoupling via Feature Reconstruction

The fundamental shift in FR-PT is the replacement of standard gradient-based backpropagation with "feature backpropagation." Instead of propagating error gradients, the framework mathematically reconstructs the "ideal" feature maps ($a^*_l$) that should have been produced at each layer to yield the correct ground-truth label.

Optimal Embedding (OE)

The process begins at the output layer. Standard classification labels are scalars, which cannot be directly backpropagated as features. The authors introduce Optimal Embedding, a "Nearest Embedding" algorithm that maps these scalar labels back into high-dimensional output vectors. This algorithm identifies a vector $a^*_L$ that is closest to the network's actual output $\hat{a}_L$ while strictly satisfying the conditions for correct classification.

Layer-wise Feature Reconstruction

Once the optimal output embedding is established, the framework reverses the network's operations layer-by-layer. This involves two core principles:

1. Minimal Deviation Principle (MDP): Applied when a layer expands the number of channels (an underdetermined system). MDP finds a reconstructed feature map that minimizes the distance to the original forward-pass feature, ensuring the updated representation remains grounded in the model's learned knowledge.
2. Computing Consistency Principle (CCP): Applied when a layer contracts channels (an overdetermined system). CCP identifies the feature map that minimizes the overall computational error relative to the subsequent layer's requirements.

Expand

Source, p. 2

To handle the computational complexity of reversing convolutional operations, the authors employ Fast Fourier Transforms (FFT). By leveraging the convolution theorem, they transform spatial convolutions into point-wise multiplications in the frequency domain, significantly accelerating the solution of the underlying systems of linear equations.

Experimental Results and Insights

The FR-PT framework was evaluated across several standard benchmarks, including MNIST, CIFAR-10, CIFAR-100, Tiny ImageNet, ImageNette, and ImageWoof.

Performance and Efficiency

The results demonstrate that FR-PT consistently outperforms standard backpropagation-based post-training (BP-PT) in terms of test accuracy. Specifically, on datasets like CIFAR-100 and ImageWoof, the "rectified" internal representations led to superior generalization. Furthermore, because FR-PT allows for the independent training of specific modules, it is significantly more memory-efficient than full-network backpropagation, requiring less GPU overhead to optimize deep layers.

Interpretability and "Internal Confusion"

A significant advantage of FR-PT is its utility as a diagnostic tool. By calculating the discrepancy (deviation) between the forward-pass feature map ($\hat{a}_l$) and the reconstructed ideal map ($a^*_l$), researchers can visualize where a network is "confused." As a model converges, these maps align; persistent deviations highlight specific layers or regions where the model's internal logic fails to support the correct output.

Discussion and Limitations

FR-PT is most effective as a refinement step for models that are already near convergence. While standard backpropagation is better suited for the initial "underfit" stages of training, FR-PT excels at fine-tuning and "rectifying" representations in overfit or nearly-converged models.

However, the method faces challenges with error accumulation. As the reconstruction propagates further back toward the input layers, information loss can compound, making it less effective for optimizing the very first layers of a deep network. Additionally, the current implementation is tailored for convolutional neural networks (CNNs).

Conclusion

Hierarchical Feature-level Reverse Propagation offers a mathematically rigorous path toward cracking the neural black box. By decoupling layers through feature reconstruction, it provides a framework that balances the high performance of E2E models with the transparency of modular systems. Future work will focus on extending these principles to more diverse architectures, such as Transformers and ResNets, and applying them to multi-task autonomous driving pipelines.