iMacSR: Intermediate Multi-Access Supervision and Regularization in Training Autonomous Driving Models

Wei-Bin Kou, Guangxu Zhu, Yichen Jin, Shuai Wang, Ming Tang, Yik-Chung WuarXiv 2025 · DOI · arXiv

TL;DR

iMacSR is a novel training framework designed to enhance the performance of deep neural networks by strategically integrating supervision and regularization at intermediate layers. By employing diverse loss functions and penalizing overconfident hidden activations, iMacSR facilitates the learning of robust, hierarchical features. This approach leads to significant accuracy improvements—up to 9.19% in Mean Intersection over Union (mIoU)—across various architectures used in autonomous driving.

Addressing the Optimization Gap in Deep Networks

Training very deep neural networks presents significant challenges. While residual connections and batch normalization have mitigated some issues, intermediate layers often remain "under-optimized," failing to develop the most effective features for complex tasks like semantic segmentation.

A common remedy is Deep Supervision, which introduces auxiliary "mini-losses" throughout the network. However, traditional deep supervision suffers from three primary limitations:

  1. Rigidity: Implementations are often hard-coded for specific architectures (e.g., ResNet), limiting portability.
  2. Premature Specialization: Using the same loss function (like Cross-Entropy) for both intermediate and final layers forces early layers to predict the final output too soon, hindering the development of complex representations.
  3. Overconfidence: A lack of regularization on hidden layers can lead to overconfident intermediate predictions, which degrades generalization on unseen data.

The iMacSR Framework

To overcome these hurdles, the authors propose iMacSR (Intermediate Multi-access Heterogeneous Supervision and Regularization). The framework is built upon three core pillars designed to be architecture-agnostic and robust.

Overview of the iMacSR framework, illustrating the integration of intermediate supervision and regularization via a dedicated bus.

Core Pillars of iMacSR:

  • Architecture-Agnostic Selection: Instead of targeting specific layer indices, iMacSR identifies logical "transition points"—such as before/after downsampling or at feature fusion points. This makes the framework a "plug-and-play" solution for both CNNs and Transformers.
  • Heterogeneous Supervision: Rather than standard Cross-Entropy, iMacSR utilizes Mutual Information (MI) loss for intermediate layers. This encourages the network's middle to capture maximum information about the ground truth without forcing it to mimic the final output layer's behavior prematurely.
  • Negative Entropy Regularization: To prevent the model from becoming "brittle," a negative entropy penalty is applied to intermediate activations. This forces the model to maintain a healthy level of uncertainty in its internal representations, typically leading to better generalization.

To facilitate this, small Dimension Adapters align internal feature map shapes with ground truth labels. The authors also provided mathematical proof that this added complexity does not compromise the model's convergence rate.

Experimental Evaluation

The researchers evaluated iMacSR on three prominent semantic segmentation benchmarks: Cityscapes (urban street scenes), CamVid (road-scenes), and SynthiaSF (synthetic urban environments). The framework was tested across three distinct model types: DeepLabv3+ (CNN-based), and SeaFormer and TopFormer (Transformer-based).

Performance Gains and Insights

The results demonstrate that the "middle" of a network should not be trained identically to the end. By using Mutual Information as a heterogeneous loss, intermediate layers focus on feature richness rather than just final classification.

Comparison of training performance between conventional methods and iMacSR across mIoU, mPrecision, mRecall, and mF1 metrics.

Key Results:

  • Accuracy Boost: On the Cityscapes dataset, iMacSR improved the mIoU of DeepLabv3+ by a substantial 9.19%.
  • Generalization: Qualitative visual maps showed that models trained with iMacSR were significantly better at capturing intricate details and broad scene context compared to those using standard training.
  • Efficiency: Despite the auxiliary losses, the time and space overhead remains linear relative to the number of intermediate points. For typical configurations (3–5 points), the overhead is marginal compared to the performance gains.

Conclusion and Future Work

iMacSR demonstrates that targeted regularization and heterogeneous supervision in the intermediate layers can significantly boost performance in semantic segmentation for autonomous driving. While the current work focuses on segmentation, the principles of iMacSR could potentially translate to other domains like Large Language Models (LLMs). Future research may explore automated weight tuning for intermediate losses—perhaps via meta-learning—to automatically find the optimal balance for any given architecture.

Summary updated

Abstract

While deep supervision is a powerful training strategy by supervising intermediate layers with auxiliary losses, it faces three underexplored problems: (I) Existing deep supervision techniques are generally bond with specific model architectures strictly, lacking generality. (II) The identical loss function for intermediate and output layers causes intermediate layers to prioritize output-specific features prematurely, limiting generalizable representations. (III) Lacking regularization on hidden activations risks overconfident predictions, reducing generalization to unseen scenarios. To tackle these challenges, we propose an architecture-agnostic, intermediate Multi-access Heterogeneous Supervision and Regularization (iMacHSR) scheme. Specifically, the proposed iMacHSR introduces below integral strategies: (I) we select multiple intermediate layers based on predefined architecture-agnostic standards; (II) loss functions (different from output-layer loss) are applied to those selected intermediate layers, which can guide intermediate layers to learn diverse and hierarchical representations; and (III) negative entropy regularization on selected layers' hidden features discourages overconfident predictions and mitigates overfitting. These intermediate terms are combined into the output-layer training loss to form a unified optimization objective, enabling comprehensive optimization across the network hierarchy. We then take the semantic understanding task as an example to assess iMacHSR and apply iMacHSR to several model architectures. Extensive experiments on multiple datasets demonstrate that iMacHSR outperforms conventional output-layer single-point supervision method up to 9.19% in mIoU.
BibTeX
@misc{kou2025imacsr,
  author = {Wei-Bin Kou and Guangxu Zhu and Yichen Jin and Shuai Wang and Ming Tang and Yik-Chung Wu},
  title = {{iMacSR: Intermediate Multi-Access Supervision and Regularization in Training Autonomous Driving Models}},
  year = {2025},
  eprint = {2505.00404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.00404},
}
Figure