Self-Supervised Interpretable End-to-End Learning via Latent Functional Modularity

Hyunki Seong, Hyunchul Shim

End-to-end learning for autonomous navigation often feels like a "black box." You feed in camera images, and out come steering commands, but the internal logic remains a mystery. This lack of transparency makes it difficult to trust these systems in safety-critical real-world environments.

In a recent paper, researchers introduced MoNet, a modular network architecture that doesn't just drive—it explains why it’s making certain decisions, all while learning in a self-supervised fashion.

TL;DR

MoNet is a modular end-to-end network for autonomous navigation that learns task-specific decisions (like "turn left" or "avoid obstacle") without manual labels. By using a novel latent-guided contrastive loss and a post-hoc interpretation layer, it achieves superior navigation performance and provides human-readable explanations for its internal decision-making process.

What Problem Does This Paper Address?

Traditional end-to-end sensorimotor networks directly map sensory inputs to control outputs. While efficient, they suffer from two major flaws:

  1. Obscurity: It’s unclear what high-level behavior the network intended (e.g., did it turn because of the intersection or to avoid a shadow?).
  2. Label Dependency: To learn complex tasks, many models require "conditional" inputs like manual "turn left" commands during training, which are expensive and difficult to scale.

MoNet aims to bridge this gap by creating a system that discovers these high-level tasks on its own and makes them accessible to human engineers.

Methods

The core of MoNet is its Latent Functional Modularity. Instead of one giant "black box," the network is split into three distinct, functionally separated modules:

  1. Perception Module: Uses a hybrid Vision Transformer (ViT) and CNN architecture to extract fused features ($z^p$) from camera images and topology maps.
  2. Planning Module: Processes these features to generate a Latent Decision vector ($h^d$).
  3. Control Module: Takes both the perceptual features and the latent decision to produce the final steering and throttle commands ($a^c$).
MoNet's modular architecture separates perception, planning, and control, utilizing a Latent-Guided Contrastive Loss for self-supervised training.

To train the Planning module without manual labels, the authors developed a Latent-Guided Contrastive (LGC) Loss. This self-supervised approach uses perceptual similarity to guide decision-making: if two scenes look similar, the network is encouraged to make similar latent decisions; if they look different, the decisions must be distinct.

Finally, to make the system interpretable, they use a post-hoc approach. A multiclass SVM classifier decodes the "hidden" latent decision $h^d$ into human-understandable probabilities for tasks like "straight," "left-turn," or "collision avoidance."

Datasets

The researchers collected a custom dataset using a 1/10 scale RC car platform (F1TENTH style) equipped with a Jetson Xavier NX.

  • Data: 2 hours of human demonstrations (88,326 sensory-motor pairs).
  • Inputs: Front-facing camera images and ego-centric coarse topology maps.
  • Environments: Two indoor corridor environments with various intersections and obstacles.

Key Insights and Ideas

  • Self-Supervised Task Discovery: The LGC loss allows the network to "discover" high-level navigational tasks. It learns that an intersection requires a different type of decision than a straight corridor simply by observing the visual context, without ever being explicitly told what a "turn" is.
  • Preventing Feature Collapse: Without the contrastive loss, modular networks often suffer from "collapse," where the planning module fails to produce distinct features for different tasks. MoNet's loss ensures the latent space remains well-structured.
  • Interpretability without Performance Loss: By using a post-hoc interpretation layer (the SVM), the researchers can extract explanations without modifying the core control logic. This ensures that making the model "talk" doesn't make it a worse driver.
The post-hoc interpretation pipeline decodes latent vectors into human-readable probabilities for specific driving tasks.
  • Uncertainty as a Safety Signal: By calculating the entropy of the decoded decisions, the system provides a built-in confidence metric. Entropy spikes at intersections or near obstacles, signaling when the network is transitioning between different behavioral states.

Key Results

MoNet was evaluated against a standard Vision Transformer baseline (ViTNet) and several ablation models.

  • Task Specificity: In a Representational Similarity Matrix (RSM) analysis, MoNet outperformed baselines by 7% to 28%, showing much clearer differentiation between tasks in its latent space.
  • Navigation Success: MoNet achieved a 100% success rate in navigating intersections, whereas the baseline ViTNet struggled significantly with right turns (63% success).
  • Qualitative Transparency: The system successfully generated saliency maps showing it focuses on corridor boundaries during turns and obstacles during avoidance maneuvers.

Limitations and Future Work

While MoNet excels in indoor environments, the authors acknowledge several areas for growth:

  • Dynamic Environments: The current model was tested in relatively static indoor scenes. Future work needs to address dynamic outdoor scenarios with moving traffic and pedestrians.
  • Temporal Modeling: MoNet currently processes single frames. Incorporating temporal layers (like LSTMs or spatio-temporal Transformers) would help the network understand the "flow" of a scene and make more consistent decisions over time.
  • Complex Maneuvers: The task set is currently limited to basic navigation. Expanding this to more complex behaviors like lane changing or merging is a logical next step.

Summary updated

Abstract

We introduce MoNet, a novel functionally modular network for self-supervised and interpretable end-to-end learning. By leveraging its functional modularity with a latent-guided contrastive loss function, MoNet efficiently learns task-specific decision-making processes in latent space without requiring task-level supervision. Moreover, our method incorporates an online, post-hoc explainability approach that enhances the interpretability of end-to-end inferences without compromising sensorimotor control performance. In real-world indoor environments, MoNet demonstrates effective visual autonomous navigation, outperforming baseline models by 7% to 28% in task specificity analysis. We further explore the interpretability of our network through post-hoc analysis of perceptual saliency maps and latent decision vectors. This provides valuable insights into the incorporation of explainable artificial intelligence into robotic learning, encompassing both perceptual and behavioral perspectives. Supplementary materials are available at https://sites.google.com/view/monet-lgc.
BibTeX
@inproceedings{seong2024self,
  author = {Hyunki Seong and Hyunchul Shim},
  title = {{Self-Supervised Interpretable End-to-End Learning via Latent Functional Modularity}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=dFEeI51O5j},
}
Figure