Introduction
In the development of intelligent systems, such as autonomous vehicles, the ability to simultaneously perceive scene semantics and spatial geometry is paramount. This typically involves two core computer vision tasks: scene segmentation and depth estimation. While Multi-Task Learning (MTL) offers a computationally efficient path by using a shared backbone to perform both tasks, it often suffers from a "performance tax." In many cases, a single MTL model performs worse than two separate, task-specific models due to conflicting gradients and the difficulty of balancing disparate objectives.
Knowledge Distillation (KD) has emerged as a popular solution to bridge this gap, where a multi-task "student" model learns from specialized "teacher" models. However, existing distillation methods often struggle with two primary issues: static weighting and knowledge forgetting. JointDistill is proposed as a novel framework to overcome these limitations through adaptive guidance and trajectory-based memory.
The JointDistill Framework
JointDistill introduces a dynamic approach to distillation, moving away from rigid, pre-defined loss weights toward a more responsive learning environment.
Self-Adaptive Knowledge Adjustment
Traditional distillation methods assign fixed weights to teachers, assuming their relative importance remains constant throughout training. In practice, a student model's proficiency in different tasks evolves at different rates. JointDistill implements a feedback loop that periodically evaluates the student on a validation dataset.
By using validation scores as a "report card," the framework creates a customized curriculum for the student. If the student demonstrates mastery in one task but struggles with another, the framework automatically adjusts the teacher's influence to provide more advanced knowledge or remedial focus where necessary.
Knowledge Trajectory Distillation (KTD)
A significant challenge in multi-task distillation is "forgetting"—as the student adjusts to satisfy one teacher, it may lose the nuances learned from another. To mitigate this without the massive memory overhead of storing previous model versions, JointDistill utilizes "Knowledge Trajectories."
The framework converts complex teacher features into 2D attention maps. By identifying and tracking the "Top-K" essential points—the pixels the teacher prioritizes most—the system creates a lightweight historical record of the teacher's focus. The student is then incentivized to follow this established "path" of attention, ensuring stable learning and preventing the erosion of previously acquired knowledge without the heavy storage requirements of saving full model checkpoints.
Experimental Evaluation
The researchers validated JointDistill across diverse environments, including urban street scenes (Cityscapes), indoor settings (NYU-v2), and large-scale facial attribute classification (CelebA).
Performance Gains
On the Cityscapes benchmark, JointDistill achieved a 9.04% average task improvement over standard MTL baselines, outperforming sophisticated competitors such as AMTL and CAGrad. On the NYU-v2 dataset, the framework showed a 2.83% improvement, demonstrating its versatility across different visual domains.
Efficiency and Scalability
Beyond accuracy, JointDistill proved to be more resource-efficient than existing methods designed to prevent forgetting. Compared to EATA, JointDistill reduced memory consumption (9.33 GB vs. 10.85 GB) and accelerated training time (16.87h vs. 19.55h). Furthermore, testing on the CelebA dataset—which involves 40 distinct tasks—confirmed that the framework scales effectively to high-dimensional multi-task problems without losing stability.
Conclusion and Future Work
JointDistill represents a shift in multi-task distillation from static instruction to adaptive mentorship. By treating the distillation process as a dynamic conversation and using attention trajectories as a compact form of memory, the framework allows unified models to approach or even exceed the performance of task-specific experts.
Future research directions include expanding the framework to incorporate multi-modal data (such as integrating CLIP-based text prompts) and exploring temporal trajectories for video-based tasks. Additionally, refining the adaptive weighting mechanism for scenarios with extremely limited validation data remains a key area for improvement in real-world edge deployments.