TL;DR
LIT (Learned Intermediate representation Training) is a novel model compression technique that slims down deep neural networks by training a shallow "student" model to mimic the internal states of a deep "teacher." By feeding the teacher’s internal representations directly into the student during training, LIT achieves up to 5.5× compression with zero loss in accuracy, outperforming standard knowledge distillation and pruning.
The Challenge of Model Compression
Modern deep neural networks (DNNs) like ResNet have achieved incredible accuracy, but they come with a massive computational cost. To deploy these models on edge devices or in real-time applications, researchers rely on model compression.
Existing methods generally fall into two camps, both with notable drawbacks:
- Deep Compression (Pruning/Quantization): This involves removing weights or reducing their precision. However, it often requires specialized hardware to realize actual speedups and struggles with modern, "sparse" architectures like ResNet.
- Knowledge Distillation (KD): This trains a smaller student model to match the final outputs (logits) of a large teacher. Unfortunately, KD often fails to capture the complex internal logic of the teacher, leading to a noticeable performance gap.
The authors identify a specific gap in the field: how do we effectively reduce the depth of a network (the number of layers) without losing the "meaning" of the intermediate data as it flows through the model?
Methods: The LIT Approach
LIT introduces a student/teacher framework specifically designed for the repetitive, block-based structure of modern DNNs. It relies on two core innovations:
- Direct Intermediate Representation (IR) Comparison: Instead of just looking at the final answer, LIT penalizes the student if its internal "feature maps" (the data between layers) deviate from the teacher’s.
- Teacher-Injected Training: This is the breakthrough moment. In standard training, a student's layer $N$ receives input from its own layer $N-1$. In LIT, during training, the student’s block $N$ receives the teacher’s intermediate output from block $N-1$ as its input.
By "forcing" the teacher's high-quality data into the student's intermediate blocks, the student learns to transform that data exactly like the teacher does, effectively training each student block in isolation. After this phase, the model is fine-tuned to ensure the blocks work together seamlessly.
Key Insights
- Stability is Key: The main reason previous "hint-based" methods (like FitNets) struggled is that student IRs are "meaningless" at the start of training. By using the teacher's IRs as inputs, LIT provides a stable "ground truth" for the student to learn from immediately.
- Depth Reduction > Weight Pruning: On modern architectures, reducing depth (layers) provides more consistent speedups on standard hardware (CPUs/GPUs) than pruning individual weights, which often creates "holes" in matrices that hardware isn't optimized to skip.
- GAN Compression: Compressing Generative Adversarial Networks (GANs) is notoriously difficult. LIT not only compressed StarGAN by 1.8× but actually improved the quality of the generated images (lower FID scores) compared to the teacher.
Key Results
LIT was evaluated across several domains, including image classification (CIFAR-10/100), sentiment analysis (Amazon Reviews), and image-to-image translation (CelebA). It consistently outperformed state-of-the-art pruning and distillation methods:
- ResNet-110 to ResNet-56: Achieved 3.4× compression on CIFAR-10 with a 0.24% increase in accuracy, while traditional pruning methods often saw accuracy drops.
- ResNeXt: Achieved a massive 5.5× reduction in depth on CIFAR-10 without losing accuracy.
- Sentiment Analysis: Compressed the VDCNN model by 1.7× on Amazon Reviews.
- Comparison: LIT outperformed "Born Again" networks (where the student is the same size as the teacher) by up to 0.64% in accuracy.
Limitations and Future Work
While highly effective, LIT does have some constraints:
- Architecture Matching: LIT requires that the teacher and student have matching dimensions at "split points" (where the IRs are compared). While the authors suggest copying layers to fix mismatches, this adds a manual design step.
- Block-Based Assumption: The method is optimized for networks with repetitive structures (like ResNet blocks). It might require more tuning for highly heterogeneous architectures.
Future directions suggested by the authors include exploring different loss functions for the IR comparison (beyond standard L2 loss) and investigating how LIT could be used to automate the design of student architectures.
Summary updated