What Is Model Distillation? How It Differs from Quantization and Pruning
Model distillation trains a smaller student network to imitate a larger teacher, which makes it the only common compression technique that changes the architecture and requires a…