Model distillation trains a smaller “student” model to reproduce the behaviour of a larger “teacher” model. It is one of four common model-compression techniques — alongside pruning, quantization, and low-rank factorization — and it is the only one that changes the model’s architecture and requires a fresh training run. Quantization stores the same weights in fewer bits. Pruning deletes weights, channels, heads, or layers from an existing network.
That difference decides the practical question. If you need a smaller download or a lower memory footprint and you want to keep the model you already validated, quantization is usually the first move. If you need a fundamentally cheaper architecture — fewer parameters, fewer FLOPs, a different shape — distillation is the tool, and it costs you data and training compute to get there. This article is written for practitioners deploying models locally or on constrained servers who need to pick a technique rather than recite a definition.
The four techniques side by side
The table below summarises what each technique changes and what its speedup actually depends on. Typical size reductions are ranges, not guarantees: the realised gain depends heavily on the architecture, the target hardware, and how aggressively you compress.
| Technique | What it changes | Typical size reduction | Speedup depends on |
|---|---|---|---|
| Pruning | Removes weights, channels, heads, or layers | 1.5–10x | Structured vs. unstructured; sparse-kernel support |
| Knowledge distillation | Trains a smaller architecture to imitate a teacher | 2–10x | Size of the student |
| Quantization | Bits per weight and/or activation (16 to 8 to 4) | 2–8x | Whether the hardware has low-precision math units |
| Low-rank factorization | Factors a matrix into two thin matrices | 1.5–4x per layer | Proportional to the FLOP reduction |
These figures come from a compression overview that also stresses the key structural point: each technique removes a different kind of redundancy, so they stack rather than compete (Andrews Notebook, Model Compression).
What distillation actually does
Distillation does not copy weights. The student has its own architecture — fewer layers, narrower hidden dimensions, or a different block design entirely — and it is trained to match the teacher’s outputs rather than only the ground-truth labels.
The standard mechanism uses the teacher’s full output distribution, often called soft targets or logits, rather than a single hard label. A teacher that assigns 0.72 to one class and 0.21 to a similar one encodes information about which classes are confusable that a one-hot label does not. A temperature parameter is commonly applied to soften the distributions during training, and the student’s loss is a weighted combination of matching the teacher’s soft outputs and matching the true labels. The same idea works for regression-style outputs and for sequence models, where the student may be trained to match the teacher’s token probabilities over a corpus.
Three consequences follow, and they are what separate distillation from the other techniques:
- It needs a training run. You need the teacher, a representative dataset, and compute for the student’s training. Quantization and most pruning workflows are post-training operations by comparison.
- It is task- and data-bound. The student learns the teacher’s behaviour on the distribution you train it on. A student distilled on English customer-support transcripts is not automatically a good general-purpose model.
- The student’s ceiling is the teacher. Distillation transfers capability; it does not create it. If the teacher has a systematic bias or blind spot, the student can inherit it, so an evaluation set that is independent of the distillation data matters.
Distillation is also the standard way to collapse an ensemble into a single deployable model, because the teacher does not have to be one network.
Quantization: the same numbers, fewer bits
Quantization keeps the network’s structure intact and reduces the precision used to store and compute with weights and activations — for example from 16-bit to 8-bit or 4-bit. Because the architecture is unchanged, no retraining is strictly required for post-training quantization; a calibration pass over representative data is typically used to choose scaling factors. Quantization-aware training, where the model is fine-tuned with simulated low precision, can recover accuracy when post-training quantization degrades too much.
The size reduction is roughly proportional to the bit width for the weights, but total memory use is not the same as weight size. Activations, the key-value cache for transformer models, and runtime overhead all consume memory and are not reduced by weight-only quantization. Similarly, the speedup is not automatic: it depends on whether the target hardware has low-precision math units that can actually execute the narrower format faster. A quantized model on hardware without those units can end up the same speed or slower after conversion overhead.
Pruning: removing parts of the network
Pruning removes parameters or whole structural blocks. The distinction that matters most in practice is structured versus unstructured:
- Unstructured pruning sets individual weights to zero. It reduces storage if you use a sparse format, but on ordinary dense hardware it often produces no latency improvement because the remaining weights still have to be read and multiplied.
- Structured pruning removes entire channels, attention heads, or layers. The resulting model is genuinely smaller and denser, so it speeds up on standard runtimes — at the cost of coarser control over which parameters disappear.
Aggressive pruning usually requires fine-tuning to recover accuracy, which brings back some of the training cost that distillation carries from the start. The 1.5–10x range in the table above is wide precisely because the outcome depends on which kind of pruning you do and whether the runtime supports sparse kernels.
Where low-rank factorization fits
Low-rank factorization replaces a large weight matrix with two thin matrices whose product approximates it. It changes the model’s structure like pruning does, but in a smooth, algebraically defined way rather than by deleting units. Typical reductions are 1.5–4x per layer, with speedup roughly proportional to the reduction in FLOPs. It is often used as a first pass before quantization.
A concrete example: compute-bound versus memory-bound
The choice between these techniques depends on which resource is actually the bottleneck, and the two failure modes look very different.
Consider a fraud-detection MLP described in a model-production course: hidden layers of 512, 512, and 256 units, 402,433 parameters, 15 input features. Offline it beats a logistic regression on the same features — AUC 0.896 versus 0.869 — and at a 0.2 threshold it blocks 1.55% of payments while catching 28.6% of fraud, where the logistic regression blocks 1.57% and catches 22.7%. The test set is 40,000 synthetic payments with 2.4% fraud, trained on 50,000 with 2.3%.
Latency is not the problem here: the network scores one payment in about 0.07 ms, comfortably inside a 50 ms budget. Cost is the problem. A batch of 32 payments takes 0.23 to 0.31 ms, where the logistic regression takes 0.01 ms — roughly 20 to 30 times the compute per payment, and the platform team pays for every core the service runs on (DataInterview, Quantization, Pruning and Distillation in ML).
By contrast, the same lesson describes an LLM feature that drafts summaries of payments sent to manual review, where the problem is memory rather than CPU: the model’s weights determine how many GPUs it needs. Those are two different compression problems. The fraud model is compute-bound, so structured pruning or distillation to a genuinely narrower network attacks the cost directly. The summarisation model is memory-bound, so quantization and distillation to a smaller architecture both reduce the number of GPUs required.
This is an illustrative case drawn from published course material, not a benchmark run by this publication. Treat the numbers as an example of the reasoning, and measure your own workload before committing.
How the techniques stack
Because each technique targets a different kind of redundancy, a typical pipeline applies them in sequence: shrink the architecture first with pruning, distillation, or factorization; then lower the precision with quantization; then compile for the target runtime. The ordering matters, because quantizing a model you are about to structurally change means doing the calibration work twice.
Do not assume the gains multiply. A 4x distillation reduction combined with 4x quantization does not reliably produce a 16x end-to-end improvement, because memory, latency, and accuracy each respond to different parts of the pipeline and interact with runtime overhead.
Choosing for a local or self-hosted deployment
A short decision guide, assuming you already know your bottleneck:
- Model does not fit in memory, quality is acceptable. Start with quantization. It is the cheapest intervention and requires no training data.
- Latency on CPU-bound inference is the limit. Structured pruning or distillation, because dense smaller models speed up on hardware without sparse-kernel support.
- You need a much smaller model for edge or mobile. Distillation, since the target size is usually below what quantization alone can reach from a large teacher.
- You must preserve exact behaviour on a narrow task. Quantize with careful calibration and validate on your own task-specific set before and after.
- You have a large teacher and limited deployment hardware but training capacity available. Distillation is the technique that changes the architecture rather than squeezing the existing one.
Limitations worth planning for
Distillation’s costs are front-loaded: teacher access, a representative dataset, and a training run. Its risks are inherited bias and a student that performs well on the distillation distribution but degrades elsewhere. Pruning’s risk is that unstructured sparsity delivers storage savings without latency savings on your runtime. Quantization’s risk is that accuracy loss concentrates in a small number of outlier-heavy layers, so aggregate metrics can look fine while a specific class collapses. Combined pipelines need end-to-end measurement, not per-stage assumptions.
What to record before and after
Whichever technique you apply, capture the same set of numbers so the comparison is meaningful: your task metric (AUC, F1, exact-match, or whatever the task uses), p50 and p95 latency, peak memory, throughput, and on-disk size. Record the hardware, the runtime and its version, and the precision format used. Without those, a compression result is not reproducible and not comparable to anyone else’s.
If you are working through this on a local setup, share your model, hardware, chosen technique, and the before-and-after numbers in the community — a concrete compression result on a specific runtime is more useful to other practitioners than a general recommendation.
FAQ
Can I distill a model without retraining?
No. Distillation is a training procedure: the student has a different architecture and must be trained against the teacher's outputs using a representative dataset. Quantization is the technique that usually works without a full training run.
Does pruning always make inference faster?
Not necessarily. Unstructured pruning sets individual weights to zero and may not speed up inference on ordinary dense hardware unless the runtime supports sparse kernels. Structured pruning removes whole channels, heads, or layers and produces a denser, smaller model that speeds up more reliably.
Should I quantize before or after distillation?
Shrink the architecture first — with distillation, pruning, or factorization — and lower the precision afterwards. Quantizing a model you are about to structurally change means repeating the calibration and validation work.