Model Quantization for ARM Cortex-M: INT8 vs FP16 vs Mixed Precision

The number that kills most embedded ML projects isn't the model size. It's the tensor arena. A 180KB int8 model will fit happily in flash on almost any modern MCU, but if the second convolution layer needs a 96KB input buffer and a 96KB output buffer live at the same time, you're out of RAM on a 256KB part before you've done anything interesting. Quantization is how you fix both problems at once — weights get smaller, and so do the intermediate activations that actually eat your SRAM.

Below is how we approach precision selection when we're targeting Cortex-M class hardware: what INT8 buys you, why FP16 is usually a disappointment on these cores, and where mixed precision genuinely earns the extra engineering effort. We'll also be blunt about the cases where the right answer is to stop quantizing and change something else.

Start with the memory budget, not the model

Before choosing a precision, write down three numbers. Flash available for weights. RAM available for the tensor arena. RAM available for everything else — your RTOS, stack, DMA buffers, comms stack, sensor rings. On a 256KB part it's common to have only 120-160KB left for inference once the rest of the firmware is accounted for.

Then estimate the arena. For a feed-forward network with a good memory planner, peak arena is roughly the largest sum of tensors that must coexist at any single operator — usually one layer's input plus its output, plus any residual branch being held for later. Frameworks like TensorFlow Lite for Microcontrollers report this number for you after allocation, and it is the number you should optimise against.

This is why input resolution and early-layer stride matter more than parameter count on tiny targets. A 96x96x1 input with a stride-2 first convolution and 16 channels gives you a 48x48x16 tensor — 36KB in int8, 144KB in float32. Same weights, wildly different feasibility.

What your core can actually execute

Precision choices only pay off if the silicon has instructions for them. Cortex-M is not one thing, and the differences matter:

• Cortex-M0/M0+/M23: no DSP extension, no FPU. Everything is scalar integer. INT8 is essentially mandatory, and even then expect modest throughput. Keep models tiny and simple.
• Cortex-M4/M7 with DSP extension: dual 16-bit multiply-accumulate instructions (SMLAD and friends) let INT8 kernels unpack to 16-bit and issue two MACs per instruction. This is where CMSIS-NN gets most of its speedup over naive float code.
• Cortex-M33: FPU is single-precision; the DSP extension is optional, so check your specific part. Plenty of M33 designs ship without it.
• Cortex-M55/M85 with Helium (MVE): 128-bit vector unit with real INT8 and INT16 vector MACs, and — importantly — native FP16 arithmetic. This is the one core family where FP16 is a genuine performance option rather than a storage trick.
• Ethos-U55/U65 NPUs paired with an M55/M85: integer only. INT8, and INT16 for some operators. Float layers fall back to the CPU, which usually erases the accelerator's benefit if it happens mid-graph.

The practical consequence: for the vast majority of Cortex-M deployments, INT8 is not a compromise you accept reluctantly. It's the only precision the hardware is genuinely fast at.

INT8: the default, and how to do it properly

Full integer post-training quantization maps float tensors to 8-bit integers with a scale and a zero point. In the TFLite scheme, weights are symmetric (zero point 0) and per-output-channel, while activations are asymmetric and per-tensor. That asymmetry in the weights layout is not cosmetic — per-channel scales are the single biggest reason modern INT8 PTQ works as well as it does.

Per-tensor vs per-channel

A depthwise convolution can easily have one channel with a weight range of ±0.02 and another at ±2.0. Force them to share a single scale and the small channel collapses to two or three distinct integer values. Per-channel scaling gives each filter its own step size and typically recovers most of the accuracy loss on MobileNet-style architectures. If your toolchain only supports per-tensor weights, that's a real limitation, not a detail.

Calibration

Activation ranges come from running representative data through the float model. A few practical rules we stick to:

• Use real sensor data captured on the target device, not synthetic or augmented samples. Range statistics from clean lab data will underestimate what the field produces.
• A couple of hundred samples is usually enough for the ranges to stabilise; going from 200 to 2000 rarely changes much.
• Cover the edges of your operating envelope — the loud recording, the dark frame, the saturated accelerometer axis. Those are exactly the inputs that clip if calibration missed them.
• Prefer percentile-based range selection over absolute min/max if your toolchain offers it. One outlier activation can stretch a scale factor and blunt everything else.

Where INT8 PTQ tends to break

For well-behaved convolutional classifiers with batch-norm folded into the convolutions and ReLU-family activations, straight post-training INT8 often lands within about a percentage point of the float baseline. That's the typical case, not a guarantee. The situations that reliably cause trouble:

• Depthwise-separable blocks with very uneven per-channel ranges, especially when combined with unbounded activations.
• Recurrent layers, where quantization error accumulates across timesteps.
• Attention blocks and softmax over long sequences — the dynamic range inside them is brutal for 8 bits.
• Regression heads. A classifier only needs the argmax to survive; a model predicting a continuous value in engineering units will show the quantization noise directly in its output.
• Anything with a residual add where the two branches have very different scales.

When PTQ drops more than you can accept, quantization-aware training is the next step: simulate the quantizers during fine-tuning so the weights learn to live with them. A short fine-tune at a low learning rate usually recovers most of the gap. It costs training infrastructure and time, which is exactly the trade-off to weigh before promising anyone a number.

FP16: usually storage, rarely speed

FP16 is attractive on paper. Half the memory of float32, no scale factors, no calibration, no zero points, and a dynamic range that handles awkward layers without complaint. On a GPU or an M55 with Helium, it's a real option.

On a classic Cortex-M4 or M7, it mostly isn't. Those FPUs are single-precision. Many of them can convert between FP16 and FP32 in hardware, but the arithmetic happens in FP32 — so an FP16 model halves your flash footprint for weights and then spends cycles converting on the way in. Compared to INT8 with CMSIS-NN kernels exploiting SIMD MACs, you're typically looking at several times the latency for the same graph, and your activations may still be materialised in FP32 in the arena unless the runtime is careful. That is the worst of both worlds on a 256KB part.

So we treat FP16 as useful in two situations: you're on Helium-capable hardware where FP16 vector maths is native, or you have a small number of numerically fragile layers and enough headroom that keeping them in float is cheaper than fighting quantization. Otherwise, skip it.

Mixed precision that actually earns its complexity

Mixed precision on Cortex-M is not a dial you turn. It's a set of specific, targeted interventions. The ones we've found worth the effort:

• INT16 activations with INT8 weights. TFLite's 16x8 mode keeps weight storage at 8 bits but gives activations 16-bit headroom. It typically recovers most of the accuracy loss on sensitive models. Expect roughly 1.5-2x the latency of pure INT8 on a DSP-extension core, and double the arena for activations — so budget for it before you commit.
• Higher precision on the first and last layers. The input layer sees raw sensor dynamic range; the output layer determines your final decision boundary. Keeping just those two in INT16 or float is often enough, and the cost is small because they're usually cheap layers.
• Float fallback for a single awkward operator. Softmax, layer norm, or a custom op can stay in float while the convolution stack stays integer. Just watch the requantization boundaries — each transition costs cycles and, worse, forces a float-sized tensor into your arena.
• Keeping the model integer end-to-end when an NPU is in play. If you're targeting Ethos-U, one float operator in the middle of the graph splits execution and can cost you more than the accuracy you gained.

The pattern is consistent: spend bits where the numerics are hard, not uniformly. Profile per-layer error before you decide which layers those are, rather than guessing.

The workflow we use

1. Establish a float32 baseline on the target task with a held-out test set you trust, and record per-class metrics, not just overall accuracy.
2. Fold batch norm into the preceding convolution and remove training-only ops. Do this before quantizing; unfolded BN is a common source of mystery accuracy loss.
3. Run full-integer INT8 PTQ with per-channel weights and a representative dataset from real device captures.
4. Compare layer-by-layer activation statistics between float and quantized runs. Cosine similarity or signal-to-quantization-noise ratio per layer points straight at the problem layers.
5. Evaluate the quantized model on the full test set on the host, then re-verify on hardware. Host and device results should match bit-for-bit for integer graphs; if they don't, you have a preprocessing mismatch, and that is worth chasing before anything else.
6. Measure arena size and per-inference latency on the actual board with the actual clock and cache configuration, not an estimate.
7. If accuracy is short, apply targeted mixed precision or quantization-aware training — in that order, because mixed precision is cheaper to try.
8. Re-check the memory budget after every change. INT16 activations have a habit of quietly doubling the arena.

That loop is most of what our AI and ML engineering work looks like on embedded targets: not one heroic optimisation, but a disciplined cycle of measure, adjust, re-measure on hardware.

When quantization is the wrong tool

We'd rather say this early than after you've spent a month on it. Aggressive quantization is not always the answer:

• If you're more than about 2-3x over your memory budget, quantization won't close the gap. Change the architecture — reduce input resolution, add an early stride, cut channel widths, or use a smaller backbone. Architecture search at the right scale beats squeezing bits out of a model that was never going to fit.
• If the task is genuinely simple, a hand-designed feature pipeline plus a small classifier — spectral features and a gradient-boosted tree, or a threshold cascade — can outperform a quantized neural network in accuracy, latency and explainability. We've seen this hold for a lot of vibration and audio-event problems.
• If the BOM can absorb it, moving from a 256KB part to a 1MB part with a DSP extension or Helium often costs less than the engineering time to make the smaller part work. Compare the two honestly before deciding.
• If the model must be updated frequently with new classes, an integer graph with baked-in scale factors is more work to maintain. Plan the retraining and requantization pipeline as part of the product, not as an afterthought.

Reporting results you can defend

Two things we insist on before anyone signs off. First, quote accuracy on the quantized model running on hardware, with the same preprocessing the firmware uses — not the float model's numbers with a footnote. Preprocessing drift between Python and C is the single most common cause of "the model works in the lab and not on the device". Second, report latency and arena from the target board at production clock settings, including whether caches are enabled and whether weights execute from internal flash or external QSPI. Those choices can move inference time by a factor of several.

If you want to see the kinds of systems we build around these constraints, our recent project work gives a reasonable picture. And when a programme needs sustained embedded ML capacity rather than a one-off engagement, we also staff dedicated engineering teams who work inside your process and toolchain.

If you have a model that doesn't fit yet, send us the architecture, the target part number, and your accuracy floor. Those three things are usually enough for us to tell you whether it's an INT8 problem, a mixed-precision problem, or an architecture problem — and we'll say so plainly. Get in touch and we'll take a look.