Running TensorFlow Lite on ARM Cortex-M: A Step-by-Step Guide

A Cortex-M4 running at 80 MHz with 256 KB of flash and 64 KB of RAM can classify a keyword, detect an anomaly in vibration data, or spot a person in a low-resolution image frame. It cannot run a ResNet. The gap between those two statements is where most edge AI projects succeed or quietly die, and it is almost entirely a matter of decisions made before you flash anything.

This guide walks the full path: choosing whether a microcontroller is the right target at all, shaping and quantizing the model, generating the C array, wiring up the interpreter, and then getting the inference time down to something your control loop can live with. We have shipped this pattern on both Cortex-M4F and Cortex-M7 parts, and the failure modes are consistent enough to be worth writing down.

Before you write any code: is Cortex-M actually the right target?

TensorFlow Lite for Microcontrollers (TFLM, now shipping under the LiteRT umbrella) exists for a narrow but real set of constraints. It uses no dynamic memory allocation, needs no operating system, and the core runtime can fit in tens of kilobytes of flash. That is remarkable engineering. It is also a straitjacket.

Be honest about your requirement before you commit:

• If your device already has Wi-Fi and the latency budget tolerates a round trip, server-side inference is cheaper to build, easier to update, and lets you use any model you like. Don't put a model on-device just because you can.
• If you need vision at anything beyond tiny grayscale resolution, or you need a transformer, look at a Cortex-A class SoC or a dedicated edge accelerator instead. You will spend less total engineering time.
• If your problem is genuinely a threshold, an FFT plus a couple of hand-tuned rules, or a small decision tree, ship that. A 4 KB classical DSP pipeline beats a 40 KB neural network that nobody on the team can debug.
• Cortex-M is the right answer when you need sub-100 ms local response, battery life measured in months, no connectivity guarantee, or data that must never leave the device.

Assuming you're still here, the next constraint is silicon. A Cortex-M0+ can run TFLM but has no DSP extensions, so expect an order of magnitude worse throughput than an M4F on the same clock. Cortex-M4 and M7 give you SIMD MAC instructions through the DSP extension. Cortex-M55 and M85 add Helium (MVE), which is a step change for int8 math. If your board has an Ethos-U55 or U65 NPU alongside the core, that changes the whole plan — see the optimization section.

Step 1: Shape the model for the target, not the dataset

The most common mistake we see is a model trained to accuracy in Keras and then handed over the wall to embedded. By then the architecture has already decided your memory budget, and no amount of conversion tricks will save it.

Work backwards from three numbers you can compute before training:

1. Flash for weights. After full int8 quantization, roughly one byte per parameter plus flatbuffer overhead. A 50,000-parameter model is about 50-60 KB of flash. That is your hard ceiling.
2. RAM for the tensor arena. This is peak simultaneous activation memory, not total. For a small CNN it is usually dominated by the largest one or two intermediate feature maps. Compute the largest layer output in bytes and assume the arena needs roughly two to three times that, plus a few kilobytes for interpreter bookkeeping.
3. Multiply-accumulate operations per inference. Divide by a realistic MACs-per-cycle figure for your core to get a rough cycle count. On a Cortex-M4F with CMSIS-NN you can plan on a small number of MACs per cycle for int8 convolutions; treat any estimate as a sanity check, not a spec.

Design choices that reliably pay off: depthwise separable convolutions instead of full ones, striding early to shrink feature maps before they get expensive, avoiding large fully-connected layers at the head, and keeping the input as small as the problem tolerates. For audio, do the feature extraction (MFCC or log-mel) in fixed-point C on the device and feed the model a small spectrogram rather than raw samples.

Step 2: Convert and quantize properly

TFLM only executes integer-friendly graphs efficiently, and many kernels only exist in int8. Post-training full integer quantization is the default path. In the Python converter you set the optimization flag, supply a representative dataset generator, and force both the supported ops and the input/output types to int8.

The representative dataset matters more than people expect. It is how the converter learns activation ranges. Feed it a few hundred samples that genuinely reflect deployment conditions — same microphone, same sensor placement, same lighting. A representative set drawn only from clean lab data will produce clipping ranges that fall apart in the field.

After conversion, validate on device-equivalent data before you touch firmware. Run the quantized .tflite in the Python interpreter over your full test set and compare against the float model. A one to two point accuracy drop is normal. A ten point drop means something is wrong with your ranges, usually a layer with a long activation tail. If quantization-aware training is available for your architecture, it will usually recover most of that gap.

Also check which operators survived conversion. Run a graph inspection and list every op. Then confirm each one has a TFLM kernel. Ops that commonly cause trouble include certain resize variants, some LSTM formulations, and anything the converter has wrapped in a Flex delegate — Flex does not exist on microcontrollers, so a Flex op means you must change the model.

Step 3: Get the model into the firmware build

There is no filesystem, so the model becomes a byte array in flash. The classic route is xxd -i on the .tflite file to produce a C array, then wrap it in a header. Two details that cause hours of confusion:

• Align the array. FlatBuffers require alignment; declare the array with alignas(16) or place it in a section with guaranteed alignment. Misaligned model data produces bizarre failures on parse rather than a clean error.
• Mark it const so the linker puts it in flash. If it lands in RAM you have just spent your entire activation budget on weights.

On the build side, you need the TFLM sources plus CMSIS-NN. Vendor SDKs from ST, NXP, Nordic and Espressif ship pre-integrated versions, and using theirs is usually the fastest route to a working baseline. Build with -O2 or -Os, enable hardware FPU flags if your part has one, and define the CMSIS-NN path so you get optimized kernels rather than reference C.

Step 4: The inference loop

The structure is always the same. Declare a static, aligned tensor arena as a plain uint8_t array. Wrap the model bytes with GetModel and check the schema version. Build an op resolver — and use MicroMutableOpResolver with only the ops your graph actually needs, never AllStopsResolver, because the all-ops resolver drags every kernel into flash. Construct the interpreter, call AllocateTensors, and check the status.

Then per inference: get the input tensor pointer, write your quantized int8 features into it (remembering to apply the input scale and zero point from the tensor quantization params), call Invoke, read the output tensor, and dequantize if you need real-world units.

For arena sizing, start deliberately too large — say 60 KB — get AllocateTensors to succeed, then use the recording interpreter or arena_used_bytes() to read the actual high-water mark. Set your arena to that plus a small margin and reclaim the rest. Do not guess. Do not leave it oversized either; on a 64 KB part that margin is your stack.

Step 5: Optimization, in the order that actually helps

Measure first. Use the DWT cycle counter to time Invoke, and time your preprocessing separately — on audio and vibration pipelines, feature extraction is frequently slower than the network itself, and people spend days optimizing the wrong half.

• Confirm CMSIS-NN is really linked. The single biggest speedup available is switching from reference kernels to CMSIS-NN, and it is easy to believe you have done it when a build flag is missing. Compare a timed Invoke before and after.
• Keep the model and arena in the fastest memory available. On Cortex-M7 parts, placing the arena in DTCM and enabling I-cache for flash-resident weights can move inference time substantially.
• Exploit Helium if you have it. On Cortex-M55/M85, MVE-enabled CMSIS-NN kernels deliver large gains on int8 convolutions with no model changes.
• Offload to an NPU if one exists. For Ethos-U, you compile the quantized .tflite through the Vela tool, which rewrites supported subgraphs into NPU commands and leaves the rest on the CPU. Check Vela's report — ops it cannot take stay on the Cortex-M, and a graph that falls back heavily will disappoint you.
• Only then revisit the architecture. Pruning, fewer channels, or a smaller input often buys more than any runtime tuning, at a known accuracy cost.

Failure modes we see repeatedly

• AllocateTensors fails silently because the return status is never checked, and Invoke then reads garbage.
• Stack overflow rather than arena exhaustion. TFLM avoids heap use but kernels still use stack; audit your task stack sizes.
• Input scaling skipped. Feeding raw int8 sensor values without applying the tensor's scale and zero point gives a model that runs perfectly and predicts nonsense.
• Training and deployment preprocessing drift apart. Your Python MFCC and your C MFCC must match bit-for-bit in behaviour, or accuracy on device will be worse than your test set for no visible reason.
• No plan for model updates. Decide early whether models ship as part of the firmware image or in a separate OTA-updatable flash region. Retrofitting the latter is painful.

None of this is exotic work, but it rewards people who have burned themselves on it before. If you are weighing a microcontroller deployment against a gateway or cloud approach, or you have a model that works in the notebook and needs to survive a 64 KB RAM budget, we are happy to look at it with you. You can see the kind of engineering we take on in our work, read more about how we approach AI and ML development, or just get in touch with your constraints and we will tell you honestly whether the model you have can fit the part you have chosen.