A client came to us with a vibration-monitoring board built around a Cortex-M4, 256 KB of RAM, and a model trained in PyTorch. Another was shipping an ARM64 Linux gateway with a Qualcomm SoC and needed sub-30 ms inference on a quantized vision model. Same question from both: TensorFlow Lite or ONNX Runtime? The honest answer was different in each case, and the reasoning had almost nothing to do with which framework is "better."
The two runtimes overlap enough to look interchangeable in a feature matrix and diverge enough that picking wrong costs you weeks. Below is how we actually reason through it — the hardware reach, the quantization semantics, the conversion friction, and the cases where you should skip both.
They are not the same category of thing
TensorFlow Lite — now officially rebranded as LiteRT by Google, though almost everyone still says TFLite and the .tflite file extension hasn't changed — is really two products sharing a name. There is TFLite proper, which runs on Android, iOS, Linux, and embedded Linux with an interpreter plus a delegate system. And there is TFLite Micro (LiteRT for Microcontrollers), a separate C++ library with no dynamic memory allocation, no filesystem dependency, and a footprint measured in tens of kilobytes. TFLite Micro is the one that runs on bare-metal Cortex-M parts and RISC-V microcontrollers.
ONNX Runtime is a single cross-platform inference engine with a pluggable execution provider architecture. It targets everything from server GPUs to mobile devices. There are minimal and reduced builds that strip unused operators and the ONNX format parsing, which brings the binary down considerably — but you still need a heap, a real C++ runtime, and typically an OS or RTOS. ONNX Runtime is not a bare-metal microcontroller story. If your target has under a megabyte of RAM, this decision is already made for you.
So the first fork is brutally simple: microcontroller-class hardware means TFLite Micro (or a vendor toolchain that consumes .tflite, like STM32Cube.AI or NXP eIQ, or a compiler path like microTVM). Application-processor-class hardware — anything running Linux, Android, or iOS — is where the real comparison lives.
Hardware reach: delegates versus execution providers
Both frameworks abstract accelerators, and both do it by handing subgraphs off to a backend that may or may not support every operator in your model. The names differ; the failure mode is identical. When a backend can't handle an op, the graph gets partitioned and execution bounces between the accelerator and CPU. Enough bounces and you're slower than pure CPU because of the memory transfers.
Where they differ is coverage:
- TFLite's strongest paths are XNNPACK for CPU (genuinely fast on ARM), the GPU delegate, Core ML on Apple, and the specialized NPU targets that were designed around the .tflite int8 spec — Coral Edge TPU, ARM Ethos-U via TFLite Micro, and many vendor DSP stacks.
- ONNX Runtime's strongest paths are TensorRT on NVIDIA (including Jetson), OpenVINO on Intel CPU/iGPU/NPU, DirectML on Windows, QNN for Qualcomm NPUs, and Vitis AI on AMD/Xilinx. On x86 and Jetson it is usually the better-supported option by a wide margin.
- Android is genuinely contested. NNAPI has been deprecated in favour of vendor-supplied paths, which weakens what used to be TFLite's biggest structural advantage. Both runtimes now lean on vendor SDKs there.
Our practical rule: pick the runtime the silicon vendor's own tooling and sample code is written against. Fighting a vendor's preferred path to save on framework familiarity is a bad trade. You will spend the savings on debugging quantization mismatches at 2 a.m.
Quantization is where the real divergence lives
This is the part that gets underestimated. Both frameworks support int8, both support float16, both support post-training and quantization-aware training. The semantics are not the same.
TFLite's opinionated integer spec
TFLite defines a tight full-integer quantization specification: per-axis symmetric int8 weights, per-tensor asymmetric activations, with quantization parameters baked into the tensor metadata. It's restrictive, and that restriction is the point. Fixed-function NPUs were designed against this spec. When a Coral or Ethos-U datasheet says "int8 quantized TFLite model," it means that spec exactly, and a model outside it will silently fall back to CPU or refuse to compile. TFLite also offers dynamic-range quantization, which shrinks weights to int8 while keeping activations in float — a very cheap win for size, less of a win for latency.
ONNX Runtime's QDQ flexibility
ONNX represents quantization by inserting explicit QuantizeLinear/DequantizeLinear pairs into the graph (the QDQ format), or with fused quantized operators (QOperator). QDQ is more expressive: mixed precision within a graph is natural, per-channel choices are yours, and downstream compilers like TensorRT and OpenVINO consume QDQ graphs and do their own fusion. That flexibility means you can express things TFLite can't. It also means two different execution providers can interpret the same QDQ graph with different fusion decisions, and your accuracy numbers can shift when you change backends.
In our experience the accuracy delta between a well-calibrated TFLite int8 model and a well-calibrated ONNX int8 model on the same architecture is small — typically a fraction of a percent on classification tasks, sometimes larger on detection where output scaling matters. The engineering cost of getting there is where they differ. TFLite's narrow path is easier to get right and harder to bend. ONNX gives you more control and more ways to be subtly wrong.
Conversion friction: where did your model come from?
Be honest about your training stack, because this often decides the question outright.
- PyTorch → ONNX is a first-class export path maintained by the PyTorch team. It mostly works, and when it doesn't the failure is usually a legible unsupported-operator error.
- PyTorch → TFLite used to mean the ONNX → TensorFlow → TFLite chain, which was reliably painful. Google's ai-edge-torch has improved this a lot, but it's newer and its coverage is narrower than the ONNX exporter's.
- TensorFlow/Keras → TFLite is the native path and the smoothest experience in this whole comparison.
- TensorFlow → ONNX via tf2onnx works, but you're adding a hop and an opset-version negotiation for no obvious gain unless your target demands ONNX.
- Anything with control flow, dynamic shapes, or custom layers will hurt in both directions. Budget time for it rather than discovering it late.
One more thing on operator coverage: TFLite has a deliberately small builtin op set, and the escape hatch — the Flex delegate that pulls in TensorFlow kernels — inflates your binary substantially. That's usually unacceptable on edge targets. ONNX has a much larger opset surface, but opset versioning is its own tax: a model exported at opset 18 may not run on a runtime build compiled for an older one.
Latency, binary size, and what actually moves the needle
We are not going to publish a benchmark table, because any number we gave you would be wrong on your hardware with your model. What we will say is what we consistently observe:
- On ARM CPU with a quantized convolutional model, TFLite with XNNPACK and ONNX Runtime with its default CPU provider land close enough that the difference rarely drives a decision. Single-digit percentage gaps, either direction, model dependent.
- On NVIDIA hardware, ONNX Runtime with TensorRT wins clearly. It isn't close.
- On fixed-function NPUs designed around the TFLite int8 spec, TFLite wins clearly, for the same structural reason.
- Binary footprint favours TFLite at the small end. TFLite Micro is tens of KB; a full TFLite build is on the order of a few hundred KB to low megabytes depending on delegates. Minimal ONNX Runtime builds can be trimmed hard but generally start higher. Treat all of these as approximate and measure your own build.
- Transformer and LLM-style workloads on edge devices are moving fast in both ecosystems and neither has a stable answer yet. If that's your workload, prototype before you commit.
Also worth saying plainly: pre- and post-processing is often a larger share of your end-to-end latency than the model itself. We've seen pipelines where image resize and colour conversion cost more than inference. Profile the whole path before you optimize the runtime choice.
A decision framework you can actually use
- Does your target have less than about 1 MB of RAM, or no OS? Use TFLite Micro, or a vendor compiler that consumes .tflite. Stop here.
- Does the accelerator vendor's SDK document one runtime and not the other? Use that one. Stop here.
- Is the deployment target NVIDIA, Intel, or Windows? Default to ONNX Runtime.
- Is the target a mobile phone with a specific NPU, or a Coral/Ethos-class NPU? Default to TFLite.
- Otherwise, does your team train in PyTorch? ONNX Runtime. In TensorFlow? TFLite. The path of least conversion friction is worth more than a few percent of latency.
- If you're deploying to several heterogeneous targets, consider maintaining ONNX as your interchange format and converting to .tflite only for the targets that require it — rather than the reverse.
When you need neither
Plenty of edge ML projects don't need a neural network runtime at all. If your model is a gradient-boosted tree, a random forest, or logistic regression — which covers a lot of sensor and tabular anomaly-detection work — you can export the trained model to plain C or use a small library and get deterministic, microsecond-scale inference with no runtime dependency. We've done this on projects where the client arrived expecting a deep learning pipeline and left with 400 lines of generated C that ran on hardware they already had.
Similarly, if your device has reliable connectivity, generous latency budget, and non-sensitive data, running inference server-side removes the entire question. On-device inference earns its complexity when you need low latency, offline operation, privacy, or bandwidth savings. If none of those apply, don't pay for it. And a third option worth knowing about: ExecuTorch, PyTorch's own on-device runtime, is maturing and is worth evaluating if you're a PyTorch-first team starting a new edge project today.
How we approach this on client projects
We benchmark on the actual target board, with the actual model, before writing production code. That means a throwaway harness that runs both candidate runtimes on representative input and reports latency percentiles, peak memory, binary size, and accuracy against the float baseline. It usually takes a few days and it has saved clients from committing to the wrong path more than once. You can see the kind of embedded and ML work we take on across our engineering portfolio, and more about how we scope AI and ML development engagements.
If you're deciding between these two runtimes for a product that has to ship, we're happy to look at your model architecture and target hardware and give you a straight recommendation — including "you don't need a framework for this." Get in touch and tell us what you're building, or bring us in as an extended engineering team if you'd rather have the benchmarking and deployment work handled alongside your own developers.
