The worst firmware bug we have seen was not the bug itself. It was the fact that the device could not be fixed. A few thousand units in the field, no update path, and a technician visit costing more than the hardware. That is the real cost of treating OTA as a feature to add later.
Over-the-air update capability is infrastructure, not a feature. It decides how fast you can respond to a security disclosure, whether you can ship a v1 that is honestly incomplete, and how much of your support budget goes to trucks and postage. This article walks through the architecture patterns that hold up in production — on constrained MCUs and on Linux-class gateways — and the decisions you should make before the first line of application code.
Start With the Failure Modes, Not the Happy Path
Any OTA design is really a list of answers to "what if this stops halfway?" Write the list first. On most projects it looks like this:
- Power is lost mid-write, mid-verify, or mid-swap.
- The connection drops at 60% of a large image and the device reboots before resuming.
- The image is complete and valid but the new firmware crashes on boot, or boots and then cannot reach the network.
- The device has been offline for a year and is several versions behind.
- An attacker on the same network serves a valid-looking but older signed image.
- Two hundred devices update at once and saturate a shared uplink or a customer's cellular plan.
Every pattern below exists to close one of these. If you can trace a design decision back to a specific failure mode, you are on solid ground. If you cannot, you are probably adding complexity you will regret in the bootloader.
The Core Pattern: Dual-Slot A/B Updates
The most reliable and most widely used arrangement is two firmware slots plus a small immutable bootloader. The device runs from slot A, downloads the new image into slot B, verifies it, then marks slot B as the boot candidate. The bootloader tries B; if B does not confirm itself healthy, the bootloader falls back to A on the next reset.
The critical piece is the confirmation handshake. A booted image is not a working image. The new firmware must actively mark itself as good — after it has come up, joined the network, and talked to the backend at least once. Until it does, the bootloader treats the slot as "trial" and a watchdog reset sends the device back to the known-good version. This single mechanism prevents most bricking scenarios.
When you cannot afford two slots
Flash is expensive on low-cost MCUs, and A/B doubles your application footprint. Alternatives, in rough order of preference:
- External flash for staging: download and verify into an SPI NOR chip, then have the bootloader copy into the single application slot. You still get atomicity, at the cost of a longer non-runnable window and a bootloader that must survive interruption mid-copy.
- Compressed or delta staging: keep the staging area small by shipping binary diffs rather than full images. Works well when releases are incremental.
- Swap-with-scratch: some bootloader frameworks perform a sector-by-sector swap using a scratch region, so both images are recoverable. Slower and more flash-wear intensive, but it fits tighter budgets.
- Recovery-mode-only: a minimal, never-updated recovery application whose only job is to fetch and flash. Cheapest in flash, but a bug in that recovery image is unrecoverable, so it must be small and boring.
Whichever you choose, flash wear matters. Estimate the number of updates a device will receive over its life, multiply by the sectors touched, and check it against the endurance rating. Devices meant to run ten years and update monthly are a different problem from a consumer gadget updated twice.
Security: Sign the Image, Not the Channel
TLS protects the transport. It does not protect the artifact. Once an image is written to flash, the only thing standing between you and arbitrary code execution is a signature the bootloader checks before handing over control. Both layers are needed, and they solve different problems.
A workable baseline:
- Asymmetric signatures over the image and its metadata, verified by the bootloader against a public key in read-only storage or an MCU security region. ECDSA on a P-256 curve or Ed25519 are common choices on constrained parts.
- Private keys in an HSM or a cloud KMS, never on a build machine or in a repository. Signing happens as a controlled step in your release pipeline, not on a developer laptop.
- Anti-rollback protection using a monotonic version counter, so an attacker cannot serve a genuinely signed but vulnerable older release. Devices with secure counters or monotonic fuses make this cheap; otherwise store the floor in protected NVM.
- Metadata bound to hardware: model, board revision, and hardware capability flags. Cross-flashing the wrong variant is a surprisingly common field failure and it costs nothing to prevent.
- Encryption at rest if the firmware itself is sensitive IP, with keys held in a secure element or per-device key storage. Be honest about your threat model here — encryption raises the bar against casual extraction, not against a determined lab.
If your hardware supports a secure boot chain — immutable ROM verifying the bootloader, bootloader verifying the application — use it. Retrofitting a root of trust after production is close to impossible, and key provisioning has to be designed into the manufacturing flow, not bolted on afterwards.
Transport, Resumability, and Bandwidth
How the bytes arrive shapes everything. Over Wi-Fi or Ethernet, plain HTTPS range requests to object storage are hard to beat: cheap, cacheable, resumable, and easy to debug. Over MQTT, you will typically use the broker for command and control and hand off the payload to an HTTPS URL, because pushing megabytes through a broker is wasteful.
Constrained links change the calculus. On NB-IoT, LTE-M, or LoRaWAN, the image is the problem. Practical measures:
- Delta updates. Well-structured firmware with stable code layout can produce diffs that are a small fraction of the full image. Gains are highly dependent on your build determinism and link order, so measure on real releases before promising anything.
- Chunked, resumable transfers with per-chunk integrity checks, so a dropped connection costs one chunk rather than the whole download.
- Blocking the update behind conditions: mains power present, battery above a threshold, device idle, off-peak window.
- Bandwidth-aware scheduling per site, so a gateway serving fifty sensors does not try to fan out fifty concurrent downloads.
- Local distribution: a gateway or hub downloads once and serves its local devices over BLE, Zigbee, or a wired bus.
Fleet Orchestration and Staged Rollouts
Device-side reliability gets you a safe update. Server-side orchestration gets you a safe release. These are separate systems and both are required.
The backend needs to know, for every device: current version, hardware revision, last check-in, update state, and the outcome of the last attempt. From that you can build the controls that actually save you:
- Canary group — internal and volunteer devices, single-digit counts, sitting on the release for days rather than hours.
- Percentage ramp — 1%, then 5%, 25%, 100%, with defined bake time at each step.
- Automatic halt — pause the rollout when failed-boot rate, crash reports, or check-in loss crosses a threshold. Automate this; humans do not watch dashboards at 3am.
- Cohort targeting — by hardware revision, region, firmware version, or customer, so you can hold back a fleet that is running a customer's critical process.
- Forced version floors — devices below a security baseline get pulled forward regardless of the customer's update preferences.
Off-the-shelf platforms cover a lot of this. AWS IoT Jobs, Azure Device Update, Mender, Balena, and the Eclipse hawkBit project all give you a rollout engine and a device agent. On the MCU side, MCUboot has become the default trusted bootloader for Zephyr, nRF Connect SDK, and many vendor stacks; ESP-IDF ships a solid A/B implementation for ESP32. Our default advice is to adopt a proven bootloader and a proven rollout service, and spend your engineering budget on the parts specific to your product. Writing your own bootloader is a real project with a long tail of subtle bugs, and it rarely earns its keep. We are happy to be talked out of that position when there is a genuine constraint — a certification regime, a very unusual memory map, a hard bill-of-materials limit — but the burden of proof sits with the custom option.
Testing an Update System You Cannot Physically Reach
OTA is the one subsystem where the test plan has to include deliberate sabotage. The tests that find real bugs are the ugly ones:
- Power-cut sweeps: a programmable relay killing power at randomised points across the download, verify, swap, and first-boot phases, repeated hundreds of times.
- Corrupted and truncated images, valid signatures over wrong payloads, correct payloads with mismatched hardware metadata.
- Downgrade attempts with genuinely signed older artifacts.
- Version-skip upgrades: v1.0 straight to v4.2, including any migration of persisted data and configuration schemas.
- Network chaos: high latency, packet loss, mid-transfer DNS failure, TLS certificate rotation on the server side.
- Long-haul soak: repeated update cycles on a rack of devices to expose flash wear, memory fragmentation, and slow leaks in the update agent.
Build this into hardware-in-the-loop CI early. A rack of a dozen devices with switchable power and a test harness is modest investment against a field recall. You can see the kind of embedded and connected-product work we take on in our project work, and if you need engineers embedded alongside your own team for a hardware programme, a dedicated team engagement is usually the cleaner fit than a fixed-scope build.
Where Fleet Data Earns Its Keep
Once telemetry from update attempts is flowing, patterns emerge that no test rack will show you: a specific hardware revision that fails verification more often, a regional carrier that drops transfers at a particular size, a battery threshold that is set too low in practice. Most of this is answerable with straightforward queries and good dashboards, and that is where we would start. If your fleet is large enough that failure signatures are getting lost in the noise, anomaly detection over device telemetry becomes worth the effort — that is where our AI and ML work tends to fit, as a layer on top of solid instrumentation rather than a substitute for it.
A Short Checklist Before Production
- Immutable bootloader with signature verification and a tested fallback path.
- Trial-boot confirmation driven by the application, not just a successful reset.
- Signing keys in an HSM or KMS, with a documented rotation and revocation plan.
- Anti-rollback floor enforced on-device.
- Hardware and variant checks in image metadata.
- Resumable, chunked transfers with per-chunk integrity.
- Backend inventory of version state for every device.
- Staged rollout with automated halt criteria.
- HIL tests covering power loss, corruption, downgrade, and version skips.
- A documented recovery procedure for the case where all of the above fails.
If you are designing a connected product now, the cheapest time to get this right is before the enclosure is tooled and the flash budget is fixed. If you already have devices in the field and no safe update path, that is a solvable problem too, though the options narrow. Either way, tell us about your hardware and constraints and we will give you a straight read on what the update architecture should look like — including when the answer is to adopt an existing platform rather than build.
