A device passes every test on the bench, ships, and then starts rebooting in the field once a week. The logs point at a hardware watchdog reset. The hardware team swears the board is fine. Nine times out of ten, when we get called into that situation, the cause is a scheduling problem: a task that holds a mutex too long, a priority assignment that made sense to nobody, or an interrupt handler doing work it should have deferred.
FreeRTOS is small, well documented, and genuinely predictable — but it is predictable in the way a manual transmission is predictable. It does exactly what you tell it to. This article walks through how the scheduler actually makes decisions, how to assign priorities you can defend in a design review, where priority inversion and deadlock come from, and how to build a watchdog strategy that catches a hung task rather than just papering over it.
What the scheduler actually does on every tick
FreeRTOS uses a fixed-priority preemptive scheduler. The rule is short enough to memorise, and worth memorising because every debugging session comes back to it: the scheduler always runs the highest-priority task that is in the Ready state. If several tasks share that priority and time slicing is enabled, it rotates between them on tick boundaries. Nothing else. There is no ageing, no fairness heuristic, no automatic boost for a starved task.
Priority 0 is the lowest and is where the idle task lives. configMAX_PRIORITIES sets the ceiling, and priority numbers count upward — which is the opposite of Cortex-M NVIC interrupt priorities, where lower numbers mean higher urgency. That inversion is one of the most common sources of confusion when a team is wiring interrupts into RTOS API calls for the first time.
A task can leave the Ready state in only a few ways, and knowing them is the fastest way to reason about a stall:
- Blocked — waiting on a queue, semaphore, mutex, event group or task notification, with or without a timeout.
- Blocked on time — vTaskDelay or xTaskDelayUntil, which is the correct choice for periodic work because it corrects for execution jitter.
- Suspended — explicitly parked with vTaskSuspend, which no timeout will rescue. Use it sparingly.
- Running — one per core.
- Deleted — and if a task deletes itself, the idle task must get CPU time to reclaim its memory.
A task that never blocks and sits at a high priority will starve everything below it, including the idle task. If you have tickless idle enabled for power savings, a busy-wait loop quietly destroys your battery budget too.
Assigning priorities you can justify
Most firmware we inherit has too many priority levels. Fifteen tasks with fifteen distinct priorities means nobody knows what the system does under load, because every relationship is a special case. We usually collapse a design into four or five bands and then argue about which band each task belongs in.
A workable starting layout for a typical connected sensor or motor-control product:
- Highest — deferred interrupt handlers. Short tasks woken from an ISR that must respond within a bounded number of ticks: a motor commutation update, a safety cutoff, a time-critical protocol response.
- High — control loops with hard periods. PID loops, sampling schedulers, anything where a missed deadline is a functional defect rather than a delay.
- Medium — protocol stacks and driver-side work. Modbus or CAN handling, a filesystem writer, a display renderer.
- Low — connectivity and housekeeping. TLS handshakes, MQTT publishing, OTA download, telemetry batching. These are slow, bursty, and tolerant of delay.
- Lowest above idle — diagnostics, statistics, logging, watchdog supervision.
For periodic tasks, rate-monotonic assignment is a defensible default: the shorter the period, the higher the priority. It is provably reasonable for independent periodic tasks and it gives you a rule to point at when someone wants to bump their task up. If two tasks genuinely have no ordering requirement between them, give them the same priority and let round-robin handle it. Equal priorities are a design statement, not laziness.
One more thing that saves grief later: give tasks names that appear in a trace, and keep a single header that defines every priority as a named constant. Scattered numeric literals are how priority creep happens over three years of maintenance.
Preemption, time slicing and where jitter comes from
With configUSE_PREEMPTION set, a higher-priority task that becomes ready preempts the running task immediately — including from inside an ISR, if you use the FromISR variants correctly and honour the higher-priority-task-woken flag. Skipping portYIELD_FROM_ISR is a classic bug: the data arrives in the queue, the waiting task is ready, and yet it does not run until the next tick. On a 1 kHz tick that is a millisecond of unexplained latency that will show up in someone's oscilloscope capture and nowhere in your code review.
Interrupt handlers themselves deserve a hard rule: an ISR posts, it does not process. Read the peripheral register, push to a queue or send a task notification, return. Task notifications are noticeably lighter than semaphores or queues when you only need to signal one specific task, and on a tight loop that difference is measurable.
Critical sections are the other jitter source. taskENTER_CRITICAL masks interrupts up to configMAX_SYSCALL_INTERRUPT_PRIORITY. Anything you do inside it — a printf, a flash read, a loop over a linked list — adds directly to worst-case interrupt latency for the whole system. If you find yourself needing a long critical section, you almost certainly want a mutex instead.
Priority inversion, and why mutexes are not semaphores
Here is the failure we see most often in production firmware, described concretely. Three tasks share one SPI bus behind a mutex. A low-priority logging task takes the mutex to append to external flash. Midway through, a medium-priority networking task becomes ready and preempts it — the logger still holds the mutex. Now the high-priority control task wakes on its 5 ms period, tries to read the sensor over SPI, and blocks. It stays blocked for as long as the medium-priority task wants the CPU. A high-priority task is effectively running at the priority of the lowest task in the system. That is unbounded priority inversion.
FreeRTOS mutexes created with xSemaphoreCreateMutex implement priority inheritance: while a lower-priority task holds a mutex that a higher-priority task is waiting on, the holder is temporarily raised to the waiter's priority so it can finish and release. Binary semaphores created with xSemaphoreCreateBinary do not do this. They look interchangeable in the API and they are not.
- Use a mutex for mutual exclusion of a shared resource — a bus, a buffer, a device register block.
- Use a binary semaphore or task notification for signalling — an ISR telling a task that data arrived.
- Use a counting semaphore for managing a pool of N identical resources.
- Never call a mutex take or give from an ISR. Priority inheritance has no meaning in interrupt context, and the API will not protect you from yourself.
- Use a recursive mutex only when a call chain genuinely re-enters the same lock, and treat it as a smell worth revisiting.
Be honest about the limits, too. FreeRTOS implements basic priority inheritance, not a full priority ceiling protocol, and chained inheritance across nested mutexes is not fully resolved. If your timing requirements are tight enough that you need provable bounds through nested locks, restructure the design so there is only one lock on the critical path. That is cheaper than fighting the kernel.
Preventing deadlock rather than detecting it
Deadlock in a small embedded system is nearly always the textbook case: task A holds lock 1 and wants lock 2, task B holds lock 2 and wants lock 1. There is no runtime deadlock detector in FreeRTOS, and adding one is usually the wrong investment. Prevention is structural and it costs almost nothing.
The rules our embedded teams apply on every project:
- Define a global lock ordering. Number every mutex in the system and require that locks are always taken in ascending order. Document it in the header where the mutexes are declared.
- Prefer one lock per critical path. If two resources are always used together, guard them with one mutex.
- Never block while holding a lock. No vTaskDelay, no waiting on a queue, no calling into a driver that might block, while you hold a mutex.
- Always pass a finite timeout. portMAX_DELAY on a mutex take converts a transient problem into a permanent hang. A timeout plus an error path gives you a fault you can log and recover from.
- Own resources instead of sharing them. Give the SPI bus a single owner task and have other tasks send it request messages over a queue. Serialisation by design removes the lock entirely — this is usually the cleanest fix.
- Enable configASSERT in development builds. It catches a large class of misuse, including API calls from the wrong interrupt priority, before it ever becomes a field failure.
That last pattern — one owner task per peripheral, everything else talks to it via queues — is the single change that has removed the most concurrency bugs from the codebases we've refactored. It costs a little RAM and a small amount of latency. It buys you a system where the worst case is analysable.
Watchdogs that actually prove the system is alive
A hardware watchdog kicked from a timer callback proves one thing: your timer is running. It says nothing about whether the control loop is executing or the network task has been stuck in a socket call for four minutes. We treat the watchdog as a system-health question, not a heartbeat.
The pattern we use:
- Every task that matters registers with a supervisor module and declares a maximum interval between check-ins — roughly two to three times its nominal period.
- Each task sets its own flag or timestamp at the top of its loop, after it has done real work, not before.
- A low-priority supervisor task runs periodically, verifies that every registered task has checked in within its declared window, and only then refreshes the hardware watchdog.
- If a task misses its window, the supervisor records which task failed into a small non-volatile region — retained RAM or a flash log — before letting the watchdog fire.
- On boot, the firmware reads the reset cause and the stored record, reports it in telemetry, and increments a counter.
That fifth step turns a mysterious weekly reboot into a diagnosis. Running the supervisor at low priority is deliberate: if a high-priority task spins forever, the supervisor never runs, the watchdog is never refreshed, and the device resets — which is exactly the behaviour you want. A window watchdog, where refreshing too early is also a fault, adds protection against a runaway loop that happens to hit the refresh call.
The instrumentation worth enabling
- configCHECK_FOR_STACK_OVERFLOW with a hook that stores the offending task name — stack overflow is the most common cause of inexplicable corruption in RTOS firmware.
- uxTaskGetStackHighWaterMark sampled periodically and reported, so you can size stacks from measurement rather than guesswork. Trim only after you have seen behaviour across worst-case paths, including error handling.
- Run-time statistics via a high-resolution timer, exposed through uxTaskGetSystemState, so you know where CPU time actually goes.
- A trace recorder for the hard cases. Seeing context switches, blocking events and priority changes on a timeline resolves inversion bugs in minutes that take days to find by reasoning.
- Queue high-water marks. A queue that regularly reaches full is telling you a consumer is under-prioritised.
When you do not need any of this
It is worth saying plainly: plenty of products do not need an RTOS. If your firmware is a sampling loop, a state machine and a UART, a well-structured superloop with a timer tick is easier to reason about, easier to certify, and uses less RAM. The moment you add FreeRTOS you take on stack sizing, lock discipline, and a class of bugs that only appear under load.
The signals that genuinely justify an RTOS are blocking I/O with long and variable latency — TLS, filesystems, cellular modems — several independent activities with different periods, or a third-party stack that expects a threading model. If that describes your product, an RTOS is the right call and the discipline above is what keeps it stable. If it does not, spend the effort on your state machine instead.
We also see a growing number of devices where a control loop shares an MCU with on-device inference. That changes the scheduling conversation: an inference task with a variable execution time sitting above a control loop will wreck your determinism, so it belongs in a lower band with a bounded work quantum. If that is your architecture, our notes on edge and embedded AI work cover how we split those workloads, and you can see the kinds of systems we build in our work.
If you are staring at an intermittent watchdog reset, planning a firmware architecture from scratch, or you need engineers who can sit alongside your existing team through a bring-up, get in touch and tell us what the device does. We are also happy to talk through how a dedicated embedded team usually works in practice before you commit to anything.