Every CI pipeline for wireless embedded eventually arrives at roughly the same shape.

There are tests that run in seconds and gate every commit. There are tests that take minutes and gate every merge. There are tests that run for an hour or two and run nightly. There are tests that run before a release. The exact details vary across teams, but the four-tier structure recurs because it reflects something fundamental about how testing trades off speed against coverage, and how those trade-offs interact with the workflows of real engineering teams.

This article is about that structure: what each tier does, why it exists, what belongs in it, and how the four tiers fit together to produce a development workflow that is fast at the points where speed matters and thorough at the points where thoroughness matters. The framing is not novel. But it is worth treating as a deliberate design rather than something teams stumble into, because the teams that understand the structure deliberately build pipelines that work well, while the teams that arrive at it accidentally tend to build pipelines that work badly.

The fundamental tension that shapes everything

Before discussing the tiers themselves, it is worth understanding the tension they exist to resolve, because that tension explains every design choice that follows.

Testing produces value by catching defects. The earlier a defect is caught, the cheaper it is to fix — this is one of the few genuinely robust empirical results in software engineering, and it applies as forcefully to wireless embedded as to any other domain. The implication is that you want every test to run as often as possible, ideally before every code change reaches the shared codebase. If every developer’s local machine ran every test before every commit, defects would be caught at the earliest possible moment and the cost of fixing them would be minimised.

This ideal collides immediately with reality. Some tests are slow. Some tests require physical hardware that not every developer has. Some tests need an entire system — firmware, mobile app, backend, frontend — to be in a consistent, working state, which is true only intermittently. Some tests take so long that running them frequently would consume more engineering time than the bugs they catch would cost. The trade-off is sharp: running every test on every change is impossible, but running tests only at release time loses most of the value that early defect detection provides.

The four-tier structure resolves this tension by recognising that not all tests are alike. Different categories of tests have different speed characteristics, different infrastructure requirements, different sensitivity to environmental factors, and different probabilities of catching the kinds of regressions that typical changes introduce. By grouping tests into tiers based on these characteristics, and by running each tier at the cadence appropriate to its characteristics, the pipeline maximises the value of testing while keeping the cost manageable.

The reason four tiers and not three or five is not magical. Three tiers tends to combine categories that have meaningfully different characteristics, producing a pipeline where some tier is either too slow for its frequency or too sparse for its coverage. Five or more tiers tends to introduce distinctions that do not actually correspond to different test characteristics, adding configuration complexity without proportional benefit. Four tiers — pre-commit, post-commit, nightly, release — captures the meaningful distinctions cleanly, and the boundaries between them tend to fall naturally along the lines of “what kind of infrastructure does this test need” and “how often is it worth running.”

Tier one: pre-commit

The first tier runs on every code change, before the change is shared with the rest of the team. In practice this often means running on every push to a feature branch and on every pull request, possibly even on every save in the developer’s editor. The defining characteristic of this tier is speed: the tests must complete in seconds to a few minutes, because slower feedback erodes the value of running them at all.

The speed constraint determines what belongs in this tier. The tests must be runnable purely in software, without any physical hardware in the loop, because hardware introduces both latency and unreliability that the pre-commit tier cannot tolerate. The tests must be deterministic, because flaky tests at this tier consume developer attention disproportionately to their value. And the tests must be focused on what individual changes are likely to affect, because exhaustive coverage at this tier would slow the feedback loop too much.

What this looks like in practice for a wireless embedded project is a combination of several things. Compilation of the firmware against all supported hardware variants, verifying that no change has broken the build for any target. Static analysis runs that catch a curated set of code quality issues — uninitialised variables, obvious null dereferences, suspicious type conversions, formatting violations — without producing the false-positive volume that comprehensive static analysis would produce. Unit tests that exercise individual functions and modules in isolation, running on the development host rather than on target hardware. Component-level tests that exercise firmware modules against software simulations of their dependencies — a connection state machine tested against a simulated link layer, a security module tested against synthetic input vectors.

These tests are valuable not because they are exhaustive but because they catch the largest class of bugs at the lowest cost. A typical change that breaks a build, introduces a static analysis warning, or fails a unit test is caught within minutes of the change being written, when the developer’s context is fresh and the cost of fixing the issue is at its lowest. The pre-commit tier is, in some sense, where most of the team’s testing investment pays back, because it catches the highest volume of regressions per dollar of infrastructure investment.

The discipline that makes this tier work is keeping it fast. Every test that gets added to pre-commit is added at the cost of every developer’s feedback latency, and feedback latency that crosses some threshold — somewhere around three to five minutes for most teams — starts to be ignored or worked around. A pre-commit suite that takes ten minutes to run is one that developers find ways to bypass, which defeats its purpose. The teams that maintain pre-commit discipline are the ones that periodically prune slow tests out of the tier and into a higher tier, accepting that some specific test will run less frequently in exchange for keeping the per-commit feedback loop fast.

Tier two: post-commit

The second tier runs when a change is merged into a shared branch — typically the main branch or a release branch. The change has already passed pre-commit, so the tests at this tier do not need to repeat what pre-commit covered. What this tier provides is the layer of testing that pre-commit could not: tests that require physical hardware, tests that take longer to run, and tests that exercise the integration of components rather than components in isolation.

The defining characteristic of this tier is hardware-in-the-loop. Real firmware runs on real silicon, communicating with real test peers — typically simulated peers driven through controllable test infrastructure rather than other real devices, but the wireless communication itself is genuine over-the-air. The tests verify behaviours that cannot be exercised in software simulation: actual radio operation, actual link layer timing, actual flash memory behaviour, actual peripheral responses. The tests also verify integration: that the various components of the firmware, when assembled into a complete build, work correctly together.

The cadence of this tier is meaningfully slower than pre-commit. A typical post-commit run takes tens of minutes, sometimes longer, because it includes provisioning hardware, flashing firmware, running tests against the live device, and capturing structured results. The cadence is also tied to merges rather than commits, which means the test runs less frequently than pre-commit but on changes that have a higher probability of causing integration issues — because pre-commit has already filtered out the changes that fail at the component level.

What this looks like in practice is a test rig with one or more devices under test, controllable test peers, instruments for measuring whatever the integration tests need to measure, and orchestration software that coordinates the whole arrangement. The tests cover provisioning, connection establishment, the major data exchange paths, reconnection after induced link loss, basic security operations, and any other integration scenarios that exercise the most-used parts of the firmware in conditions close to deployment.

The post-commit tier catches a different category of bugs than pre-commit. Bugs that only manifest with real radio communication. Bugs in the interaction between components. Bugs in timing-sensitive code that software simulations approximate but do not replicate exactly. Bugs in flash memory handling, peripheral driver behaviour, or any other code that depends on physical hardware in ways that simulation cannot capture. These are bugs that pre-commit cannot catch, and they are caught here, before they reach the more expensive testing tiers above.

The discipline that makes this tier work is treating the test rig as production infrastructure. The hardware must be reliable, the configuration must be version-controlled, the orchestration must be observable, and a flaky test rig must be treated with the same urgency as a flaky production system. Teams that let the post-commit infrastructure decay end up with intermittent failures that the team learns to ignore, which destroys the value of the tier. Teams that maintain it well find that it catches a meaningful flow of integration bugs continuously, with relatively modest engineering attention per bug caught.

Tier three: nightly

The third tier runs on a daily cadence, typically overnight when the test infrastructure is otherwise idle. The defining characteristic is breadth: the nightly tier exercises the entire product ecosystem rather than the firmware in isolation. This is where end-to-end testing happens — full provisioning flows from the mobile application through the firmware to the cloud backend, full data exchange across the complete system, full firmware update operations including over-the-air delivery, recovery scenarios after various failure modes.

The reason these tests do not belong in earlier tiers is that they require multiple components to be simultaneously in a known state, and they take long enough to run that running them on every commit or merge would consume disproportionate test capacity. They are also more sensitive to environmental factors — backend availability, mobile application versions, cloud service quirks — than the lower tiers, which means they are harder to make perfectly deterministic. Running them nightly gives the test infrastructure room to handle these complications without blocking the per-commit and per-merge feedback loops.

What this tier produces is the team’s daily health report. Each morning, engineers see whether the integrated system passed last night’s full test cycle, and if not, which scenarios failed. The granularity of failure information is important: a nightly suite that produces a single pass-or-fail result for the whole system is much less useful than one that produces structured results across hundreds of scenarios, because the latter lets engineers understand which specific integration broke and which subsystem is responsible.

A practical nightly suite includes coverage of: representative end-to-end user flows that exercise the major product features; firmware update operations across the supported version transitions; recovery scenarios after various induced failures; protocol edge cases that the lower tiers do not have time to cover; performance measurements that produce trend data over time; and any other tests that benefit from running with the complete system but do not need to run more frequently than once a day.

The nightly tier also serves as a holding area for tests that are still being stabilised. A test that catches real bugs but is occasionally flaky is more valuable in the nightly tier — where its flakiness is annoying but not blocking — than in pre-commit, where its flakiness would destroy the team’s trust in the entire pipeline. Once a test is reliably stable in the nightly tier for some time, it can be considered for promotion to a faster tier. This staging pattern keeps the high-cadence tiers reliable while still allowing new tests to be added without risk to the overall pipeline trustworthiness.

Tier four: release

The fourth tier runs when a release is being prepared. It is, in a sense, the union of everything: the full pre-commit and post-commit suites, the full nightly suite, plus any tests that are too expensive or too specialised to run regularly even at nightly cadence. Certification automation runs at this tier. Comprehensive interoperability testing across the matrix of supported peer devices runs here. Long-form performance benchmarks that produce the data backing product specifications run here.

The defining characteristic is comprehensiveness. The release tier is the gate that determines whether the candidate firmware is fit to ship, and it covers everything that the team considers part of release-worthiness. The tests at this tier do not need to run frequently, because most changes have already been validated by the lower tiers; what this tier provides is the final, definitive check that nothing has slipped through.

The cadence is whatever the release cadence requires — typically per release candidate, sometimes more frequently if the team is doing rapid iteration on a release. The runtime is measured in hours: a full release pipeline that includes certification automation, interoperability testing, and performance benchmarks can easily take a full day of test infrastructure time. This is acceptable because release frequency is bounded; even a team releasing every two weeks is only running this tier a few times a month, and the comprehensive coverage justifies the runtime.

The most important property of this tier is that it is a gate. A release that fails any test at this tier does not ship, full stop. The tests are not advisory, and their results are not subject to interpretation. This discipline is what makes the release process reliable: when the pipeline says the release is good, the team can ship with confidence; when the pipeline says it is not, the team works on what failed rather than debating whether the failure is acceptable. Teams that allow exceptions at the release tier — overriding test failures because of schedule pressure or because the failure is judged to be a false positive — gradually erode the meaning of the release gate, which produces inconsistent release quality and eventually the kinds of field issues that proper release gates exist to prevent.

The tiers that do not fit cleanly

Several categories of tests do not fit neatly into the four-tier structure, and they are worth addressing because every team encounters them.

Long-term stability tests run for days at a time and cannot fit any of the standard tiers. They typically run on a weekly or per-release-candidate cadence, against dedicated test infrastructure that has nothing else to do during the run. The results feed into a stability dashboard that tracks trends over time rather than producing pass-or-fail outcomes per run. These tests are essential for catching certain categories of bug — memory leaks, slow resource exhaustion, watchdog drift — that the other tiers cannot catch, but they exist as a parallel track rather than as part of the main four-tier flow.

Performance benchmarks have similar characteristics: they produce continuous metrics rather than discrete pass-or-fail results, they benefit from being run under controlled conditions that are not always available, and their value is in trend analysis rather than per-run validation. Most teams run them on a regular cadence — often nightly for a subset and per-release-candidate for the comprehensive suite — and feed the results into dashboards that track key performance indicators across firmware versions.

Certification testing typically belongs at the release tier but can also be run more frequently if the test infrastructure supports it. Teams that have automated their Bluetooth qualification process, for example, often run a subset of the qualification suite nightly and the full suite at release, catching specification compliance regressions early without consuming the full runtime cost daily.

Security testing similarly spans tiers. The fast security tests — input validation, basic authentication checks, replay rejection — belong in pre-commit or post-commit because they are fast and they catch regressions in security-critical code. The slower or more specialised tests — fuzzing campaigns, comprehensive cryptographic protocol testing, hardware-level security verification — typically run at nightly or release cadence depending on their runtime and infrastructure requirements.

The general pattern is that specialised testing categories slot into whichever tier matches their speed and infrastructure characteristics, with the boundaries occasionally blurring between tiers. The four-tier structure remains the organising backbone, but real pipelines include some additional structure for the categories that do not fit.

Common antipatterns

Three antipatterns recur often enough across teams to be worth flagging explicitly, because each of them defeats the purpose of having tiered testing in the first place.

The first is letting the pre-commit tier grow until it is no longer fast. A team adds tests to pre-commit because the tests genuinely catch bugs, and over time the tier slows down. Once it crosses the threshold where developers find it too slow to wait for, they start finding ways to skip it. The cure is periodic review of the pre-commit suite with explicit focus on runtime: tests that have grown slow are migrated to higher tiers, sometimes painfully but always worth it, because the pre-commit tier’s value depends on being fast.

The second is treating the post-commit tier as a place where flaky tests can live indefinitely. A test that fails intermittently for reasons unrelated to the code change is corrosive to the pipeline regardless of which tier it lives in. The post-commit tier does not have the same fast-feedback constraints as pre-commit, but it still gates merges, and merges that get blocked by flaky tests train the team to ignore failures. Flakiness must be fixed or the test must be removed; there is no third option.

The third is allowing exceptions at the release tier. The release gate’s value comes from being unconditional. A team that develops a culture of overriding release test failures gradually loses the discipline that makes the release tier valuable, and eventually finds that releases are shipping with known issues because the gate has become advisory. The cure is cultural rather than technical: the release tier must be treated as a hard gate by everyone in the organisation, including leadership, and overriding it must require an explicit, documented exception that is genuinely uncomfortable to invoke.

What this structure produces

The four-tier pipeline, when properly implemented, produces a development workflow with characteristics that compound favourably over time. Developers get fast feedback on most changes, which makes them more productive and more willing to take on tasks that touch sensitive parts of the codebase. Integration regressions are caught at merge time, when the change is recent and the cost to fix is low. Daily nightly runs produce a continuous health signal that the team can rely on. Releases happen on a predictable cadence, with a quality bar that is enforced automatically rather than relying on human discipline.

The investment to build this structure is meaningful but bounded. A team that starts with no automation and aims for the full four-tier structure can typically reach a working version of all four tiers within a year, with continued maturation over the following year or two. The pieces are mostly modular, so progress at each tier produces incremental value rather than waiting for everything to be in place before any of it is useful.

For wireless embedded teams in particular, the tiered structure is well-suited to the operational realities of the domain. The split between software-only tests at the lowest tier and hardware-in-the-loop tests at higher tiers maps naturally onto what is fast versus slow, what is deterministic versus environmental, and what catches what kinds of bugs. The four-tier pipeline is, in some sense, the structure that wireless embedded testing converges to when the team thinks carefully about what it actually wants from its CI infrastructure. The teams that recognise this and build deliberately tend to end up with pipelines that work well; the teams that build reactively, adding tests where bugs hurt most without thinking about tier discipline, tend to end up with pipelines that work badly. The difference between the two outcomes is mostly architecture, and the architecture is worth getting right.

needCode designs and delivers tiered CI/CD pipelines for embedded wireless products, with the test infrastructure, hardware test rigs, and orchestration that each tier requires. We have built four-tier pipelines across BLE mesh, multi-protocol IoT, and LTE-connected devices, and we know what makes the structure work in practice. If you are designing a pipeline from scratch — or trying to fix one that is not delivering the value it should — we are happy to talk through what a tier-by-tier rebuild would involve.

Book a free discovery call or get in touch

Further reading