Most engineering teams know, intuitively, that manual testing has limits.

What fewer teams appreciate is that the absence of automated testing does not eliminate the cost of those limits — it simply moves the cost somewhere else, usually somewhere more expensive, and almost always somewhere harder to see on a budget line.

This is the quiet problem with manual testing in wireless embedded products. It looks free because nobody invoices you for it. It looks reasonable because it produces results: bugs do get found, releases do go out, products do ship. And for a small team launching its first device, it really can get you a long way. But as the product matures, as the codebase grows, and as the team scales, the bill starts to come due — not in the QA budget, where you might notice it, but scattered across firmware engineer time, support tickets, delayed releases, and the occasional field incident that nobody saw coming.

It is worth being specific about where this hidden cost actually lives.

The cost that lives in engineer time

Every manual test cycle costs an engineer somewhere between several hours and several days, depending on the breadth of the regression suite. That cost is invisible because the engineer was already on payroll, but it is real: the time spent flashing devices, pairing phones, checking connection times by stopwatch, and clicking through pairing flows is time not spent designing features, fixing bugs, or improving the product.

The cost compounds as the test surface grows. A test plan that took two hours to execute when the product had three features takes two days when it has thirty. The number of test cases grows roughly linearly with feature count, but it is rare for the QA budget to grow at the same rate. What happens instead is that the regression suite gets pruned — quietly, by individual engineers making individual judgement calls about which tests are “probably fine to skip this time.” Coverage erodes, and nobody notices until something breaks in production that should have been caught.

cover93

The cost that lives in feedback latency

In modern development, code changes happen continuously. Manual testing cannot. A bug introduced on Monday morning typically does not get discovered until Friday’s pre-release test run, by which point the engineer who wrote the bug has context-switched two or three times and has only a faint memory of why the change was made.

The cost of fixing a bug rises sharply with the time between introduction and discovery, because debugging requires reconstructing context that has decayed. A bug caught within minutes of being written takes minutes to fix. The same bug caught a week later takes hours, because the engineer must rebuild the mental model from logs and commit messages. Caught a month later, it might take days.

This is not a theoretical claim. It is observable in the time-to-fix data of any team that has moved from release-gate manual testing to continuous automated testing: the median time-to-fix collapses, often by an order of magnitude, because engineers are working on bugs that they wrote yesterday, not bugs that someone else wrote in code they have never seen.

The cost that lives in environmental noise

Manual testing happens in whatever environment the tester happens to be in. One engineer uses an iPhone 14 in a quiet office. Another uses a Pixel 7 in a meeting room next to a Wi-Fi access point and a microwave. A third uses an older iPhone with a beta iOS build. Results vary, and not because of the product.

The cost of this noise shows up in two places. First, real bugs get dismissed as flakiness because they cannot be reproduced reliably across testers. Second, environmental issues get reported as bugs and consume engineering time before being closed as “not reproducible.” Both failure modes are corrosive: the first lets defects ship, the second wastes the time you have.

The deeper cost is that reproducing a failure reported two weeks ago, under conditions that no longer exist, is often impossible. The bug stays open. It accumulates comments. Eventually it gets closed without resolution, and the team learns — without ever explicitly deciding to — that some classes of bug are simply not worth investigating.

cover67

The cost of the tests you never run

Some test categories are essentially impossible to execute manually, and so they do not get executed at all. Long-term stability is the most obvious example: a device that drops its connection after seventy-two hours of continuous operation cannot be discovered by anyone holding a stopwatch. The test takes seventy-two hours, and no engineer is going to sit in a lab watching a device for three days.

Performance benchmarks fall into the same category. Connection establishment time, throughput percentiles, reconnection latency after link loss — these are quantitative, time-sensitive measurements that humans cannot capture accurately. Power consumption regressions are even worse: a firmware change that increases advertising current by ten percent will not show up in any functional test, but it will show up in the field as a halving of battery life six months after the device ships.

These categories of testing do not appear in manual test plans because the people writing the plans know, sensibly, that humans cannot perform them. The result is that entire classes of regression — stability, performance, power — go undetected until they reach customers.

The cost of features that never get built

The most insidious form of hidden cost is the one that never appears as a cost at all, because it shows up as something that did not happen. Every hour an engineer spends running manual regression tests is an hour not spent designing the next feature, exploring the bug list, refactoring fragile code, or thinking about the architectural decisions that determine the product’s trajectory. This is opportunity cost, and it is genuinely large.

It is also genuinely difficult to measure, which is part of why it gets ignored. There is no line item for “features we did not build because we were testing instead of building.” The product roadmap simply moves a little slower than competitors. The technical debt grows a little faster than it should. Customers wait a little longer for the improvements they have asked for. Each individual decision to defer a feature in favour of a manual regression run looks rational in the moment, because the regression has to be done. The cumulative effect, observable only in retrospect, is a product that lags.

Teams that move to automation often report a curious second-order effect: their feature velocity increases not just because automation finds bugs faster, but because the engineers who used to spend Friday afternoons running manual tests now spend that time building. The reclaimed hours are surprisingly significant. A small team running an eight-hour regression suite once a fortnight is losing roughly five percent of its engineering capacity to manual testing, and that is before counting the time spent writing new test cases, training new engineers on the test procedure, or investigating false positives. Five percent of engineering capacity, recovered, is the equivalent of hiring half an engineer at no cost.

background image for partnerships

The cost that lives in team morale

There is a human cost to manual testing that engineering leaders often underweight, because it does not appear in any operational metric. Running the same checklist of test cases for the third release cycle in a row is genuinely demoralising work. It is repetitive, it is uncreative, and it rewards diligence rather than skill. The engineers asked to do it are typically the same engineers who chose this career to build interesting systems, not to verify that a pairing flow still works the way it worked last quarter.

The morale cost manifests in two ways. First, the work itself gets performed less well over time, because human attention degrades on repetitive tasks. The fifth pass through the regression checklist is genuinely less thorough than the first, regardless of the engineer’s intentions. Second, and more strategically damaging, talented engineers leave teams where their time is consumed by repetitive verification work. The market for embedded firmware engineers with wireless protocol experience is competitive enough that those engineers can choose where to work, and they tend not to choose the team that spends three days every release cycle clicking through manual test plans.

The retention cost compounds with the cost of hiring replacements. Each departure takes institutional knowledge with it, requires onboarding for the replacement, and during the transition leaves the team with one fewer experienced engineer to perform the manual testing it was already struggling to keep up with. The cycle is self-reinforcing in a way that is genuinely difficult to recover from once it sets in.

This is not an argument that automation is purely a retention play. It is an observation that the framing of manual testing as “cheap labour we already have” misses a real and significant cost: the engineers performing the labour are not infinitely available, and the quality of their work, the speed of their execution, and their willingness to do it all degrade with time in ways that do not show up on any spreadsheet until they show up all at once in resignation letters.

The cost that lives in production

This is the cost that finally becomes visible, because it shows up as support tickets, returns, escalations, and reputational damage. A bug found by a customer is the most expensive bug there is: it requires triage, root cause analysis, a hotfix release, customer communication, and frequently a post-mortem that consumes more engineering time than the original feature ever did.

The conventional wisdom that bugs are roughly an order of magnitude more expensive at each stage of the pipeline — cheapest at developer desk, expensive in QA, very expensive in production — applies to wireless embedded systems with particular force, because field issues are often genuinely difficult to reproduce. A connection drop that happens once every four hours in a customer’s apartment building does not happen at all in your office. Without an automated test that can simulate the conditions, you may never reproduce it. The bug stays open. The customer churns. The cost is real but it does not appear on any line item.

What this adds up to

Teams that rely on manual testing typically do not see a single large cost. They see many small ones, distributed across functions, none of them obviously attributable to the testing strategy. The release cycle is a little longer than it should be. The bug list is a little longer than it should be. Engineers spend a little more time on triage than on features. The product is a little less stable than competitors. None of these are catastrophic in isolation, and so none of them trigger a strategic response.

The argument for automation is not that manual testing is wrong. It is that the costs of relying on it are real, large, and systematically underestimated, because they are spread across functions and time horizons in ways that no individual budget captures. The teams that move first are not the ones with the largest QA budgets. They are the ones whose leadership decides to start counting costs that nobody else is counting.

needCode designs and delivers complete automated test systems for embedded wireless products — from BLE mesh and multi-protocol IoT to LTE-connected devices and BLE mobile applications. If you are weighing the cost of automation against the cost of not having it, we are happy to walk you through what a test infrastructure tailored to your product would actually involve.

Book a free discovery call or get in touch

Further reading