Every embedded team has lived through some version of the same conversation.

A developer pushes a change that builds cleanly on their machine. The CI build fails. The developer pulls the failing log, cannot reproduce the failure locally, and spends two hours establishing that the CI is using a slightly different version of the ARM toolchain than they are. Three weeks later, a new team member joins, and their first three days are consumed by toolchain installation, vendor SDK setup, and the discovery that the project’s build script depends on a Python package that conflicts with the one their Linux distribution ships by default. Six months later, an SDK customer reports that the example code does not build on their system, and after a long support thread it turns out that their version of make handles a particular variable substitution differently from the version the team uses internally.

These are all manifestations of the same underlying problem: environment divergence. The build of an embedded project depends on a long list of tools — compilers, linkers, flashers, debuggers, code generators, vendor utilities, scripting languages, libraries — each of which has versions, configuration, and platform-specific behaviour. Across developer machines, CI runners, and external customers, the versions and configurations drift in ways that produce failures with no obvious cause. The cost of these failures is large and largely invisible, because each individual incident looks like an isolated configuration problem rather than a symptom of a structural issue.

This article is about the structural fix for environment divergence in embedded development: containerising the build environment with Docker, providing developer-friendly access through dev containers, and using the same containers as the distribution mechanism for SDKs delivered to external customers. The technique is not new in software development broadly, but it is genuinely under-applied in embedded specifically, and the benefits when it is applied properly are larger than they look from the outside.

Why embedded is particularly vulnerable

To understand why containerisation matters more for embedded than for, say, a typical web application, it helps to think about what an embedded build environment actually contains. A web project might depend on a particular version of Node.js and a list of npm packages — both of which are themselves managed with established tools that handle version pinning reasonably well. The build environment is essentially Node plus dependencies, and the dependencies are themselves containerised by virtue of living inside node_modules.

An embedded project is structurally different. It typically depends on at least the following: a cross-compiling toolchain such as gcc-arm-none-eabi or armcc, with specific version sensitivity because subtle code generation changes between versions can affect timing-critical code; a vendor SDK supplied by the silicon vendor, often with its own installer that mutates system state in ways that are hard to reverse; flashing tools that talk to specific debug probes through specific USB drivers; a code generator that produces, for instance, the Bluetooth GATT database from an XML description, where different versions of the generator produce different output; a static analysis tool with its own configuration database; a Python environment for build scripts and tests, with packages that may conflict with system Python; a documentation generator; and any number of vendor-specific utilities for things like signing firmware images or generating provisioning artefacts.

Each of these tools has version sensitivity. Each has installation procedures that vary across platforms. Several of them assume specific paths, environment variables, or registry entries that conflict with each other when multiple versions are installed side-by-side. The total complexity of an embedded build environment is genuinely larger than most developers realise, and the divergence that accumulates between developer machines is correspondingly larger.

The result is that “works on my machine” failures in embedded are both more frequent and harder to diagnose than in most other categories of software. The environments are more complex, the failure modes are more varied, and the diagnostic process — comparing two heterogeneous machine configurations to find the relevant difference — is genuinely difficult.

What Docker actually does

Before discussing how Docker solves this, it is worth a brief refresher on what Docker actually is, because the technology is sometimes treated as more magical than it is. A Docker image is, mechanically, a packaged filesystem containing everything a particular workload needs to run: the operating system files, installed tools, configuration, environment variables, scripts. A Docker container is a running instance of an image, isolated from the host system in a way that prevents the container’s environment from affecting the host and vice versa.

The relevant property for our purposes is reproducibility. An image is built once from a specification — a Dockerfile — and that image is then bit-for-bit identical wherever it runs. If your CI runs in a Docker image that includes gcc-arm-none-eabi version 12.2 and a specific Python environment with specific packages, then a developer running the same image gets the exact same toolchain and Python environment, regardless of what is installed on their host machine. The environment is no longer something each developer assembles individually; it is a versioned artefact that everyone shares.

The Dockerfile that specifies the image is itself plain text and is checked into the same repository as the code. When the toolchain version needs to change, the Dockerfile is updated, the image is rebuilt, and everyone — developers, CI, customers — moves to the new toolchain at the same time. The environment becomes part of the project, traceable through version control alongside the code that depends on it.

This is the core property that Docker brings to the table: the build environment becomes a versioned artefact rather than a collection of installation procedures. Everything else about Docker — the containerisation, the isolation, the orchestration — is incidental from the embedded developer’s perspective. The thing that matters is the reproducibility.

What dev containers add

Docker by itself solves the CI side of the problem reasonably well. Configure the CI to run inside the project’s image, and the CI environment is guaranteed to match the specified configuration. The developer side, however, is more nuanced. Developers want to run their editor with full code intelligence, debug the running build, navigate the codebase with their familiar tools — and forcing them to do all this through docker run commands and bind-mounted volumes is awkward enough that adoption tends to be patchy.

This is the gap that dev containers fill. A dev container is, at its simplest, a Docker container configured for interactive development, with the editor running inside or outside the container in a way that makes the experience feel local even though the build and runtime are happening inside the container. Visual Studio Code’s Remote Containers extension was the first widely-adopted implementation of the pattern, and several other editors have similar capabilities now. The user opens a project, the editor launches the project’s container, and the developer sees their familiar editor with full code intelligence, debugging, and terminal access — all operating inside the project’s containerised environment.

The configuration for the dev container is itself part of the repository. A .devcontainer/devcontainer.json file specifies which Docker image to use, which extensions to install in the editor, which ports to forward, which folders to mount. A new developer who clones the repository and opens it in a compatible editor sees a prompt: “this project has a dev container, would you like to open it?” One click later, the editor is running inside the project’s environment, and the developer is ready to build and debug.

This dramatically changes the new-developer onboarding experience. The traditional path — install the toolchain, install the SDK, install the flashing tools, configure paths, install Python and the project’s Python dependencies, troubleshoot when something breaks because of an existing tool that conflicts — typically takes a day or two and produces a working environment that nevertheless drifts from everyone else’s environment over time. The dev container path takes a few minutes and produces an environment guaranteed to match the team’s standard. The productivity difference for the new developer’s first week is substantial, and the long-term divergence problem is essentially eliminated.

What a containerised embedded project looks like

In practice, the pieces fit together fairly cleanly. The project’s Dockerfile installs the toolchain, the vendor SDK, the code generators, and the Python dependencies. It also installs any utilities the build needs, configures environment variables, and sets up a non-root user with appropriate permissions. The image is built and pushed to a private container registry, where it is tagged with a version string that the project’s documentation references.

The CI configuration runs the build inside this image. A typical CI job is essentially a sequence of commands inside the container: build, run unit tests, generate artefacts. The CI configuration is brief because most of the complexity has moved into the image. The same image can be used for multiple CI workflows — pre-commit checks, integration tests, release builds — with the only variation being which commands run inside the container.

The dev container configuration references the same image, possibly with a few extensions for development-specific tools that are not needed for CI: language servers, formatters, debuggers, additional shell utilities. The dev container also configures volume mounts for the project workspace and any persistent data the developer needs across container restarts.

A simplified Dockerfile fragment for an embedded project might look like this:

# Use a small Linux distribution as the base — Debian slim is a common choice

# for embedded build environments because it has a familiar package manager

FROM debian:bookworm-slim

# Pin the toolchain version explicitly — this is the single most important

# line in the file, because toolchain version drift is the most common cause

# of build environment incidents

ARG ARM_GCC_VERSION=13.2.rel1

RUN apt-get update && apt-get install -y –no-install-recommends \

    wget \

    xz-utils \

    python3 \

    python3-pip \

    && rm -rf /var/lib/apt/lists/*

# Download and install the cross-compiler at the pinned version

# Using a specific archive URL ensures we get the exact version we expect

RUN wget -O /tmp/arm-gcc.tar.xz \

    https://example-toolchain-host/arm-gcc-${ARM_GCC_VERSION}.tar.xz \

    && tar -xJf /tmp/arm-gcc.tar.xz -C /opt \

    && rm /tmp/arm-gcc.tar.xz

# Add the toolchain to PATH so build scripts find it without extra configuration

ENV PATH=”/opt/arm-gcc-${ARM_GCC_VERSION}/bin:${PATH}”

# Install Python build/test dependencies — pinning versions matters here too

COPY requirements.txt /tmp/requirements.txt

RUN pip3 install –no-cache-dir -r /tmp/requirements.txt

This is simplified for clarity, and a real project’s Dockerfile would include the vendor SDK installation, code generators, flashing utilities, and any other project-specific tooling. The point is that everything is explicit and reproducible. Anyone who builds this image at any time gets the same environment, which is the property that eliminates “works on my machine” failures.

The hardware access limitation

It would be misleading to suggest that containerisation is a pure win in embedded development, because it has one genuine limitation that needs to be planned around. Containers, by design, isolate the running process from the host’s hardware. When the build environment needs to flash a physical device, run a debugger against it, or capture serial output, the container needs to be granted access to the relevant USB devices.

On Linux hosts, this is straightforward: USB devices can be passed through to containers with appropriate permissions, and tools like the –device flag in Docker handle the configuration. The flashing tool inside the container talks to the debug probe on the host as if it were running natively. On macOS and Windows hosts, this is more complicated, because Docker on those platforms runs inside a Linux virtual machine, and USB pass-through to that VM is more limited.

The practical resolution depends on the team’s needs. For pure build and test workflows that do not need hardware access, the limitation does not apply. For developer workflows that require flashing and debugging, the common pattern is to run the build inside the container but the flashing and debugging on the host, with the container producing the firmware binary and the host’s native flashing tools deploying it. For CI workflows, the test rigs are typically Linux machines where USB pass-through works cleanly, so this limitation primarily affects individual developers’ workflows on non-Linux machines.

This is worth being honest about, because teams that adopt containerisation expecting it to solve every environment problem can be frustrated when the hardware access piece requires a workaround. With realistic expectations, the limitation is manageable; the reproducibility benefits for the build environment itself are still large, even if the flashing step happens outside the container.

The SDK distribution superpower

The most underdiscussed benefit of containerised embedded build environments is what they enable for SDK distribution. Most embedded vendors who ship SDKs to external customers spend significant support effort helping customers establish working build environments. The customer downloads the SDK, attempts to follow the installation procedure, hits a version conflict with their existing tools, contacts support, and the SDK provider’s engineers debug the customer’s environment remotely. This support burden is real, expensive, and largely thankless.

Shipping the SDK as a Docker image solves this problem almost entirely. The customer gets a single command — docker pull vendor/sdk:1.4.2 — that downloads a complete, working build environment for the SDK. There is no installation procedure that can fail in customer-specific ways. There are no version conflicts with the customer’s existing tools, because the SDK lives in its own container. The customer’s host system is essentially untouched; everything happens inside the container.

For the SDK provider, this transforms the support relationship. The customer’s environment is now identical to the environment the SDK was developed and tested against, by construction. Bug reports become reproducible because the reporting customer’s environment matches the environment the support engineer can spin up. SDK upgrades become a tag change rather than a multi-step uninstall-and-reinstall procedure. The whole category of “the SDK does not work on my machine” support tickets largely disappears.

This is genuinely a competitive differentiator in markets where SDK quality matters. The vendor whose SDK can be evaluated by a prospective customer in fifteen minutes — docker pull, build the example, see it run — wins evaluations against vendors whose SDKs require a multi-day setup. Customer onboarding accelerates. Support costs drop. The relationship between the SDK provider and its customers becomes more about the substance of the SDK and less about troubleshooting installation procedures.

For any team that ships an SDK, the case for containerised distribution is strong enough that it should probably be the default delivery mechanism. The technical lift to add Docker distribution to an existing SDK is modest. The customer experience improvement is large. The reduction in support burden is substantial and persistent. It is one of the highest-leverage changes a team that ships embedded SDKs can make.

Common objections and how they resolve

A few objections to containerised embedded development come up frequently, and each is worth addressing honestly because the resolution informs how the technique should be adopted.

The first objection is image size. Embedded build environments include large toolchains and vendor SDKs, and the resulting Docker images can be several gigabytes. This is real but manageable. Modern container registries handle large images well, layer caching means only changed layers need to be re-downloaded, and the time to download a multi-gigabyte image is small compared to the time to install the same tools manually. The image-size cost is a one-time cost paid at first download; the reproducibility benefit is paid every day thereafter.

The second objection is build performance. Some teams worry that running the build inside a container adds overhead. In practice, on Linux hosts, the overhead is essentially zero — Docker containers on Linux are not virtual machines and the build runs at native speed. On non-Linux hosts where Docker runs in a VM, there is some overhead from the virtualisation layer, but it is generally small enough not to dominate build times for typical embedded projects.

The third objection is the learning curve. Engineers who are not familiar with Docker need to learn it, and there is a real cost to that learning. The cost is bounded — Docker is not a particularly complex technology once the core concepts are clear — and it is a one-time investment that pays back over the entire career of an embedded developer, since containerisation is increasingly the standard approach to environment management across the industry. Teams that adopt the practice deliberately, with some investment in training and documentation, find that the learning curve is shallow and the benefits become evident quickly.

The fourth objection is editor and IDE integration, particularly for developers using IDEs that do not natively support dev containers. This is a real friction in practice. The mitigation is partly technical — most modern editors do support dev containers, and the integration improves continuously — and partly cultural, with teams sometimes choosing to standardise on editors that support the workflow rather than fighting the integration in less-supported tools.

The right adoption pattern

For a team considering containerisation, the practical question is how to adopt it without disrupting ongoing work. The pattern that tends to work involves three phases.

The first phase is to containerise the CI environment without changing developer workflows. Build the image, configure CI to use it, verify that the CI builds match what developers produce locally. This phase eliminates one source of “works on my machine” failures — those that arise from CI-developer divergence — without requiring developers to change their habits. The CI containerisation alone is a meaningful improvement, and it lays the groundwork for the next phase.

The second phase is to introduce dev containers as an option. Add the .devcontainer configuration, document the workflow, and let developers opt in. New team members are usually the first adopters because they have no existing environment to migrate; they get the dev container from day one and never experience the environment-divergence problem. Existing team members migrate at their own pace, often as their personal environments break and need to be rebuilt anyway. Within a year of introduction, most teams find that dev containers have become the default workflow without requiring a hard migration.

The third phase is to extend the same approach to SDK distribution if the team ships SDKs externally. By this point, the team has confidence in the containerised environment, the image build process is mature, and the leap to publishing the image as the SDK delivery mechanism is straightforward. The SDK customers then experience the same environment-reproducibility benefits the internal team has already realised, with the support cost reductions that follow.

This phased approach minimises disruption and maximises buy-in. The team experiences benefits at each phase before committing to the next. By the time the third phase completes, the team has eliminated environment divergence as a meaningful source of friction across developers, CI, and external customers — and the cumulative effect on productivity, support cost, and customer experience is large enough to make the investment unambiguously worthwhile.

For embedded development specifically, the case for containerisation is strong enough that the question is increasingly not whether to do it but when. The teams that have made the move treat their previous environment-management practices as quaint. The teams that have not yet made the move continue to absorb the daily cost of environment divergence in ways that are largely invisible until the alternative is experienced directly.

needCode designs and delivers the full development infrastructure for embedded wireless products, including containerised build environments, dev container configurations, and Docker-based SDK distribution. We have built these systems for engagements where the same image supported developers, CI, and external SDK customers identically. If your team is still living with environment divergence as a fact of life, we are happy to talk about what changing that would involve.

Book a free discovery call or get in touch

Further reading