Linux Distributions: Considerations for IoT Projects

Linux is the operating system of choice for Internet of Things devices. But when it comes to choosing Linux distributions, developers don’t have much say in the matter. Even worse, the version of Linux they launch their IoT solution with is likely to be the same version that will run in production for months, possibly years down the road.

“The hardware manufacturers pay some attention to what’s popular, and that tends to be Ubuntu,” said Christopher Biggs, founder and director, Accelerando Consulting in Australia. “In general, the offerings from hardware manufacturers are not great in that they are probably offering a distribution that’s a couple years old and they might have a kernel that’s out of date. If it develops a flaw, you may never get an update.”

This differs from what developers are accustomed to. “The PC you bought three or four years ago will run the new version of Windows. In the IoT space, that’s not necessarily the case at all, and it might be that if you used a chip built three or four years ago, then you’re stuck with a version of Linux that’s also three- or four-years-old,” said Christian Daudt, senior member technical staff engineer at Cypress Semiconductor Corp.

The problem is that hardware vendors aren’t providing updateable devices. “The low margins and highly competitive environment make this an unacceptable cost for most,” said Biggs.

The cost for companies building an IoT solution with these outdated chips is another thing altogether. “The biggest risk of a device that is not receiving updates is that an exploitable vulnerability is found,” explained Biggs. If there isn’t a vendor fix, “there is no defense for owners of affected devices, short of cutting them off from the internet.”

Another potential problem: “The kernel can be so old, it doesn’t work with the current version of Docker,” said Biggs.

Linux distributions — or the vendor’s release cycle — aren’t typically considerations when developers choose the hardware for an IoT solution. Instead, the hardware choice is driven by factors like the specific use case and amount of memory required to support it. “Hardware can force you into a distribution because the hardware vendors provide the distribution,” explained Andrey Katsman, vice president of engineering at Canary.

What then? “The simple way, the path of least resistance is to stick with [the operating system] the hardware vendor gives you, but you can be stuck with bugs and restrictions. It’s a trade-off calculation between will you spend the time upgrading it yourself or will you pick a different piece of hardware that’s more up to date?” said Daudt.

There is a cost to consider if the organization decides to upgrade the OS itself.

“Software is usually the most costly part of these endeavors,” said Daudt. As an example, he explained that smartphones are complex, so manufacturers rely on the chip vendors to manage that complexity and optimize the phone for all the different aspects of the chip. “They can’t do that and have a fast turnaround on new phones if they have to write a significant part of the software stack,” he said.

“It’s a major effort to upgrade the versions because a lot of software goes into it. That’s why hardware vendors don’t do it. They don’t want to invest more and more if it’s not the next-generation device,” Daudt said.

If the development organization decides to update the OS, those distribution choices will once again be limited to the use case. The more specialized the use case, the fewer the OS options the organization has. “If a button is the simplest device and the PC is the most advanced, depending on where you’re at will dictate the variety of choices available. A PC will be almost anything. A button will be bare metal and have some infinite loop that dispatches an event,” said Katsman.

Organizations commonly have a choice between two distributions. “Manufacturers tend to support one Yocto-based and one Debian-based distribution,” said Daudt.

He recommends choosing the distribution based on the amount of customization required. “The trade-off is how much you can use out of the box versus how much you have to modify. There’s a tipping point where one becomes more interesting than the other,” said Daudt. “Binary is easier if you don’t have to make a lot of changes, but if you have to do a lot of customization then the source-based one like Yocto becomes the better alternative.”

Ultimately, however, experts recommend that developer organizations make the operating system a consideration early on in the hardware evaluation. “Try to understand how much you’re going to have to change the software from what you get from the vendor and based on that, see which options you have within a vendor’s software offerings in terms of their software evolution and how important that is for you or not,” said Daudt. “If you think you’ll stick to 99% of what you have [from the operating system], and not support field upgrades, then all these things are moot from your perspective. If you want to send upgrades every three months to your devices over the course of four or five years, then this is an absolutely important factor for you to determine how to pick your hardware and how the vendor supports the software evolution themselves.”