Want Better IoT Software Testing? Test in the Real World

By Valerie Silverthorne

The struggle is real for software testers, particularly in the IoT space.

At a time when the IoT market is expanding rapidly — Bain & Company predicts it will be worth $520 billion by 2021 — the pace of development has never been faster while the pressure to get it right, and in production, is also intense. That means testing, always a potential stumbling block in any development environment, is under more scrutiny than ever in the IoT space today.

Simply put, IoT software testing is difficult, said Arthur Hicken, evangelist with software testing tools maker Parasoft. “IoT represents a double challenge to testers,” Hicken said. “Embedded software is always tricky to test … and then you have safety (concerns), hardware that has to perform in real time, memory constraints, and it’s often all done in (the programming language) C. And you’re not just testing the device itself; you have to stand up the entire ecosystem and test that, so there’s a big chance a lot of things won’t be fully tested.”

The solution, Hicken suggested, is for IoT testers to approach the problem the way a hardware designer would. “Hardware companies in general have tended to behave better (when it comes to testing),” he said. “They think more about testing because they’re testing critical things like cars or planes. They look at the bigger picture.”

It’s time for IoT testers to, literally, go outside. “They have to go and test in the real world,” said Jeffrey Hammond, vice president and principal analyst at market research firm Forrester Research. Without a real-world vetting, even the best IoT project can fall apart, he said, and pointed to a company that put a beacon near a bridge not understanding that every time the bridge went up communication would stop completely.

The Case for Testing Early

Early IoT software testing “in the wild” would no doubt have picked up on the bridge problem quickly, which is why experts argue the sooner an IoT project is tested out in the real world, the better. In a Forrester report, “IoT Delivery Best Practice: Perform Real-World Testing,” Hammond and his colleagues explained how early testing in the place a device will be used can result in vital feedback that’s in time to make changes to the software, the hardware and even the customer’s expectations. And prompt testing of a product in its destined location will quickly spotlight challenges around installation, deployment or updating.

While it’s possible to make the case that all IoT software testing should be done early and on site, the potential of IoT devices to generate insane amounts of data makes the case for that type of testing even stronger. Real-world testing will show external problems — like electromagnetic interference — but it will also make it far easier to see and understand the impact of data transmission, the Forrester Report argued. Testers and developers will be able to see how networks slow down or crash, the impact on performance and the potential costs involved. Find this out early and decisions to put computing power on the edge versus the cloud become much easier to make and may not require a project get scrapped.

Tester as Guardian

Moving testing out into the real world will also mean that the role of tester must change, Hammond said. Testers will be on hand to see exactly what works, and what doesn’t, and will perhaps for the first time have an insight into customer needs and expectations that’s greater than the developer’s. With that is going to come pressure, Hammond warned. “This is the time to be skeptical about the use case,” he said. “You need to make sure that the use case is providing real value.” IoT testers should have the opportunity to weigh in on every part of the process, he said, including things like how updates are going to be addressed.

And it won’t stop there, Hicken added. The complex nature of IoT is going to put pressure on testers to become more technical because otherwise they simply won’t be able to do the job, he said. IoT testers will need to be able to at least read code and have a deep bench of usability and user experience skills. These so-called hybrid testers are going to be the backbone of IoT moving forward. “In this environment having a hybrid tester is super critical,” he said.

Setting Expectations and Standards

But even the best hybrid testers can’t catch every single error, Hammond said. Companies should test in the real world, and upskill their testers, but they should also be prepared for the process to be time-consuming, costly and not bulletproof, he said. His best advice: Build in a “debug mode” that will allow testers to capture anomalies over a period of time, as that is sometimes the only way to ensure thorough testing.

There may also be some light at the end of the tunnel from standards organizations finally ready to consider software, Hicken said. Certain industries like aerospace and health care have had robust standards and expectations around testing for years, but largely on the hardware side, he said. What IoT software development needs is more standards bodies outlining the scope of software testing. And he thinks UL2900 might be the right place to begin. “I think UL has made electrical devices easy to test in an objective manner,” he said. “Software gets a lot trickier. But the organization has been trying to craft a standard and is going through a revision to tighten it and make it cleaner. It’s a challenge — how do you know what’s in a device and how do you apply appropriate tests?”

The standards bodies aren’t there yet, Hicken said, but it is coming. “What you’re seeing is that software is being put into a more constrained box that is very normal for what the hardware people are used to. We’re now applying those principles to software from development to engineering but we’re not there yet,” he said. “We’re still waiting for agreement on a set of standards.”