Using IoT Augmented Reality to Enhance Workplace Operations

Enterprises now use the Internet of Things, along with augmented reality, virtual reality, and video capabilities to operate more efficiently.

Digital twins, or virtual replicas of objects, buildings or systems connected to their physical counterparts through real-time information feeds, are becoming cornerstones for design, planning and operations from the products consumers use to the cities and buildings they live in, said Sandra Marshall, chief technologist, strategic innovation group at Booz Allen Hamilton.

“The applications are limitless, from monitoring energy consumption and security in real time to helping hospitals analyze the potential of contagion spread through a facility, to enabling stakeholders to ‘walk the halls’ of their future building prior to construction,” she said.

Given the decreasing costs of augmented reality (AR))/virtual reality (VR) technology and the rise of real-time data and Internet of Things, digital twins will become more prevalent, Marshall said. In fact, the market is predicted to grow from $5.1 billion in 2020 to $115 billion by 2035.

When integrated with Internet of Things (IoT), digital twins can provide value across the lifecycle of a product or process by creating a digital thread of data, performance and decisions that bring transparency and enhanced communication to stakeholders across an enterprise, according to Marshall.

Digital Twins and IoT for Robotics

“You can build a digital twin of objects and robots, such as a robotic patrol dog, to model and simulate performance prior to manufacturing or deployment,” she said. “As the robot is manufactured and outfitted with sensors, you can track performance in test runs and tweak software to improve performance.”

Tyndall Air Force Base in Florida was one of the first military bases to implement the robot patrol dog to enhance security operations. The team drives the robots via a virtual reality headset within the base’s defense operations center and sees what the robot dog detects through its mobile camera and sensor platform.

“When the robot is operationally deployed, the digital twin of the robot can be used to monitor performance and run simulations to predict failure and prevent downtime,” Marshall said.

Once deployed, AR can place service instructions, data and knowledge directly at the fingertips of the user, while directly connecting them to experts for remote help, Marshall said.

A digital twin can then be used to support product owners, users and servicemen in virtual reality training, so they can practice using and servicing the robots in any environment prior to deployment, she said.

Digital Twins and IoT for Design and Construction

Designers can use virtual reality to undertake virtual walkthroughs of a space with stakeholders from across the globe before starting construction, according to Marshall.

“And upon entering construction, engineers can use the digital twin combined with IoT to perform tasks such as remote monitoring of construction sites,” she said. “AI object recognition can be running on cameras and sending alerts to security personnel if a vehicle with unauthorized access drives onsite.”

Through construction and operations, engineers, electricians and plumbers can use the digital twin and augmented reality to “see” wiring and plumbing behind walls for faster field service, she said.

Having the digital twin itself can also enable a multitude of augmented reality applications from AR navigation through the facility to localized, real-time information and alerts, Marshall said.

“As the building is built and embedded with sensors, you can move into the operational phase of the digital twin, where facility managers can see all of their sensor data in one place, contextualized and visualized to enable insights and understanding,” she said.

The ability to see the data, such as temperature, humidity, occupancy, camera feeds and air quality sensors, all contextualized through sensor fusion enabled by IoT, will provide greater understanding and situational awareness, according to Marshall.

Smart Cameras and IoT

One interesting application of smart cameras and IoT is bin monitoring – monitoring what goes into a trash bin or recyclables bin, said Dan Shey, vice president of enabling platforms at ABI Research.

Some McDonald’s restaurants, for example, have put trash-monitoring smart cameras in dumpsters and trash containers to recycle more efficiently.

The system uses cameras and sensors that take numerous pictures each day. AI software analyzes the photos and alerts McDonald’s on-site employees if something isn’t in the right place, such as a bag full of trash in a recycling bin, so they can remove the trash before the recycling truck comes.

In addition, smart AI-based cameras can enable such traffic management applications as adaptive traffic lights, parking access and detection and electronic tolling. Shipments of smart AI-based intelligent transportation system cameras are expected to grow from around 33,000 in 2020 to more than 155,000 in 2025, according to ABI Research.

“[Take] smart parking — instead of having a sensor in the parking spot that would sense a vehicle when it drove in and when it left, you have a video camera with a sensor that’s posted on a pole or something in a parking lot,” Shey said. “Then you put some special algorithms on top of that to detect whether that spot is filled or not.”

IoT lets these devices transmit data back to a server. The server then applies these algorithms to detect parking spots with cars and spots without cars, he said.

“But you have to have something in the artificial intelligence realm to be able to do that,” Shey said. “It’s no different than with facial recognition. There has to be some artificial intelligence to detect the presence of an object [e.g., a vehicle) through that video feed.”

This information then goes back to the supplier of the parking lot, for example, which takes the information and uses it for its smart parking application, which users have downloaded to their mobile devices.

“Say you come into New York City and you want to find a parking spot and so this app says, ‘Oh, there’s one right here.’ It knows that there’s an open spot because of the sensor in the video camera,” Shey said.

AR/VR and Remote Workers

Augmented reality, the user interface for IoT, and virtual reality create significant use cases for workers on the frontline, according to Ian Hughes, senior research analyst at 451 Research.

AR, for example, can enable a frontline engineer to speak with a remote expert who can annotate and guide them through a process that they may have previously worked on together, according to the 451 report “How IoT will Support New Productivity Models after COVID-19,” co-authored by Hughes.

“[AR] is also used to record in place detailed 3D work instructions to allow the engineer to operate independently, with many systems utilizing AI techniques to ensure they receive the correct data at the right time, or can query and adjust,” the report noted.

“An engineer is told to go fix a pump and he’s standing in front of the pump,” Hughes said. “The very basic version is that they point their tablet or phone at that machine and the machine will be recognized and they can see this live information, maybe how hot something is or which bolt to remove first, on their [mobile device]. Its contextual information laid over the physical object that they’re engaging with, with AR.”

But the engineer could also engage with a remote expert who can see what the engineer can see, Hughes said. And that remote expert will annotate and draw on their screen.

“They undo this, switch this bit off first, change this part, which is then locked into the view that the engineer has of the world,” he said. “It’s like somebody standing next to you, drawing and pointing at something, but they’re doing it remotely.”

Gautam Goswami, chief marketing officer, TeamViewer, said his company has a new series of products, TeamViewer Frontline, that, in part, enable warehouse order pickers to carry out vision picking.

“The workers put on smart glasses and as they walk around the warehouse, they automatically receive step-by-step instructions that appear in front of their glasses about what they need to pick up next,” he said. “The instructions also tell them where to put each of the pieces they’ve picked up.”

Goswami said that enterprise TeamViewer customers use this combination of augmented reality and IoT vision-based picking technology.

Virtual reality can be used for training, gaining situational awareness of new locations, developing muscle memory and practicing maintenance and engineering tasks, according to the 451 report.

The point of VR is to remove the physical world, the point of AR is to engage with the physical world, Hughes said.

VR enables users to either practice or experience something that they wouldn’t ordinarily be able to experience, or to be able to continually repeat a task until they have committed that specific task into memory through repetition, he said.

“It might be that something so dangerous that you don’t want to expose somebody to it, but you want them to know what to do in an emergency, such as a fire,” Hughes said. “If you go into a VR model of a building and walk around it, you can see the fire escape, and you will remember where it is when you go to the real building. There are also retail and food places that use VR to train people how to flip burgers and things like that.”

Finally, the key for organizations using augmented reality, virtual reality and video capabilities in their day-to-day operations is integrating those technologies with IoT along with their AI analysis systems.