Thermal Tech Allows Machines to See in The Dark

A team of researchers from Purdue University has created a novel machine vision system to allow machines to see in the dark.

Leveraging thermal imaging, the new tech is hoped to provide higher perception capabilities for autonomous driving systems and robotics, allowing them to see as clearly in the dark as in the day.

Currently, self-driving vehicles “see” the road and its obstacles using sonar, radar and lidar sensors. These tools work to a high level but have a key limitation in that they struggle to distinguish between the thermal radiation between objects and their environment. 

"Objects and their environment constantly emit and scatter thermal radiation, leading to textureless images famously known as the 'ghosting effect,'" said Fanglin Bao, study co-author. "Thermal pictures of a person's face show only contours and some temperature contrast; there are no features, making it seem like you have seen a ghost. 

“This loss of information, texture and features is a roadblock for machine perception using heat radiation."

With the team’s development of heat-assisted detection and ranging (HADAR) imaging, the researchers were able to create a vision system that can see accurately in low visibility situations, such as in darkness, fog and smoke. 

The team’s HADAR system combines thermal physics, infrared imaging and machine learning capabilities.

"Our work builds the information theoretic foundations of thermal perception to show that pitch darkness carries the same amount of information as broad daylight,” said Zubin Jacob, study co-author. “Evolution has made human beings biased toward the daytime. Machine perception of the future will overcome this long-standing dichotomy between day and night.”

"HADAR vividly recovers the texture from the cluttered heat signal and accurately disentangles temperature, emissivity and texture (TeX) of all objects in a scene,” said Bao. “It sees texture and depth through the darkness as if it were day and also perceives physical attributes beyond visible imaging or conventional thermal sensing."

The team tested its HADAR system using an off-road nighttime scenario, with positive results. In the test, HADAR could distinguish between people, cardboard figures and cars in the scene, while other sensor tech such as lidar could not. 

"HADAR TeX vision recovered textures and overcame the ghosting effect," Bao said. "It recovered fine textures such as water ripples, bark wrinkles and culverts in addition to details about the grassland."

Initial applications for HADAR include automated vehicles and robots that interact with humans in complex environments, though the tech could also be developed for use cases in agriculture, defense, geosciences, health care and wildlife monitoring applications.