Edge Computing, Deep Learning Power Amazon Go

In 2018, Amazon launched its first, public automated bricks-and-mortar store in Seattle. Dubbed Amazon Go, shoppers could walk in, pick up items and leave without going to checkout. An intuitive system of cameras and AI would determine what they purchased and charge it to their card.

The idea was to alleviate customer frustrations from queueing to pay for purchases. Today, Amazon’s "Just Walk Out" concept has been adopted by more stores and even lets customers pay with their palm prints.

At the World Artificial Intelligence Cannes Festival (WAICF), Gérard Medioni, an AWS vice president who helped build the company’s physical store tech, explained how the ‘Just Walk Out’ idea went from an initial concept to being implemented in supermarkets, airports and stadiums.

Upon launching Amazon Go in 2018, Amazon built its own depth cameras designed to capture not only an image but also a depth map for every pixel. Those cameras were then combined with load cell sensors in shelves that are triggered when something is taken and the weight is displaced. The pixel and image data were then stored in the cloud.

But five years on, the technology has come a long way. Medioni outlined several changes, including the decision to drop both depth cameras and load cell sensors. Depth cameras were instead replaced by RGB cameras, which he said offer improved detection, while load cell sensors were seen as outdated due to the introduction of computer vision.

Amazon deploys both deep learning and computer vision in its stores, Medioni explained, to perform the trigger operation by detecting the motion of the shopper.

To power the newest, computer vision-enabled camera store system, Amazon moved its data from the cloud to the edge via processors behind the cameras, allowing stores to perform computations on-premises.

Having initially applied the ‘Just Walk Out’ concept to a convenience store setting in 2018, Amazon has taken its idea in two different directions, Medioni explained. The first is deploying it in supermarkets, as well as in several Whole Foods and Amazon Fresh stores, and the other is offering the Just Walk-Out-as-a-service.

The executive said the as-a-service concept enables other stores and venues such as sports arenas to use ‘Just Walk Out.’ Medioni showcased Lumen Field, home of the NFL team Seattle Seahawks, which has a ‘Just Walk Out’ store built inside the stadium, allowing fans to purchase drinks and snacks in the same way they would in an Amazon Go.

Medioni went on to explain how the company is attempting to take its idea one step further, through Amazon One.

Amazon One lets shoppers to enter its Just Walk Out stores using only the palms of their hands as certification.

He explained that upon entry, shoppers wave their palms above a receiver that takes two images: one of the surfaces of the shopper's palm and another that maps the subsurface vein and artery patterns via infrared technology. The images are then passed through a neural network, compared to feature vectors in a database and once confirmed, are allowed entry into the store.

While currently only available in the U.S., Medioni said consumers enjoyed both Amazon One and ‘Just Walk Out’ due to its “wizard-like” nature. Channeling sci-fi author Arthur C. Clarke, he said “when the technology disappears, the experience is no different from magic.”