Amazon’s till-less grocery store recently opened in London, in March 2021- the first “Just Walk Out” store outside of the United States. The store operates in the same way as the other 20 ‘Amazon Go’ stores in the US – you enter the store, pick up the items you wish to purchase, and just walk out. The technology used to make this possible is stated to be the same as that used in self-driving autos, primarily through Artificial Intelligence. The debut store in the UK made its appearance at Ealing, West London – branded as ‘Amazon Fresh’ rather than ‘Amazon Go’ as seen in the US.
We at the Centre for Finance, Technology and Entrepreneurship (CFTE), have always kept pace with tech developments in the industry as we teach digital finance to professionals and enthusiasts through courses on emerging technologies like embedded finance and invisible finance. We’re always excited to learn about new real-world applications of topics we teach around us and our team never misses an opportunity to do so!
Once we saw the Amazon Fresh store – the epitome of embedded finance and invisible finance – had popped up right in front of our office, we knew we had to try it ourselves first hand, and maybe even pull off a few tricks to see if we could break the system!
How Does Amazon Fresh Work?
Before getting right into our shenanigans to game the Amazon Fresh system, it would be worthwhile to try and understand how does the system even work?
As we made our way to the Amazon Fresh storefront we couldn’t help our mind from wandering into the various possibilities that made the “Just Walk Out” store a reality. Was it cameras behind display cases or on the ceiling? Were they using technology like sensors to gauge proximity or weight? How does the system process all this information? Our questions were soon to be answered.
As we got into the entrance of the Amazon Fresh store at Ealing, we needed to scan a QR code on our Amazon app downloaded on our smartphone. It’s just one simple scan and you’re in! As you make the scan, you are also being timed for your in-store session (this is revealed at the time you get your receipt).
So once you’re in, how are your movements and purchases monitored? It’s through dozens and dozens of cameras mounted on the ceiling covering every square inch of the store through perfectly positioned cameras at every angle.
The shelves also have weight sensors embedded in them and the system is aware of every item’s exact weight to identify when what item is picked up.
And no, there wasn’t any facial recognition mechanism – probably a good strategy to avoid driving away privacy-conscious shoppers.
These cameras and sensors are developed with software that uses deep learning artificial intelligence techniques to help determine the shopping process.
According to Dr. Gerard Medioni, Chief Scientist and former USC Computer Vision Professor – the cameras and sensors in the store are solving 6 major problems through Artificial intelligence:
- Sensor Fusion – The process of aggregating signals and vision across computers
- Calibration – Each camera understanding accurately its placement in the store
- Person Detection – The capability to identify and track every individual in the store
- Object Recognition – The ability to differentiate between different items being sold in the store
- Activity Analysis – Determining the intention of individual activity in terms of whether they are picking something up or returning it.
Finally, after gathering all the inputs from the camera and sensors, a central system analyses these movements to determine purchases that have been made so you can be charged for them on your Amazon account. This does not happen immediately and may take an hour or two to process.
Could We Break The Amazon Fresh Tech?
Starstruck by all the technology in the store we knew it may all be too good to be true. We were sure there was a way to get around these high-tech robotic machines and we wanted to test our theory. Here are some lessons we learned from our antics to prove the system was not immaculate:
- Help yourself, not others!
As I grabbed a sandwich, I did so for my friend too and gave it to him to store in his backpack, only to realise an hour later that I had been charged for it! We realised that being charged for an item was not based on who walks out with it, but rather who took something off the shelf. So yes, helping someone to grab something off a shelf is not a good idea, even if you’re helping an old woman – unless you wish to pay for everyone’s purchases!
- Tricking the cameras and sensors will backlash!
To see if Amazon Fresh’s camera and sensor technology was really as smart as they claim it was, we tried to play around with items we picked up. We juggled between picking up and placing back different items. We switched the position of a sandwich and a drink around the store and then left them there. What we realised this has done was make the system believe that we had actually purchased those items, because if you pick something up and do not place it back in the same place, the system does not recognise what you just did, resulting in you making a purchase for something you just switched up!
- Of Course Identical Twins can game the system, right?
I have an identical twin myself and having been able to trick Apple’s face recognition a trillion times, we were sure this would be a piece of cake too. We were dressed similarly in the store and we shuffled between each other as we tried to make a purchase – we thought this would baffle the cameras – just as it had all of our friends and family all these years. But, unfortunately, or rather, fortunately, Amazon’s cameras and sensors do amazingly well even with crowds around and identical twins in the same outfit scrambling the store in different locations! There were no mix-ups in our purchases.
- Shoplifting at Amazon Fresh – is it really possible?
We didn’t try to do this ourselves of course, but we know of a case where it actually happened. Ars Technica reporter, Sam Machkovech was hell-bent on finding a way to break the system and he took things to a whole new level. What he figured out was that Amazon’s cameras keep a close eye on customers while they shop, but fortunately not while they’re using the restroom. Machkovech used this to his advantage as he picked up some merchandise, headed to the restroom for a quick wardrobe change. Once changed, he then picked up some more items before he made his way out. The result? The cameras couldn’t figure out who Machkovech was after the change of clothes and appearance, and he wasn’t charged for the items picked up after he changed.
Amazon does not really care for the minority of the population that go to extreme lengths to shoplift. An Amazon spokesperson said, “We spend the vast majority of our time thinking about improving the experience for the 99.9% of well-intentioned shoppers vs. designing a system that is focused on deterring the few bad actors.”
After spending over an hour at the store finding ways to get around its technology, we were all actually quite impressed at the seamlessness of the system. There is no doubt that it is a clever system, and the areas where it may fall short are not quite alarming, though there could be scope to get the system to a 99.9% accuracy.
Is Amazon Fresh Technology Set To Become The Future Of Grocery Retail?
After testing the system and technology for ourselves and realising that it was truly ‘convenient’ and quick (only if you aren’t spending hours figuring out how to game the system), we wonder if it could change the grocery retail shopping industry forever.
Amazon’s highly tech-driven brick-and-mortar stores offering a frictionless experience at Amazon Go in the US and now Amazon Fresh in the UK is limited to smaller convenience-type stores. If this technology can successfully be ramped up and scaled, it could surely lead to widespread cashier-less shopping that could revolutionise the shopping experience. Amazon’s moves in the space are definitely urging other companies to adopt more digital solutions in-store, but Amazon Go and Fresh’s launch isn’t proving to be an immediate threat to established grocers – as they have time on their hands to develop their own check-out free tech. In fact, Amazon is looking to get them on their side.
Amazon is not taking the path to compete with established supermarkets with bigger stores of its own. The company is now offering to sell its “Just Walk Out” technology to other retailers – as possibly, that’s where more money could be made.
While this technology is great and could save a lot of time for customers and redefine ‘convenience’, some argue that tech is being applied today where it is not needed – by siphoning large amounts of ingenuity and resources into something that people don’t care about, or even consider as a problem. This is definitely debatable, and understanding if and how consumers catch on to this trend of frictionless and cashier-less shopping would be vital for organisations to plan their next move in the industry.
