Feature

Checking Out: Amazon, Microsoft, and the Future of Automated Grocery

Some of the region’s biggest tech players are racing to rewrite how America shops—again.

By Stefan Milne Illustrations by Koren Shadmi July 23, 2019 Published in the August 2019 issue of Seattle Met

A tube, somewhere between vacuum nozzle and robotic pig snout, approaches an apple in an orchard somewhere in Central Washington. The tube inhales the ripe fruit from the branch. It draws back, searches. It pecks another fruit from the tree. Quickly: another, another. The machine rolls down the orchard row. Perhaps, then, the fruit is sent to Crunch Pak, the biggest sliced-apple processor in the U.S. There, in Cashmere, Washington, it’s cored and spun in a ruddy blur, sliced, sprayed with ascorbic acid and calcium carbonate, and portioned into plastic bags. 

Some days later, apples arrive at Amazon Go in downtown Seattle. The automated store sits across an artificial lawn from the Spheres, and if its futurism is subtler than those glassy biodomes brimming with exotic plants, it’s no less dislocating. To enter, you touch the Go app, scan a QR code at the turnstile, and pass through the opening gates. On the ceiling: a cosmos of cameras in black boxes. They note that you are not the person to your right, who, as it happens, wears the same North Face fleece and Mariners cap as you. In demos of similar systems, neon lines trace people’s skeletons in jittering stick figures and shapes—circles, rectangles—enclose bodies like embryonic membranes.

In the refrigerator case sit those Crunch Pak organic apple slices. “Wholesome and Crunchy.... Saves you precious time, too!” You take the bag. Cameras and shelf sensors note this. At the turnstile, gates part. When you’re back at your office desk, two hours and seven minutes later, apples long since eaten, the receipt lands in your email. One item, $3.99. You were in the store, as your receipt notes, only “41s.”

The scenario is hypothetical (but barely), a composite of existing tech. A new wave of food system automation—AI from farm to aisle—is here. This year, that robo-picker, made by Abundant Robotics, will participate in a Washington apple harvest for the first time. Robots now milk cows and flip hamburgers. Governor Jay Inslee signed a law in April that allows self-driving delivery pods to zoom down our sidewalks. For many people, though, changes will be most visible in stores. An IBM report issued in January said that by 2021 store automation will jump from around 30 percent of retailers to over 70 percent. Grocery is on the vanguard of that shift, and it’s found a nexus here in Amazon and Microsoft, two companies with the robust AI research and cloud capabilities to siphon the experience of virtual shopping into physical spaces. 

Image: Koren Shadmi

In late 2016 Amazon charged into grocery. It announced Amazon Go, introduced Amazon Fresh months later, bought Whole Foods, and spurred a race to automate, to remove “friction” from physical shopping. Early this year, Microsoft and Kroger (which owns local chains like QFC and Fred Meyer) announced a partnership to “redefine grocery retail.” Together they launched a pair of high-tech grocery test stores near their respective headquarters—in Redmond, Washington, and Monroe, Ohio.

For over a century now, this region has helped revise how America shops. We were, long ago, home to a forbearer of self-service grocery. We’re still home to Costco—that synonym of biggest box shopping, now the fourth largest retailer in the nation, according to the National Retail Federation’s 2018 rankings. Kroger is second. Walmart, another Microsoft partner, looms over all. But the list’s third company, Amazon, has done more than any other to define and redefine shopping. One of its most disruptive recent changes, that Go store, began seven years ago over lunch.

• • • •  

After 16 years in tech, five of them at Microsoft, then eight at Amazon, moving between various managerial positions, Gianna Puerini was done. She wanted to do something physical. So in 2011 she quit her vice president job and became a general contractor, building houses. She’d gotten through two projects when, in 2012, an Amazon executive called. He wanted to have lunch. She figured it was a job offer.

“I thought there’s probably a 5 percent chance I’m going to entertain this idea because now I use power tools… I had a table saw,” she said at the 2017 Lesbians Who Tech conference in San Francisco (Amazon declined any interviews). Recounting the story on stage, Puerini sat forward in her chair, looking like a cyberpunk gone corporate—a pierced eyebrow, a short sweep of silver-touched hair, a leather motorcycle jacket. And she exuded, with her refined charisma and lightning grin, the same qualities as the stores she helped create. Quick, engaging, cutting edge.

Puerini went to the lunch. But when she heard the company wanted to experiment with physical retail, “my jaw kind of dropped,” she said. “I immediately decided I would come back.”

At first she was the only employee. And since Amazon works “backwards”—coming up with what she termed “basically a fake press release” and then making its contents possible—she shopped for an idea.

She went to stores, pondering the experience, and came away with a question. What if you could get rid of lines?

The idea’s simple: You take, therefore you’re charged. Practically, it was anything but: The existing tech—AI, machine learning, sensor fusion—couldn’t yet solve fundamental problems. Say, for instance, Amazon wants to watch you and I shop at the same time. Well, what if we look alike? What if two different products look alike—say dark and milk chocolate bars—and by picking one up, you cover the only visual difference? What if you’re standing in front of the dark chocolate, and I’m in front of the milk chocolate, and our arms cross as we grab? Will you be charged for my dark chocolate? I took two.

Amazon’s solutions are proprietary, but four and a half years later, in December 2016, the company debuted its first Go store on its Seattle campus to employees only, announcing that it would likely open to the public early the next year. The following March, though, Amazon wasn’t ready. If more than 20 customers shopped at once, or if a family came in under the same account, it could flummox the store. The staff continued to tinker. By November, Amazon teased again. The company had dressed three of its employees in Pikachu costumes and set them shopping, to see if they confused the system. Now Go had it right; the Pokémon were duly charged. And on January 22, 2018, over five years after Puerini’s return, Amazon Go opened to the public. In everyone’s favorite irony, the first crowds waited in a long queue to enter, in front of windows that declared, “No Lines. No Checkout.” The lines soon dispersed. Other effects lingered.

• • • •  

“Just Walk Out technology, right? Is this groundbreaking?” asked James Corden on The Late Late Show. “Like, that’s how I shopped all through my teenage years.” The Atlantic and Forbes concurred—this feels like shoplifting. Even Puerini mentioned that Go is “very similar and natural to how you shop today, except for the part about, you know, stealing.” The thrill of sanctioned theft quickly fades, but the actions we associate with transaction—a total, cash or credit, a signature, even an online click—also left with the lines.

This mediates “the pain of payment,” says Akshay Rao, a professor at the University of Minnesota who studies branding and marketing. Paying with cash hurts, counting bills, handing them over. So we spend less. Credit cards defer that hurt. “And if you don’t actually see the mechanics of the transaction,” Rao says, “that makes it even less painful.” Go’s receipt delay—my email usually arrives two hours later—heightens the effect: You disassociate what you spend from what you get.

The store contains another psychological twist. You can, as Puerini said, “take stuff as if it were your own pantry.” You’re entitled, because you were permitted through the gates, because it’s exclusive, a word that garnishes the store, above endcaps of Go-branded Theo sea-salted dark chocolate bars and Macrina Bakery ginger molasses cookies (both of which are widely available in Seattle—so only the “Exclusives” labeling is, in fact, exclusive). Amazon has created a retail gated community. To enter, initially, you needed a phone, an account, a credit card.

After Go debuted and began to propagate (there are now 13 total, spread between Seattle, Chicago, San Francisco, and Manhattan), Amazon became a target in a pushback against cashless stores. How, critics asked, could they not accept actual money—especially given that 6.5 percent of American households in 2017 didn’t have bank accounts? By March 2019, Philadelphia, Massachusetts, and New Jersey all had bans on cashless stores. Chicago and New York pondered bans. In May, San Francisco passed one, and Amazon gave in. Its first Manhattan location accepted cash; the rest will follow (on an unspecified time line). Checkout is back, and the store of the future has taken a step into the past.

Image: Koren Shadmi

In the 1910s, much as today, Seattle was flush with farmers markets. Stands at the downtown Westlake Public Market hawked fish, oysters, pickles, eggs, hundreds of crates of peaches. Carnation Butter Store advertised its milk with a five-week-old baby bear, reared on the brand’s dairy, crouched on a display of cans. In 1915, brothers Alvin and Walter Monson—a pair of entrepreneurs from Nebraska—surveyed downtown Seattle. Noticing Westlake’s vim, they signed a lease across the street (now home to a Gap).

Back then, grocery stores were run like delis: You gave your list to the counter clerk, who gathered your goods. The Monsons had run such a store, but they had a new idea. “Help Yourself and Pay As You Go Out,” wrote The Seattle Daily Times in November 1915 of the brothers’ new store, Groceteria. The rest of the article reads like a precursor to Amazon Go write ups: “Clerks are eliminated… Three persons can be waited on in a groceteria while one is being attended to in the average grocery store.” The Monsons claimed prices 12.5 percent lower than their competitors’. The Times wrote that Groceteria was the fourth such business in the country.

A few years ago, Rob Ketcherside—an engineering manager for Dell, who’s also published a history book, Lost Seattle—was researching local grocery chains. “I hit Groceteria, which had like 20 [stores]… and that was just baffling to me.” How could a chain have grown that large back then, and how had he not heard of it?

Look up the history of self-serve grocery and Tennessee-based Piggly Wiggly gets credit, since it took the model national. But by the time the first Piggly Wiggly opened in September of 1916, Groceteria had 10 stores in the Seattle area. A year later there were 30.

Yet as the company seemed poised to make history, a larger history intervened. Alvin Monson, the savvy innovative sibling, was drafted into World War I, and in France, his company’s train collided with another. More than 30 died. Alvin wandered the scene in shock; he helped put a decapitated corpse on a stretcher. Eventually he returned to Seattle and was diagnosed with post-traumatic psychoneurosis. As other chains grew, Groceteria dwindled, finally going bankrupt in 1926.

On that self-service model, though, coupled with advances in food processing and refrigeration, the American supermarket rose to capitalist icon. In 1989 when Boris Yeltsin, then a congress member in the Soviet Union, visited a Houston grocery store, he was awed at our glut of choice—the average supermarket then carried around 7,000 items; today it carries over 30,000. After that trip, an aide said “the last vestige of Bolshevism collapsed” in Yeltsin. He left the Communist Party a year later.

Grocery should be a hotbed for innovation. The margins are slim, 1.3 percent on average, and the shopping habitual, once or twice a week generally. The 1970s brought bar codes and scanners, then thought to be harbingers of greater automation. “Checkstand girls,” wrote the Times in 1976, “will be a thing of the past like dodo birds and elevator girls.” And developments have continued: self-checkout, new methods of delivery, member cards to glean data.

Yet step into the standard grocery store today—as many retail tech innovators quickly mention—and it looks a lot like the model Groceteria helped pioneer. These innovators are also quick to mention that’s about to change.

• • • •  

The Kroger QFC looks, at first, like any other suburban grocery store. Here in Redmond, 15 miles east of Seattle, are rows of diagonally parked cars. Here’s a Starbucks, where a pair of old men share a Seattle Times, snapping their respective sections into manageable squares. Here are carts and cashiers and employees grooming piled produce. If Amazon Go offers an experience of subtraction—no lines, no checkout—you’ll find its antonym here, just past Microsoft’s campus: an interactive store. While conventional supermarkets fall under casino logic—get you a little lost so you spend time and money—Kroger now gets in your phone and guides you down the aisles.

To begin, I made a list in the QFC app and downloaded a second app called Scan, Bag, Go. (Or you can grab a handheld scanner with the heft and big rubbery buttons of an old school remote.) These predate the Microsoft partnership and are already in hundreds of Kroger stores. New was the small map on my screen, leading me to my first item, a bag of Kettle chips. In the center aisles and endcaps, store shelves have LED displays instead of paper tags (part of Kroger’s mission to cut waste and energy use, since store lights can run lower). Occasionally the displays turn to ads. “No Preservatives. No Artificial Colors,” a Chef Boyardee shelf strobed at me. Mostly, though, the labels of items on my list blinked with a personalized symbol—for me, a halved avocado. I grabbed the chips, scanned them with my phone’s camera, continued.

Then the problems started. In the mayo section, with its 60 or so types, my avocado stopped appearing. And though the path through the store started logically, proceeding numerically down the aisles, soon it scattered: aisles, dairy, aisles, produce. On one trip, the map just said “Lemons,” leaving me to surmise where. If you carry a basket or bag (one hand) and your phone and scanner (other hand), you have to constantly put one down to grab things.

Yet Kroger and Microsoft take such qualms in stride. Rather than weeding problems out in a controlled environment, says Greg Jones, a Microsoft director of business strategies and solutions, the companies wanted to toss the system into real stores “to see what the customers think about this.” 

Truly: On the ceilings, cameras using Microsoft AI check that shelves are stocked, detect spills, and track general traffic patterns. Dime-size camera lenses in endcap displays use facial recognition to sense demographics—age, sex—and basic emotional response. So based on my trips, I may now be a tally mark in “31-year-old males who smile at Kettle Chips.” Kroger can also market in real-time, with personal precision, flicking coupons to phones.

To pay, you can scan a code at self-checkout or charge your credit card through the app. On the way out, I showed the clerk my phone’s seven-digit receipt code, and he typed in the number. He looked at my cotton bag, plump with product. “Everything in there?” he asked.

“Yup.”

Then, I think, we silently acknowledged the absurdity: He didn’t know what I had, beyond my bag’s general swell. But he waved me through. What could he do? Perhaps Kroger hopes a warning will work. Here and at other QFCs, screens now hang from the ceiling, showing you a feed of yourself and a message: Recording in Progress.

• • • • 

“This recalls that scene from Minority Report, right?” Mike Katell says of surveillance supermarkets and real- time advertising. He’s a University of Washington PhD candidate studying information and data ethics. In Steven Spielberg’s 2002 movie, as Tom Cruise’s character struts down a corridor, cameras scan his eyeballs and personalized holograms assail him: Lexus, Guinness, American Express.

“All this sounds great if you’re a marketer,” Katell says, “but it is potentially corrosive socially.” Say, for instance, you’re a recently sober alcoholic. A guided shopping experience, with AI noting a lapse in purchase, could hit you with a coupon right when you’re most vulnerable. Maybe retailers can account for that—no real-time alcohol marketing. But what about something, Katell notes, like an eating disorder?

As facial recognition and biometric sensing advances—if it can sense my pupils’ subtle dilation as I behold those chips and tweak its advertising accordingly—at what point are retailers more aware of our internal lives than we are? Are we comfortable with this, especially when, as at that QFC, there’s no clear indication our emotions are being tracked and aggregated?

Akshay Rao, the UMN professor, readily acknowledges the privacy problem. Stores can also craft an intensely personalized shopping experience, he says, using this unprecedented data to deepen “consumer intimacy.” If the store does its job right, truly knowing you better than you know yourself, eventually brand and customer can end up “sort of married to each other.” 

Image: Koren Shadmi

Of course to get that ring on the customer’s finger, retailers will have to get people using their technology. I have yet to see another customer use the generously advertised scanners—either in Redmond or my neighborhood QFC. In Google and Apple stores the Scan, Bag, Go’s ratings hover at two out of five stars.

Microsoft, though, has already been at work on other solutions. In February 2018, Microsoft Technology Licensing published a patent for a shopping cart that would track everything you put inside with an array of sensors, such as computer vision and scales. It’d provide “real-time...personalized recommendations” possibly via a tablet on the handle. When I asked Greg Jones about a cart, he just smiled and said, “I can’t share anything about that.”

Such technology is not a distant dream. A Brooklyn startup called Caper has a similar cart, already in a few stores. Currently, customers scan items’ bar codes as they place them in the basket. As they do so, the cart watches and once it has enough data, it automates fully. People check out and receive tailored recommendations as they shop on a touch screen on the handle. Cofounder Lindon Gao says it will “probably be out with one of the major retailers” by August.

The company is one among many. Venture capital has gushed into brick-and-mortar automation. Along with Caper, there’s Standard Cognition, Mighty AI, Zippin, Inokyo, Aipoly, Trigo, Grabango, AVA Retail, AiFi. And if this is truly a race, the leader might not be as clear as it seems.

• • • •  

Two months after Amazon Go opened to the public, Gianna Puerini and Amazon Go vice president Dilip Kumar strolled onto stage, to the “hey, ho, let’s go!” of the Ramones’ “Blitzkrieg Bop,” and gave a keynote at the retail convention Shoptalk. Jeffrey Dastin from Reuters interviewed the pair and pressed them on one question: Why not put this in Amazon’s freshly purchased Whole Foods?

Jeffreyyyy,” Puerini intoned through a grin, as though questioning the company’s plans violated propriety. Eventually she said there were no plans to do so—still Amazon’s line—and that appears true, if only because of logistics. When they debuted, the stores had “beyond state of the art” technology, said Kumar, yet he also noted how nascent AI still is. Human intelligence might recognize a new can of soup after seeing it once, but machines remain grindingly slow learners: Training them takes heaps of data. Each Go store also contains so much hardware—hundreds of cameras and sensors, computing rooms sometimes as large as the sales floors—that scaling it at the moment is “extremely impossible,” Michael Suswal, the cofounder and COO at startup Standard Cognition, told me.

But questions about Whole Foods are less about that chain specifically, and more about what these competing technologies mean for retail and what they mean for the 3.5 million employed as cashiers in the U.S.

Bloomberg reported last year that Amazon plans as many as 3,000 Gos by 2021. Sandeep Krishnamurthy, the dean at the UW Bothell School of Business, thinks widescale automation will come quickly, perhaps a quarter of U.S. stores in the next five years—if politics don’t intervene. “We’ll see who opposes what,” he says. “I think generally people are worried about losing their jobs.” Indeed, United Food and Commercial Workers—the largest private sector union in the country, which represents Kroger employees—immediately slammed Amazon Go, adding another spat to the centuries-long debate over automation and the future of work.

Others move away from the trend. Last year PCC Community Markets decided to run a test at its Redmond location. For three months, the company—Whole Foods’ organic doppleganger, which has 11 co-op stores in the region—monitored which checkout methods people actually used. Two-thirds went for the staffed registers, so executives decided to pull all self-checkout kiosks from stores. “We wanted to create more of a community experience,” Heather Snavely, the company’s vice president of marketing, says. “We’re more high touch than high tech.” The company is also, notably, high end. While some stores will push against automation, peopled registers may increasingly come at a price.

• • • •  

Grocery, unlike most retail, is founded on an existential ultimatum: Eat or wither. Unless we’re ready to subsist only on restaurant food, coax our own from the earth, or get everything delivered, we genuinely need the market. “People will always eat. The way people eat will always change,” Rodney McMullen, Kroger’s CEO, said this year at the National Retail Federation conference.

So how will future stores—bent on ease—change how we eat? In that Redmond QFC after my phone guided me through my list, I realized I needed garlic and wanted some shiitake mushrooms. Yet thinking about the produce hurdles (scanning, weighing, scanning), my mind drifted to the center aisles, their sole bar codes. While not an especially good thing, jarred garlic is a thing.

Ultimately, I bought a head of the fresh stuff, but the shiitake I abandoned. It was a tiny side-effect, based on tinier inconvenience (two bar codes! a scale!), and one that could be remedied, perhaps by a smart shopping cart. The more we use systems, though, the more we can mimic their ideologies: to go fast, to avoid friction. Yet if you’re inclined to believe that our emotional and spiritual health is generally in friction with corporations’ bottom lines—and that our humanity is, in fact, tied up in such friction, since we’re founts of it—then perhaps its erasure isn’t the ultimate goal.

Amazon Go is a novelty still. Tourists flock to the location by the Spheres, taking selfies, buying branded merch. Mugs implore you, in bright orange lettering, to “Just Walk Out.” But for some who work in downtown Seattle, it’s turned to habit. I’ve shopped there 26 times over a month and a half, spending a little more, aware a little less. Not waiting in line, but still checking out.

Share
Show Comments