Total recall: Microsoft's Gordon Bell feeds the data of his life into a surrogate brain. Soon you will be able to as well
Clive Thompson
Thursday, March 01, 2007
Gordon Bell will never forget what I look like. He'll never forget what I sound like too. Actually, he'll never forget a single detail about me.
That's because when I first met the affable 72-year-old computer scientist at the offices of Microsoft Research Labs, in Redmond, Washington, he was carefully recording my every move. He had a tiny bug-eyed camera around his neck, and a small audio recorder at his elbow. As we chatted about everything from Australian jazz musicians to Seattle's gorgeous weather, Bell's gear quietly logged my every gesture and small talk, snapping a picture every 60 seconds. Back at his office, his computer had carefully archived every document related to me: all the e-mail I'd sent him, copies of my articles he'd read, pages he'd surfed on my blog.
"Oh, I've got everything," Bell said cheerily. The next day, in his cramped personal office in San Francisco, he gave me a glimpse of the memories he'd collected. He plunked down in front of his computer, pulled up a browser, typed in "Clive Fast Company," and hundreds of pictures of the meeting scrolled by on his screen, and the sound of our day-old conversation filled the room. It was a deeply strange feeling. My random chitchat is being preserved? For all eternity? He nodded, pointing to a mundane Dell computer beneath his desk. His massive store of data. His "surrogate brain."
Because I'm not the only thing Gordon Bell will never forget. His goal is never to forget anything.
For the past seven years, Bell has been conducting an audacious experiment in "lifelogging" -- creating a near-total digital record of his experience. His custom-designed software, "MyLifeBits," saves everything it can get its hands on. For every piece of e-mail, every document, every chat session, every Web page surfed, a copy is scooped up and stashed away. MyLifeBits records his telephone calls and archives every picture -- up to 1,000 a day -- snapped by his automatic "SenseCam," that device slung around his neck. He has even stowed his entire past: The massive stacks of documents from his 47-year computer career, first as a millionaire executive then as a government Internet bureaucrat, have been hoovered up and scanned in. The last time he counted, MyLifeBits had more than 101,000 e-mails, almost 15,000 Word and PDF documents, 99,000 Web pages, and 44,000 pictures.
That load has endowed Bell with the ability to perform supernatural feats of memory. He can dredge up the precise contents of an inspirational note above his desk 30 years ago (a set of aphorisms, including "Start many fires"). He knows who passed him on the street on the way to work four weeks ago. And when someone disputes his recollection of a conference call the previous day, he can end the argument by pulling up the audio stream and listening to it again. Instantly.
"It gives you kind of a feeling of cleanliness," Bell tells me. "I can offload my memory. I feel much freer about remembering something now. I've got this machine, this slave, that does it."
It gives his mind the chance, he says, to be more playful, to have more energy for creative thinking. But it is also a double-edged sword. Bell suspects MyLifeBits might be slowly degrading his real, carbon-based brain's ability to remember clearly. When you have an outboard mind doing the scut work, you tend to get out of practice. "It's like doing arithmetic," he says. "Who does it anymore? You've got pocket calculators for that. I know I can do long division. But I haven't done it for a long time."
It's a crazy experiment. But perhaps its craziest aspect is that soon you'll be part of it too -- whether you want to be or not. The way Bell sees it, computers and the Internet are now rapidly becoming capable of storing everything you do and see. Hard-drive space has exploded in size, and every day people record more and more of their lives: We blog about our thoughts, upload pictures to Flickr, save every e-mail on our infinitely expanding Gmail accounts, shoot video on our cellphones, record phone calls straight to our hard drives when we use Skype.
"People say, 'Oh, what you're doing is revolutionary!'" Bell says. "I say, 'No, no, it's evolutionary. Because it's happening to you. It's happening as you speak.'"
So what will life be like when nothing is forgotten? Provocative as that question may be, it's hardly theoretical. The thinking behind MyLifeBits and other lifelogging research is already seeping into our lives. It's changing the way our search engines work. It's affecting corporate strategy. And the power of machines to create boundless memory -- and to augment and even transform human thinking -- is only going to become more pronounced. We've arrived at a time when the memory of machines creates ideas we've never considered.
Paperless Trail
In 1945, presidential science adviser Vannevar Bush published a provocative essay in The Atlantic Monthly entitled "As We May Think." Bush argued that man's mind could be perfected by technology. He envisioned a device called a Memex, "in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility." A user would wear a "walnut-sized" camera on his forehead, capturing everything he saw, then sit down at his Memex to browse thousands of personal letters, newspapers, and encyclopedias instantly. It would be, Bush argued, "an enlarged intimate supplement to his memory."
MyLifeBits was born from a much humbler idea: Bell was sick of carting around stacks of paper. He was a veteran of the computer revolution -- indeed, he helped kick-start it in the 1960s and 1970s by building the first refrigerator-sized "minicomputers" for DEC, a pioneering computer firm. In the 1980s, he helped the government bootstrap the Internet into existence, then worked as an angel in Silicon Valley, growing wealthier and wealthier as his investments took flight. Hired in 1995 by Microsoft Research Labs, a wing of the company devoted to designing the future of computers, Bell was given carte blanche. He decided to become the first person in history "to truly go paperless."
So he bought a scanner, and his poor assistant Vicki began making PDFs of four enormous filing cabinets' worth of stuff. The archive begins with photos of Bell's mother's birth in 1900 and basically never stops, sucking in everything from the sublime to the ridiculous: Bell's medical records, his Japanese-made notebooks filled with his elegant sketches of computer circuitry, phone bills and stickie notes.
His appetite whetted, Bell decided to store even more data. Two Microsoft researchers, Jim Gemmell and Roger Lueder, who built software to automatically save digital copies of everything Bell generated: chat transcripts, every Web page he looked at, even records of his keystrokes. Then Lyndsay Williams, a Microsoft inventor in Cambridge, England, came up with the SenseCam. It would snap pictures either at regular intervals or when triggered by a meaningful event, such as when its infrared detectors sensed someone standing in front of Bell, or when its light sensors saw that he'd entered a new room. A tagalong GPS device stamps each picture with its geographic location.
At first, Bell was worried about filling up his hard-drive space too quickly. He accumulates one gigabyte of information a month. At that clip, the average MyLifeBits for an 72-year-old person would require one to three terabytes -- a hefty amount of storage. But by 2000, driven by teenagers' insatiable desire to store MP3s and video clips, hard drives had dropped radically in price and grown enormously in capacity. Bell figures that in a few years, even a cheap cellphone will have enough space to store your entire existence. "We've gone from this period of scarcity, when you had to always go, 'Jeez, I can't keep this video file because my hard drive is full,' to the opposite," Bell says. "I tell people, 'Never throw anything out. You'll never have to worry about space for the rest of your life.'"
Slowly, in often subtle ways, MyLifeBits began to affect Bell's life. During a phone call to discuss a heart problem last year, Bell couldn't follow his doctor's flood of jargon--but he could listen to the call again and decode it at his leisure. A friend passed away; Bell was able to pluck a piece of 20-year-old correspondence from the mists for his eulogy. Meanwhile, the presence of the SenseCam and audio recorder began creeping out his "significant other," who wasn't sure she liked having everything set in stone. "We'd be talking, and she'd suddenly go, 'You didn't record that, did you?'" Bell chuckles. "And I'd admit, Yeah, I did. 'Delete it! Delete it!'"
Bell also discovered he was getting annoyed by experiences that couldn't be stuffed into a hard drive. During a ride in a cab in Australia, a tiny security-cam surveyed him, and he wondered why he couldn't automatically get a copy of the feed. And books drive him crazy. "I virtually refuse to own any books at this point," he says. "I mean, I get them, I look at them, I occasionally read them. But then I give them away, because they're not in my memory. To me they're almost gone."
HEALTH BENEFITS
Martin Conway, a psychologist and memory expert at the University of Leeds, argues that projects like MyLifeBits can actually improve mental health by freeing our brains to be more productive and more creative. "We're moving into an age when technology is going to massively enhance our cognitive abilities, our problem-solving abilities," he says. It's rather like the way Google has already become an indispensable part of how people think about things -- sitting at their desks, constantly tapping into the world's massive trove of information. "Your real memory becomes a sort of executive manager for all these other technological abilities."
"Forgetting is how we make sense of life," says one skeptic. "We need to forget." Personal-productivity guru David Allen also has long argued that the frailty of everyday memory is the primary source of stress for overburdened corporate types. We sit around anxious about our to-do lists because we can never entirely remember them (while we're at work) or entirely forget them (when we're not).
Yet Bell's project has also made some observers nervous. It may not be a good idea, they argue, to tamper with human memory -- because it's such a powerful part of what makes us who we are.
"I'm a big fan of forgetting," says Frank Nack, a German computer scientist who published a critique of lifelogging experiments last winter. "It's how we make sense of life, how we interpret things. Everybody is building a life story; we all need to forget certain stages. I don't want to be reminded of everything I said." Forgetting, he points out, is key to cultural concepts like forgiveness and nostalgia. Sure, we lose track of most of what happens to us--but that natural filtering process results in what we call knowledge and wisdom. When memories are only a click away, Nack says, they're cheapened.
James L. McGaugh, a memory expert at the University of California, Irvine, points to the sad spectacle of Funes, a character in a Jorge Luis Borges story who suffers a head injury that renders him incapable of forgetting. "He says, 'My mind is like a garbage heap.' That's what it'd be like," McGaugh adds. "You have to watch what you wish for with memory."
As Bell's significant other realized, if everyone had a record of every conversation, it could turn everyday life and work into a maddening series of gotchas.
What's more, knowing that everything is being logged might turn us into different people. We might be less flamboyant, less funny, less willing to say risky but potentially useful things, much as politicians on-camera tamp down their public statements into stifled happytalk. "There'd be a chilling effect," particularly early on, says Mark Federman, former strategist for the McLuhan Program in Culture and Technology, a high-tech think tank. "We'd all be on our best behaviour. Reality would become reality TV."
Bell too acknowledges that all this remembering could have a downside. "Fifty years from now, do you want to know that, gee, I visited a porn site today?" he asks with a smile. And when it comes to corporate information, he admits that "ownership, deniability, privacy, expungability -- they're important." Microsoft hasn't yet objected to all of its sensitive corporate memos going into Bell's off-site brain, but he suspects that day will come. When he eventually retires, he says, he'll be in the weird position of having to shave those memories off and give them back. "I'll need a lobotomy," he says, only half joking.
Still, Bell insists the trend toward total memory isn't going away. More and more, it is happening automatically. Those tens of millions of bloggers and Flickr users -- all out there recording their thoughts and pictures -- have clearly decided that there's enormous value not just in capturing those thoughts but in sharing them with the public. The choice isn't whether you'll join the revolution but whether you'll embrace it.
Shaping Chaos
For all of its machine muscle, Bell's virtual memory still faces hurdles. MyLifeBits is now so big that it faces a classic problem of information management: It's hellishly difficult to search. Bell often hunts for an e-mail but can't lay hands on it. He gropes for a document, but it eludes him. When I ask him to locate a phone call from a colleague, he can locate the name of the file, but when he clicks on it the data are AWOL. "Where the hell is this friggin' phone call?" he mutters. "I either get nothing or I get too much!"
This turns out to be the central question behind MyLifeBits: Yes, it's possible to store a lifetime of memories, but what do you do with them?
To figure that out, I made a visit to Mary Czerwinski, a principal research scientist at Microsoft Research Labs whose team has developed "Facetmap," an audacious piece of software designed to visualize the contents of Bell's cybermemory.
Czerwinski pulls me over to a massive LCD monitor on her office wall. It shows a collection of colorful blobs representing different parts of Bell's life. There's a blob for people, another for calendar dates, and a bunch for different types of documents like e-mail or Word files. If you click on any blob, it instantly expands to show you everything it contains. Click on the blob for "Jim Gemmell," Bell's main collaborator, and you'll see a blob containing all their e-mail traffic, another with documents that mention Gemmell's name, and a third with events where he appears. The more data in each category, the bigger the blob, "so you can quickly see which area has had the most action," she notes.
The truly intriguing part about Facetmap is that it shows how Bell's information is connected. I start poking around, clicking on Gemmell's blob, then drilling down to a particular e-mail Bell sent him on Feb. 25, 2005. As I zoom in, the software automatically creates new blobs showing everything else Bell did on that day -- e-mails to other colleagues, photos Bell took, websites he viewed. It feels like flying freely through Bell's memories, flitting anywhere I want. And it re-creates those same loosey-goosey linkages that tie memories together in our real-life minds. It's like software for productive daydreaming.
Facetmap is based on a truth psychologists have long understood: We organize our memories by time and people. One of Czerwinski's colleagues at Microsoft, Susan Dumais, analyzed how people search for things on their computers and found that about one-quarter of the queries were for someone's name. She also explored ways to use "landmark" events to index our memories. "You'll think, 'Oh, I'm sure it happened right before the wedding, or just after Hurricane Katrina,'" Dumais says. She developed an experimental piece of software that embedded those landmarks into search tools so that you could, say, start with a major event and then see all the e-mail or Web pages you looked at that day. It drastically improved the ability to find things in the distant past, she says.
These sorts of tricks are already helping Bell find his memories, and in fact, everyday search tools are already moving slowly in this direction. In the past year, free "desktop search" programs by Google and Microsoft -- which scour your hard drive -- have begun incorporating sophisticated filters that let you work in similar ways: You can start by looking for a person, then find all the memos you've written to them, then quickly zoom in on a day.
Instant Replay
One day when I met Bell, I saw his life through his eyes. He'd worn his SenseCam that day, and when he plays the images back rapidly, it's like watching a crude, stop-motion movie: buying coffee at Starbucks, grabbing a paper, entering his building, and finally dropping down at his desk.
Yet here's the problem: The pictures pose an even bigger search dilemma, because computers can't "see" the contents of a photo. It's impossible for Bell to hunt for "pictures of my desk at work," or "that tall blond guy I met at the party"; at best, he can sort them by date or GPS coordinates. And while he has added keyword "tags" to many shots, it's time-consuming and still not terribly accurate. Even he admits he rarely peruses any of his thousands of SenseCam pictures.
So are all those photos a waste of memory? Or can that kind of exhaustive visual record actually be worth something?
Alan Smeaton, a professor of computing at Dublin University, thinks it can. After hearing about Bell's project, Smeaton got Microsoft to lend him a few SenseCams. His students began wearing them all day long and discovered an intriguing psychological effect: If, at the end of each workday, they spent a minute scrolling through the thousands of pictures taken -- a high-speed replay of their day -- it had the effect of stimulating their short-term memory.
"You'd see somebody you met in a corridor and had a two-minute conversation with that you'd completely forgotten about. And you'd go, 'Oh, I forgot to send an e-mail to that guy!' It's bizarre. It improves your recall by 100 per cent," Smeaton says.
In fact, "refresher" imagery is so powerful that it seems to help restore recall in people who have very little memory, or none at all. Ken Wood, a computer scientist in Microsoft's research lab in Cambridge, gave a SenseCam to a UK woman who had lost her short-term memory due to encephalitis. She began wearing it to events she wanted to retain. After each of them, she reviewed the pictures several times over the course of the following two weeks. When researchers quizzed her a month later, she still had "significant recall."
"She was over the moon," Wood marvels. "Totally thrilled."
Silicon Cortex
Consider for a second how, precisely, we think. We use our memories all the time, of course, often by "active" remembering -- scrolling through our minds to locate a tidbit. But much mental labour is passive. We think about something in the background, subconsciously letting a problem brew. Then we suddenly hit upon an interesting combination of things, a new way of thinking about a problem: the elusive, all-important epiphany.
What if our computers had their own intelligence, and could do that background work for us? What if they could mine our memories for new ways of thinking? And what if they could prioritize the vast heaps of material in the backs of our minds, shaping the informational chaos that often leaves Bell so baffled? A memory system that could think on its own would unlock the lifelog's full potential.
Bell and Gemmell have played with this effect using the SenseCam data by developing a screen saver that displays random snapshots from their personal archives. Bell finds it oddly mesmerizing: Pictures of long-ago birthdays or family trips trigger waves of nostalgia, he says. Czerwinski predicts that a similar screen saver could become a killer app in the office. When you're working on a project, the screen saver would cycle randomly through any documents, pictures, e-mails, or Web pages pertaining to your work -- and you would see if the unpredictable combinations inspired fresh ways of understanding it.
But the real goal is "to discover things that even you didn't know that you knew," says Bradley Rhodes, a computer scientist with Ricoh Innovations. In the lifelogging community, Rhodes is famous for creating the "Remembrance Agent," an experimental piece of software, as a PhD student at MIT. The Agent sits in the corner of your screen and pays attention to everything you type; every few seconds it checks inside your hard drive to see if it can find anything relevant. If it does, it alerts you in the corner of your screen by showing a line or two of the related document.
The connections the Agent discovers are surprising, often valuable. When Rhodes first started using it, he'd begin writing an e-mail to ask a colleague a question, but before he could even push "send," the Agent would reveal that a long-forgotten document on his hard drive already contained the answer. Other times, a colleague would e-mail him a question and the Agent would remind Rhodes that he'd been asked that once before and had forgotten to reply. "So, in mid-e-mail, I realize I have to switch gears and apologize and go, 'Sorry for not getting back to you.' It actually would change my behavior," he says. By actively reminding you of things from the past, "it keeps you from looking stupid." Now there's a killer app.
GIFT OF MEMORY
In spring 2004, Gemmell lost a chunk of his memory. The Microsoft senior researcher had built his own personal MyLifeBits database, filling it, like Bell, with oodles of his e-mail, Web surfing, and pictures. But one day, Gemmell's hard drive crashed, and he hadn't backed up in four months. When he got his MyLifeBits back up and running, the hole that had been punched in his memories was palpable, even painful. He'd be working at a project and vaguely remember some website or document that was important, then begin drilling down to find it -- only to discover that it was part of the missing period.
"It was like having my memories stolen," he says. He was amazed to realize his backup brain was no longer some novelty but a regular part of his psychological landscape. "I realized I count on this now. It's like I expect to drive cars and have flush toilets."
Machine memory is obviously giving us astonishing abilities, but can we deal with the change it'll wreak in our lives and work? We might become so reliant on artificial memory that we lose the habit of noticing things. "What is going to happen to us if we bypass something and we ought to have noticed it?" wonders Frank Nack, the lifelogging skeptic. Is it possible to forget how to remember? Perhaps so: Most of us have lost a cell phone only to realize that we can no longer recall the phone numbers of even our closest friends because the machine remembered them for us.
Whatever it all means, Bell will likely be the first person on the planet to find out. As I leave him at his San Francisco office, he offers me a parting gift, and fittingly enough, it's a memory device: one of the notebooks he buys at a nearby stationery store. It's beautiful, oddly old-fashioned. Yes, the man who swore off paper knows it would be much easier for MyLifeBits if he did all his writing electronically, on a digital tablet or whatnot. "But I can't help it," he says, "I just love these gorgeous Japanese-made notebooks." He did, after all, spend decades as a young engineer recording every idea on pads of his own. Some habits die hard.
This article first appeared in Fast Company
