On Sept. 20, Nicholas Brealy Publishing is releasing The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture, by John Battelle.  --  This edited extract from the book, which was published Friday by the Financial Times of London, describes the benefits and risks businesses incur when they rely on the internet for customers, and the speculates about the future social effects as more and more human acitivities are mediated through search engines.  --  At the end of the passage, Battelle holds out the promise that generalization of optimized search programs in commerce may lead to a more just world:  "Vendors of products that have been made in third-world sweatshops or in factories that overpollute; or vendors that support causes some consumers do not wish to support; would be called out in a far more transparent fashion.  Refusal to participate in such a system would mean that vendors or merchants had something to hide, and so the system could be a major force for good in the global economy, forcing transparency and accountability into a system that has habitually hidden the process of how products are made, transported, marketed, and sold from the consumer."  --  Perhaps.  --  A more certain effect, it may be, is a world in which even the simplest human actions become the object of invasive advertising....

Arts and Weekend

By John Battelle

Financial Times (UK)
September 16, 2005

http://news.ft.com/cms/s/bda649e4-25a8-11da-a4a7-00000e2511c8 (subscribers only)

Neil Moncrief couldn’t afford to have a bad quarter. In fact, even a bad month made things a bit tense at home -- running your own business is like that. When things go south at the office, you take it home with you. “It’s not like working for the man, where you leave it at the office at five o’clock,” he tells me in a soft southern drawl. As a small businessman in the southern U.S. state of Georgia, Moncrief lives on the edge of profit and loss -- a bad month means avoiding his local banker, putting off home and car payments, and having less meat on his family’s table. But Moncrief is proud of what he has achieved. He built a small e-commerce company, survived the nuclear winter of 2001-2002, and emerged with enough cashflow to take care of his family.

Moncrief has search engines to thank for that cashflow, Google in particular. Thanks to the traffic that Google drove to Moncrief’s online storefront, Moncrief no longer worked for the man. But as the holidays approached in 2003, Moncrief got a new boss. His name was Google, and he made old Ebenezer Scrooge look like a saint.

Moncrief’s business is shoes, in particular, big shoes. His company starts at size 13 and goes up from there. Moncrief’s a size 14, and as all in the fraternity of the large-footed know, it’s a pain in the ass to find shoes that fit properly. So Moncrief hooked up with a technically inclined friend “who handles anything with wires coming out of it,” and the two launched 2bigfeet.com in 1999.

Moncrief is one of the tens of thousands of merchants who have taken to the web since the internet became a global phenomenon. For every household brand built during the bubble’s infamous glory -- eBay, Amazon, Expedia -- thousands of Neil Moncriefs toiled in relative obscurity, building the web’s bike shops and insurance agencies, its button merchants and stroller stores. These digital cousins of strip-mall America are the very beating heart of the U.S. economy -- small business, writ large across cyberspace. You think Amazon’s got scale? You think eBay is huge? Mere drops in the bucket. Amazon’s 2000 revenues were around $2.76bn. But the Neil Moncreifs of the world, taken together, drove more than $25bn across the net that same year, according to U.S. government figures.

The logic of selling big shoes over the internet is quite compelling. Only a small percentage of folks are big-footed, and they don’t tend to hang out in geographically concentrated areas. Launching a chain of retail storefronts for such a thinly spread population would be a pretty huge waste of money. Moncrief does have one storefront in Georgia, but it’s mainly a stockroom for the 4,000 or so pairs of shoes he ships around the world every month. It’s fair to say that 2bigfeet.com is a business that owes its existence entirely to the geography-busting elements of the world wide web. On the web, no one cares if you’re based in Albany, Georgia.

Folks in search of a decent-looking pair of shoes for their oversize feet are a pretty motivated set of customers. These are customers that, given the right tools, will search for your business, as opposed to making you search for them. But while the web may offer access to hundreds of millions of customers, you still have to let them know you exist.

Back in 1999, there weren’t a lot of options available to a small partnership with a few $10,000 small-business loans and a website. Moncrief couldn’t afford to cut a big deal for real estate on AOL or Yahoo; he couldn’t even afford banner ads on those sites. (Moncrief was suspicious of them in any case; he didn’t believe folks paid them much attention.)

Given that he had no choice, Moncrief counted on the one thing he thought was a hard-and-fast rule in the internet. When folks went looking for something, they usually started at a search engine. And through some combination of luck, good karma, and what seemed like fair play, when folks punched “big feet” or similar keywords into Google, Moncrief’s site came up first.

Thanks to Google, the orders flowed in. Life was good. By the middle of 2003, Moncrief was moving more than $40,000 worth of big shoes a month, with 95 percent of it coming in through search engine referrals -- the majority of those from Google. The best part was that Moncrief had never purchased an advertisement -- all those search engine referrals were “organic.” People found Moncrief through Google because, well, Google worked as it was supposed to work.

”I figured folks who had to buy an ad, well, there must be a reason they needed to,” Moncrief told me. “We were the right answer for the search, so why buy an ad?”

Then, right before the critical holiday shopping season of Thanksgiving and Christmas, the phones stopped ringing at 2bigfeet.com. Mid-way through November -- November 14, 2003, to be precise -- the orders stopped coming in. For two weeks, Moncrief didn’t know what had hit him. But then he began to wonder -- maybe Google was broken? The very thought seemed ludicrous -- Google, broken? But a quick search on Google confirmed his suspicions -- 2bigfeet.com was no longer the first result for “big feet” on Google. In fact, it wasn’t even in the first hundred results.

As Moncrief put it, it was as if the Georgia Department of Transportation had taken all the road signs away in the dead of night, and his customers could no longer figure out how to drive to his store. What the hell had happened?

In short, Google had tweaked its search result algorithms, something the company does quite frequently. But this time Google’s modifications, which were intended to foil search engine spammers -- people who dishonestly modify their web sites to rank higher in search engine results -- had somehow sideswiped Moncrief’s site as well. What Google giveth, Moncrief learned the hard way, Google can also take away.

Thanksgiving was looming, and Moncrief was facing the loss of his entire Christmas season. What to do? He quickly went to the Google website and attempted to find a number to call or an e-mail contact where he could petition for redress. After all, everything was working before, so why change it now? Why would Google, a billion-dollar Silicon Valley giant, take the time to single out a father of two who runs a tiny shoe business in Georgia? Doesn’t Google realize, Moncrief wondered, that it’s wiped out my business, my livelihood?

Well, in fact, no. Moncrief called Google’s headquarters in Mountain View, California, but never got more than voice mail and nary a single call back. He e-mailed This email address is being protected from spambots. You need JavaScript enabled to view it. and This email address is being protected from spambots. You need JavaScript enabled to view it., but never got a response.

It was then Moncrief realized that while he may have stopped working for the man, he was now working for a far more capricious overlord, one who had no idea he even existed. Moncrief is a cautious man, conservative, a Republican. He’s not the type to ask for government intervention. But when I first spoke to him back in 2003, he was ready to string up the bastards at Google. They were messing with his family, he wasn’t sleeping well at night, and they didn’t even return a simple phone call. Moncrief had four or five bank loan books sitting on his desk, mocking him, and no money to pay them. And there were 4,000 pairs of oversized shoes on his shelves, going nowhere but out of style. How had it come to this?

Moncrief wasn’t the only one wondering. In fact, beginning on November 14, an entire industry of search fanatics went on full-blown alert, tearing up internet message boards with speculation about the latest Google dance -- the moniker given to Google’s periodic updates of its algorithms. These updates had grown increasingly dramatic, and this latest one, coming on the heels of a slew of hurricanes that had hammered the Sunshine State, was dubbed Florida by the search industry. It was Google’s most dramatic yet. On WebmasterWorld, the king of all forums for practitioners of search engine marketing and optimization (SEM/SEO), the reports came in from around the world: Google was updating again and, this time, the SEO industry appeared to be the target.

Optimization companies tune people’s websites so that they rank better in organic search results. While most SEOs are legitimate businesses, many sites promoting optimization sport loud come-ons reminiscent of late-night television, replete with garish promises, many written in poor English (for some reason, SEO seems to flourish in eastern Europe). Some of them take extraordinary and unsavory measures to fool search engines such as Google.

The trouble is that the early SEO industry was not entirely sure which practices were legitimate and which weren’t. In fact, thanks to the rather vaguely worded guidelines on Google’s site, coupled with the fact that the company keeps its algorithms closely guarded, SEO firms were increasingly tempted to push the limits. Many firms also made claims that were simply unreasonable -- “pay me and I’ll guarantee you are listed first in all major search engines,” for example. The result: legitimate business owners unknowingly engage in suspect practices, and their sites get banned from the Google index. When spammers first realised they could boost their rankings with high-traffic words such as “cars” for example, they would hide such words all over their sites, often in small white letters on a white background.

Because Google had become the source of so much traffic for so many, any burp or shudder in the company’s indexes had exponential implications throughout the young world of search-dependent online businesses, which is exactly what happened with Florida.

”Well, this is just terrific. *£%$î*@*¨!” posted a typically exasperated search engine consultant. “I’m going to have a simply fantastic day come Monday, explaining to clients why almost all their sites appear to have been removed from Google. GRRR! Why don’t I just pack up shop now?! I’m going to get crucified. I cannot believe Google has done this again. Geez this makes me angry! (and severely worried about the future of SEO as a viable business).”

In short, Google had updated its indexes to penalize what the company viewed as spam. And a lot of folks, including Moncrief, were caught in the crossfire.

GoogleGuy, an anonymous forum participant who works at Google and has the thankless task of damage control during updates, responded on the same day: “Hey everyone, we’re always looking for ways to improve the quality of our rankings and algorithms. I’ll post more over the next few days -- just wanted to let people know that I’ll be around.”

But while GoogleGuy did keep reading the forum, and even posted carefully worded exhortations that everyone should be patient, the net result of Florida was clear: Google had taken a major stand against what it determined were search engine spammers, and those who felt their legitimate businesses got hurt were told, in essence, to pound sand.

As an agonized poster to WebmasterWorld wrote, summarizing the complaints of thousands: “GoogleGuy please listen to what people are saying. A lot of us are hurting after this update. It couldn’t have come at a worse time, just as the Christmas business was starting. Fifty percent of my business is gone overnight and I may need to lay off warehouse staff or have them standing about with nothing to do. We have always done everything by the book, I can only conclude that our large affiliate network has been penalized by the new algo. Overnight the bulk of our best affiliates have just disappeared out of the index together with two of our best performing sites. Three years’ hard work wiped out in 24 hours.”

That may as well have been Moncrief posting, although Moncrief had never heard of WebmasterWorld, search engine optimization, or affiliates. When I asked him if he had engaged in any spamlike optimization practices, he threw up his hands. “I just have a site that sells shoes,” he told me. “I’m not optimizing anything.”

In the end, engines like Google reserve the right to determine what they believe is the best approach to relevance, and they will tweak their algorithms to ensure the results they feel are most relevant come first. It was clear that by the time of the Florida update, Google had decided that SEO spam had reached unacceptable levels.

But Moncrief and others had additional suspicions as to why so many blatantly commercial results suddenly disappeared from Google’s organic results. After Florida, Moncrief tells me grimly, “I had to buy AdWords. They forced me to do it.”

AdWords is Google’s own text-based advertising program. Taken together with AdSense, its syndicated cousin, AdWords accounted for about 95 per cent of Google’s billion-plus in revenues. After Moncrief dropped to 500th for “big feet” and related searches, he had no choice but to buy his way back up to the top of the list. Otherwise, he’d face extinction.

He did so, but with very mixed results. “[The ads] didn’t work that well,” he complains. That’s not surprising: people tend to click on organic results far more often than they click on ads.

Moncrief was not alone in voicing suspicion that Google’s direct attack on commercial spam ended up benefiting Google’s bottom line. And while it’s impossible to determine whether this fact affected Google’s decision-making process (Google plausibly claims it did not), it’s clear that spam was a drag on the company’s AdWords business. To claim otherwise is to be naive. After all, why buy AdWords if your site gets all the traffic it needs from organic listings? Whether that was the main reason for the Florida update is a matter of contention. But that it helped Google’s business can’t really be disputed.

I called Moncrief eight months after Florida, and he told me his business had once again risen to the top of the Google rankings for “big feet.”

How did he do it? “We hung in there, cleaned up the site a bit, and waited patiently,” he told me. “We worked our way back up.”

What about the last holiday season: did he get back in time for that? “No,” he replied. “We had about four horrible months, and we lost the holiday season, which was the biggest part of our sales.” He paused, reflecting on the experience that, at least until the next time Google decides to dance, is in the past. “It was a tough Christmas for the family,” he concluded.

Moncrief’s story is one of a small-business owner tossed about on the seas of what has become a very big business, but that is by no means all there is to the search economy. In fact, search plays a much larger role in the world of marketing and commerce.

Let’s say you are a young father-to-be. It’s 9pm and your wife has settled, uncomfortably, on to her favorite couch. Clearing her throat, she politely reminds you that you’ve been a bit distant lately, that you haven’t done a hell of a lot to help her around the house. You cringe. She continues: she’s eight months pregnant, for God’s sake, and when are you going to get around to reading that copy of What to Expect When You’re Expecting that she left none-too-subtly in your briefcase six months ago?

Now, you’re in your den, avoiding dealing with the sheer terror of becoming a father by checking your e-mail for the tenth time in so many minutes, but a pang of guilt finally reaches your irresponsible heart. So you start searching the web, trying to get smart quickly. You Google “pregnancy baby” and head to the first link, Babycenter.com, where you read up on the eighth month.

You then find a link to an article that lists 10 things you can do to be a better husband. The fourth suggestion reminds you to read the books your wife has purchased, so you head to Amazon and buy another copy of What to Expect When You’re Expecting, as you left the one your wife gave you next to the Gideon Bible on your last business trip. “I’ll read it, I promise,” you tell your wife, and then add, “I’m on Babycenter right now, in fact.” Pleasantly startled, your wife springs off the sofa -- well, lumbers, perhaps -- and peers over your shoulder. In a flash of inspiration, you intuit that there might be something you could watch together on TV that relates to the whole parenting thing. “Let’s see if there’s anything on TV that might be good,” you say.

You click over to your TiVo home page, which lets you manage your television service through a search-based interface. You search for “parent childbirth newborn” or something like that and find that there are five shows in the next week that focus on the course of pregnancy, three of them on the Learning Channel. You tell TiVo to record them all, noting that the first one will be available to download tonight, in half an hour, no less.

In the background of your computer, as you jump from site to site and page to page, several marketing-related actions are occurring. A cookie set by your local cable company notes that you’ve visited several sites that are marketing potentials -- Amazon.com, TiVo.com, and Babycenter.com, all sites that indicate significant intent to purchase products or services. You’ve also alhat [sic] you intend to download five new programs, and the system takes note of content tags associated with those programs, cross-referencing them with your recent search history.

The cable cookie shares this information with a marketing application running in the background of your computer, perhaps as part of that Google Desktop Search (GDS) program you downloaded last year. Alerted by the marketing potential that your recent surfing has created, GDS instantly uploads new tags to Google’s central advertising marketplace.

Up on Google’s ad marketplace, millions of similar potentials are aggregated and presented to hundreds of thousands of advertisers for sale in a modified real-time auction. Most of those advertisers have preset their spending levels, demographic preferences and, most important, intent-based targeting profiles.

In the time it takes for an average Google search to finish -- less than a second -- several advertisements have already been sold against each of the five programs you’ve selected.

Half an hour later, you and your wife turn on your television to catch the Learning Channel show. As it starts, a small box appears on the bottom of the screen, alerting you to several advertisements that have appeared in your feed. You pause the show, hit the ads button, and scan the commercials.

Only they’re not just commercials; they’re offers as well. The first is from Gerber for a free month’s supply of formula. (Pass; you and your wife have agreed that breast-feeding is the way to go.) Next up is a Pampers ad offering a free box of diapers. (Sure, why not? You accept that one, clicking the box that allows the system to send your details to the Pampers marketing machine.)

Then comes the killer ad: “Click here for $50 off a Peg Perego Stroller. Ships in 24 hours!” Huh? you think to yourself. That’s the one your wife said was the Mercedes of strollers. Maybe you can afford one after all. “Honey,” you begin. “What do you think? Should we go for it?” Her eyes light up (you had said no to this exact request twice -- $300 for a fucking stroller?!! were your exact words) and you click to accept the offer. Your wife snuggles into your side, pleased that, for once, her husband actually gets it.

Is such a scenario possible? While the details will inevitably vary, I honestly think it is not only plausible; it is inevitable.

For an advertiser like Peg Perego, such a scenario not only makes television advertising affordable; it turns the medium into a new sales channel. Instead of buying time on the Learning Channel on Mondays at 8pm (a content-attached purchase), Peg Perego will buy direct access to the intentions you have declared through a blend of your search history and your television watching habits. Once it is satisfied that you are a potentially high-value customer, it will then push advertising offers down the cable line to your digital video recorder.

The beauty of this scenario lies in how it changes the economic model of marketing. First of all, Peg Perego has never been a television advertiser because the medium has never lent itself to a high enough return on investment -- the company relies mostly on word of mouth and distribution through a network of retail outlets for its sales. But because it can identify exactly who its customers might be, on the basis of intent, it can change its model completely, and view a marketing investment in television to be, well, not a marketing investment at all, but rather -- this is worth stating again -- a new sales channel. This in turn means that tens of thousands of marketers who otherwise may never have considered television a viable medium will soon see it as such.

That is the magic of intent-based marketing -- it shifts marketing dollars from the unknown to the knowable. As Tim Armstrong, vice-president of advertising at Google, puts it, “search turns a cost center into a profit center." Think about that for a minute. The entire foundation of marketing -- a $100bn industry driving, well, nearly every business on this planet -- is shifting, slowly but surely, to a new model, one informed by the simple idea of people looking for things on a search engine. No wonder Jan Pedersen, chief scientist for Search Marketplace at Yahoo, recently quipped: “We think of shopping as basically an application of search.”

Marketing is not the only industry facing massive change in the search economy. There is probably no greater example of a thriving off-line search business than the yellow pages, now worth around $15bn in the U.S. alone. Within one generation, however, the yellow pages will be viewed as a dead industry.

Now, before you tell me that flipping though a printed directory is far more convenient than turning on your computer and punching in some search terms, let me remind you that local search, as it’s called in the search industry, is still in its very early stages. There will be about 1.7 billion mobile phone handsets in use by the year 2006, and most of those will have internet access.

When finding a dentist is as easy as punching “dentist” into your phone (or, with new technology already on the market, simply speaking it), the idea that anyone will pull out a 10lb paperweight to execute a search will seem as quaint as hand-cranking a car.

Imagine it’s the near future and you’re in your local grocery store on a mission to pick up food for a Saturday night dinner party. Because you’ve got oodles of disposable income to burn, it’s a high-end Whole Foods store, the aisles dripping organic righteousness and whole-grain goodness. You know that dinner for eight is going to run to at least $200, not counting the wine, but that’s OK compared with the tab at the local bistro. You’ll be coming out ahead. But you do want to make sure you’re not spending money you don’t have to, especially on the wine.

Now, Whole Foods has quite a wine selection, but the store isn’t known for its discount prices on anything, and when it comes to wine, you’ve got a sneaking suspicion that the store is really sticking it to you. But it’s a convenience buy, you’ve always thought, and you’re willing to put up with it for the most part.

As you slip your Neiman Ranch tri-tip roast beef into your basket and thank the butcher, you head to the wine aisle. What might go with that grilled tri-tip? A nice Cabernet, no doubt. Whole Foods’ wine aisle, a testament to hierarchy and peer pressure, places the most expensive bottles on the top, and the cheap juice on the bottom. No self-respecting Whole Foods shopper wants to be seen bending down to check out a bottle of wine. Then again, those bottles staring out at you from eye level are exactly the kind that you suspect Whole Foods marks up with the glee of a four-star sommelier.

What to do? Not to worry; you’ve got Google Mobile Shop installed on your phone. You whip out your Treo 950, the one with the infrared bar code reader installed, and you wand it over that $52 bottle of 2001 Clos du Val now lovingly cradled in your arm. In less than a second, a set of options is presented on the phone’s screen. It reads:

2001 Clos du Val Merlot, Lot 21
Stags Leap District, Napa Valley
Average Retail Price: $38 (click here for more)
Click here for a list of prices at nearby stores
Click here for stores selling similar items
Click here for reviews of 2001 Clos du Val Merlot
Click here for more on this vendor [ecological impact, vendor labor policies]

You’re pretty sure that Clos du Val isn’t employing child laborers, and anyway you’re really interested only in price comparisons, and the first screen has confirmed your initial suspicion: Whole Foods is ripping you off.

You click on “list of prices at nearby stores” and see that the liquor store up the street is selling the same bottle for $39. You click on that store’s link, and then choose the “reserve this item for same-day pickup” option. With a satisfied smirk, you replace the bottle on its perch on the top shelf, and head over to compare prices and recipe tips for $6 boxes of imported pasta. As you leave, the fellow who runs the store’s wine department eyes you warily, then picks up the phone to talk to his manager. “Herb,” he says. “Did you get my message about banning cell phones in the store?”

Is this scenario possible? For it to happen, a few nontrivial things need to occur. First, the entire UPC barcode system must be made open and available as a web service -- a nontrivial event, to be sure. Those bar codes and the information within them are not yet a public resource, though a small coterie of researchers and entrepreneurs is looking to change that. Second, merchants must be compelled to make their inventory open and available to web services. Third, mobile device makers must install readers in their phones, essentially turning phones into magic gateways between the physical world and the virtual world of web-based information. And fourth, providers like Google must create applications that tie it all together.

While the first few hurdles to the realization of this scenario have yet to be jumped, it’s certainly a no-brainer that Google and Yahoo would love to tie everything together should it become possible to do so. The implications of search breaking out of the PC box and making real-time information available at the point of purchase has been the failed business model of several web 1.0 companies. But with recent developments in local and mobile search, it is far closer to happening now.

What might be the effects of such a system coming to fruition? For one, markets would have to compete far more on service, convenience, ambience, and other factors unrelated to price. And vendors of products that have been made in third-world sweatshops or in factories that overpollute; or vendors that support causes some consumers do not wish to support; would be called out in a far more transparent fashion. Refusal to participate in such a system would mean that vendors or merchants had something to hide, and so the system could be a major force for good in the global economy, forcing transparency and accountability into a system that has habitually hidden the process of how products are made, transported, marketed, and sold from the consumer.

Whether or not such a system actually develops, the fact remains that it could, and that alone is a powerful force on the vast ecosystem of local and global commerce. Decision by decision, possibility by possibility, click by click, search is shifting the firmament of our economic world, and what we’ve see so far -- the billions upon billions of dollars running over the servers at Google, Yahoo, and others -- is simply the first indications of that shift.

--This is an edited extract from The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture, by John Battelle, to be published by Nicholas Brealey Publishing on September 20. To buy a copy for £16.99 with free postage and packing (in the UK only) call 020 7239 0360, or email This email address is being protected from spambots. You need JavaScript enabled to view it.