Spiders in the Web

Searching for profit has become highly competitive.
Economist StaffMay 17, 2004

The tech folk in Silicon Valley knew something was up last year when another web spider appeared. Web spiders are powerful software programs that crawl around the world wide web, automatically analysing words, following links and collecting vast amounts of data. The catalogues they assemble are used by search engines to display an index of web pages in response to key words entered by a user. What was intriguing about this particular spider is that it belonged to Microsoft. Even the world’s biggest software company cannot afford to ignore the power of search.

Only a few years ago, search did not seem to matter very much. The gateways to the internet were going to be “portals”: sites such as Yahoo!, Microsoft’s MSN and AOL. These sites would provide users with everything they needed, including links to whatever internet pages the portals’ editors deemed useful. The idea was to keep users cosseted within the portals, where they could be sold things, presented with ads and encouraged to sign up for additional services.

But Sergey Brin and Larry Page had a different idea. They started building a search engine that would not only examine the words on web pages, but also look at how and where those words were being used and at the number of other websites linked to that page. Their search engine would rank pages in their likely order of usefulness.

Drive Business Strategy and Growth

Drive Business Strategy and Growth

Learn how NetSuite Financial Management allows you to quickly and easily model what-if scenarios and generate reports.

In 1998, the pair put their studies at California’s Stanford University on hold to start Google. Having raised $1m, they set up in a friend’s garage, answering 10,000 search queries a day. These days Google deals with more than ten times that number every second. Its search methods, which form part of a secret algorithm, have become far more elaborate. Put in the word Google, and you get links to over 45m other web pages in a quarter-second—with at the top of the list. Instead of encouraging you to linger, Google’s spartan website is designed to find links as quickly as possible and move you on. For many people, it is now their launch point on to the internet.

Despite its simple appearance, Google has found a way to make lots of money. Just how much became clear for the first time last month when the privately held company announced plans for a stockmarket listing, with its shares to be sold by public auction. Its net revenue in 2003 turned out to have been $962m, 176% up on the previous year. Google is an extremely successful company, but it is going to have a fight on its hands.

The way to make money out of search is to sell the words people put in when they look for things on the web. This means that whoever bids the most for a particular term, “digital cameras”, say, gets their link on the top of a list whenever anyone types that phrase into the search box. The convention is to distinguish between independent search results and those that are paid for. Google lists its “sponsored” links on the right of its page; Yahoo! puts them in a box at the top.

Searches are now such an important tool for connecting buyers and sellers that some companies spend most of their marketing budget on getting a high page-ranking in the sponsored lists and on various services which can help websites move up the independent listings. A website operator could, for instance, pay a search-engine-optimisation company to help him design his site to maximise its page ratings. Such companies sometimes employ tricks such as setting up “ghost” web pages, which make a site appear more widely used than it really is. Google and other search companies constantly fiddle with their algorithms to try to prevent their independent search results from being manipulated in this way.

But there is also a grey area. Yahoo!, for instance, offers a service that lets traders pay to have their website re-crawled every 48 hours instead of waiting (sometimes up to a month) for a web spider’s regular update of the search engine’s catalogue of pages. Paying to be re-crawled ensures that when a search is carried out the company’s latest products and prices will be included. Yahoo! says such payments do not buy a better placement in its independent results. But sites with more up-to-date information are more likely to be relevant to a search, and therefore tend to rise higher.

According to most analysts, the paid-search business is leading the recovery in advertising expenditure on the internet. A study by the New York-based Interactive Advertising Bureau in partnership with PricewaterhouseCoopers, a consultancy, found that online ad spending in America last year grew by 21% to $7.3 billion.

Count the Clicks

The great advantage of paid search as a marketing expenditure is that it has a built-in evaluation mechanism: companies pay only if a user clicks through to them from a search-engine link. Companies can also track how many of those visitors eventually end up buying something, so they can work out just how much it is worth bidding for a search term.

Google’s paid-search service, known as AdWords, in effect sells advertisers a “pre-qualified lead”, says Sheryl Sandberg, who runs the company’s global online sales and operations. Google employs an automated bidding process using a version of what economists call a Vickery second-price auction. Winning bidders pay only one cent more than the bidder below them. Hence, if there are bids of $1, 50 cents, 25 cents and so on, the winner of the top place pays 51 cents a click-through, the winner of the second place pays 26 cents, and so on. This, Google believes, encourages companies to bid whatever they think a search term is worth, safe in the knowledge that the final price they pay will be less. But even payment does not necessarily guarantee the top slot. Google also ranks its sponsored links according to their usefulness, which means that if more people click on a sponsored link further down the list, that link can overtake sponsors who are paying more. “That way we reward relevance,” adds Ms Sandberg.

The opening shot in the search war came in February when Yahoo!, also based in Silicon Valley, sacked its neighbour Google as the provider of its search engine and replaced it with a proprietary service. Yahoo! has been investing heavily in search technology. In July 2003 it paid $1.6 billion for Overture Services, a company that pioneered much of the paid-search advertising business, and in March this year it agreed to buy France’s Kelkoo for euro475m ($576m). Kelkoo operates an internet search service for comparison shopping. Google also recently put a link to Froogle, its comparison-shopping service, on its main site.

Yahoo!, which was also founded by two Stanford students, David Filo and Jerry Yang, has been growing strongly with the help of paid-search advertising. The first quarter of 2004 was the most successful yet in the company’s history, says Terry Semel, Yahoo!’s chief executive. Both revenue and profits more than doubled on a year earlier, to $758m and $101m respectively. Yahoo! is transforming itself into a site that is part portal, part shopping mall and part search machine. Its strategy is based on what Daniel Rosensweig, Yahoo!’s chief operating officer, calls the “desert island question”: if you were stuck on a desert island and could have access to only one website, which one would it be? Abandoning Google, he says, gave Yahoo! a more differentiated product.

Yahoo! is also starting to integrate the results of its searches with other services it provides on its website. For instance, it recently launched a service called SmartView that links searches with its online maps. This allows someone looking for a petrol station in San Francisco, for instance, to type in his location to get an interactive map with all the nearby petrol stations, and driving directions if required. Finding local information is becoming an area of hot competition.

Some people believe Microsoft could put a stranglehold on Google, as it did on Netscape in the battle to control the web-browser business. At the World Economic Forum earlier this year, Microsoft’s founder, Bill Gates, admitted that so far Google had well and truly beaten his company in the search business. But Microsoft will try to leapfrog the existing search engines with its own technology and integrate its service within its own software, including its Office programs and the new version of Windows, known as Longhorn, due to appear in 2006. At present, searches carried out on Microsoft’s MSN site are partly powered by Yahoo!’s search engine, but that relationship is likely to end once Microsoft introduces its own technology.

Windows runs more than 90% of the world’s PCs, so this is bound to raise further antitrust concerns. As more computers become permanently plugged into high-speed internet connections, many more people will search directly without opening a web browser while working on a report or presentation. As with media players, the providers of rival search engines will be watching closely to see just how difficult Microsoft’s new products will make life for them.

At present, search marketing is considered a relatively cheap and cost-effective method of finding customers, but there are only a finite number of search terms, says Kenneth Cassar, director of strategic analysis for Nielsen//Net Ratings. This means the price of popular search terms is bound to rise, which will encourage websites to find new places for context-based ads. Websites will also try to keep their exit barriers to users high, not least by making it more convenient to stay with them than to start again somewhere else, with all the hassle of filling in personal details and credit-card information.

The search business is still young, and could fragment into many different products for different purposes. That would provide opportunities for newcomers beside Microsoft. Ask Jeeves, for one, has become a slightly less distant third after improving its technology. And there are a number of smaller companies that could yet produce a breakthrough product with mass appeal, just as Google did when it operated out of a garage.

More specialised searches, such as those concentrating on web pages that match users’ interests, will be a big area of development. Someone entering the word “tuning” on a car-related search, for instance, would no longer have to scroll past links about musical instruments, television stations and faster software to find pages that explain how to adjust car engines. Search sites could also “learn” about users’ interests from the links that are clicked through, or users could be invited to provide their personal preferences. Google and Yahoo! have already gone some way down this route. Both, for instance, now operate news services that not only report the latest events but also allow users to search through thousands of publications to retrieve past articles.

Advertising helps pay for the internet’s free services, such as search. To continue to get these things free, users may have to be prepared to accept ads. Gmail, a new free e-mail service which Google is testing, offers users more than 100 times the storage space of rival e-mail services, in return for agreeing to sponsored links being placed in their e-mails. Google’s computers analyse the content of an e-mail and then dish up ads that seem appropriate.

Worth Paying For?

Those who want to avoid ads, find a wider range of useful information or make sophisticated searches on the internet are increasingly being asked to pay. Online newspapers and magazines are one example: the free searches provided by search engines often link to pages available only on subscription.

Even though so much information is available free, it can be worth paying to use a specialist search service, says Clare Hart, the boss of Factiva, a web-based news and business-information service. A joint-venture between Dow Jones and Reuters, it has around 1.6m paying subscribers and draws on almost 9,000 sources in 22 different languages, many of which are subscription-only. The tools of the system are designed to allow more precise searches and to provide results in a single format.

Some companies use such services to provide virtual “clippings”—all the stories from newspapers and magazines that mention their firm or their product. But just think how much more valuable it would be to know in advance what might be written about you in future. A new type of search engine could get close with an automated process called data mining. WebFountain has been developed by IBM’s research centre at Almaden, California, to use text analysis as a way of identifying trends, patterns and relationships from massive amounts of both unstructured and semi-structured text, ranging from bulletin boards, chat rooms, web logs, newspapers and trade journals.

Factiva has licensed the technology to provide a new reputation-management service, launched this month. The idea, explains Ms Hart, is that, say, a brand manager will be able to see how his company is being perceived on the internet without having to keep checking hundreds of different sites, some of which he may not even know about. This could provide an early indication of trends and troubles.

Companies could also use such services to find out what their rivals are up to. Nor would they have to be big and powerful to do so. One of the most heartening aspects of e-commerce is that the little guys often have the edge.