Yahoo! Birth of a New Search Engine

Yahoo is rolling out a brand new search engine today, with its own index and ranking mechanisms, casting aside its long-standing use of Google-powered search results. The move is bound to roil the industry and sets in motion a new race for the claim of web search champion.

Ever since Yahoo's acquisition of Inktomi nearly a year ago, speculation has focused on when the company would replace its Google powered search results with those from the Inktomi index.

In a surprising move, Yahoo isn't replacing Google with Inktomi. Rather, the company has developed a brand new search engine, drawing on the lessons learned from what the company calls the "critical mass" of search engineering talent that it has brought together through hiring and acquisitions, as well as investment in infrastructure and product quality.

"High quality, talented search engineers are in very short supply these days," said Jeff Weiner, Yahoo's senior vice president of search and marketplace. "Regardless of how good your planning process is, at the end of the day it comes down to people and chemistry."

Weiner said Yahoo has waited until now to make the switch from Google to be certain users would have the best experience possible after the transition. "It was absolutely essential to us that we had a roadmap in place that not only let us sustain our quality, but build on it."

Although the change to self-powered search results is a radical change, Yahoo has steadily made incremental improvements in its search capabilities for more than a year. In October 2002, the company made the most significant change to its operation since its birth, replacing its human-compiled directory listings with Google search results.

Then in April of last year, the company rolled out its new Yahoo search, introducing a streamlined search page. It also added new tabs to search result pages offering access to its directory listings, news, images, and yellow pages.

Today's launch is the beginning of a progressive rollout that will take place over the next few weeks. It is also the beginning of numerous planned enhancements focusing on web search, personalization and vertical search.

It's important to note that the new search engine is for web results only. Image search is still powered by Google, and News search is still a combination of Yahoo's own editorial and technological resources.

How does the new Yahoo search engine differ from Google? The presentation of the results is very similar. Yahoo has wisely opted to keep things looking mostly the same, with a few exceptions. There's a linked to the cached copy of each indexed page -- now being served from Yahoo, not Google. Just about everything else on search result pages looks the same.

The actual results returned by Google and Yahoo depends on the query. For popular or common queries, there seemed to be very little difference between the two engines in top few results. But once you get past those, the results tend to diverge dramatically. And for less common or non-popular queries, Yahoo results look quite different from Google results.

While Yahoo and Google are likely using similar algorithms, one reason for the differences in what's displayed is that Yahoo's email and search teams are now working together to leverage what they've learned about spam. Since Yahoo mail processes billions of email messages, this knowledge is likely quite helpful in providing Yahoo with a much deeper understanding of the characteristics of spam -- and helping keep the nasty stuff out of the web page index.

Bottom line: I'm impressed with the quality of results that Yahoo is delivering. It's a very viable alternative to Google and the other "last engine standing," Ask Jeeves/Teoma.