Marilyn Rosenthal
Library Reference Department, Long Island University
Brookville, New York
In order to overcome this difficulty in retrieving information from WWW, more than two dozen companies and institutions quickly developed various search aids [2] such as Lycos and Excite. However, since there are usually only one or two search aids for other Internet applications (e.g., Archie for FTP, and Veronica for Gopher), why have at least two dozen search engines been developed for the Web so far? The sheer number invites research. For instance, what features do various Web search engines offer? How do they differ from one another in performance? Is there a single Web search engine that out-performs all others in information retrieval? The current study attempts to seek answers to those questions.
Eventually, people went a step further by starting to evaluate Web search engines in addition to describing them. Notess examined Lycos, WebCrawler, World-Wide Web Worm, Harvest Broker, CUI, and CUSI in one article (1995a) and InfoSeek in another (1995b). Based on online documentation provided by those Web search engines and personal usage, Notess recommended that "for single keyword searches of a large database, use Lycos". "For multiword searches with an AND, try WebCrawler". "For a time-consuming comprehensive search, use CUSI". In addition, Notess also compared InfoSeek with Lycos and WebCrawler in terms of coverage, precision, and currency.
In a more recent publication, Courtois, Baer, and Stark (1995) evaluated the performances of about 10 different Web search aids including CUI, Harvest, Lycos, Open Text, World-Wide Web Worm, and Yahoo. Using 3 sample search questions along with other information available about the search engines, the authors concluded that, among other things, Open Text was the best at the time of their study "with its flexible, powerful search interface and quick response". They also concluded that "For novices, WebCrawler offers the easiest interface". In a different study, Scoville (1996) surveyed a wide range of Web search engines, and suggested that Excite, InfoSeek, and Lycos should be added to one's list of favorites because they can retrieve "accurate results from easy-to-use interfaces".
Leighton (1995) did a study of Web search engines for course work, actually employing the evaluation criterion of precision. The findings were not submitted to a journal for publication because of the fast changing nature of the search engines. Leighton evaluated InfoSeek, Lycos, WebCrawler and World-Wide Web Worm using 8 reference questions from a university library as search queries. The author found that "Lycos and the free part of InfoSeek have about the same precision with Lycos just a nose ahead" while WebCrawler gave "surprisingly bad precision". "WWWWorm was good enough that usually retrieved at least one or two hits" for the given queries with high precision.
Kimmel (1996) examined World Wide Web Worm, Lycos, WebCrawler, Open Text, Jumpstation II, AliWeb, and Harvest based on documentation provided by the search engines along with a couple of single word test searches (e.g., pollution, ebola). The author's focus was, like many other publications, on describing the features of these various search engines even though the number of hits produced by test searches were also listed. The author, in summary, indicated that "Of the robot-generated databases presented here, Lycos appears to be the strongest system overall".
c|net, a company specialized in evaluating online products and services, distributed a comparative study of 19 Web search engines on its Web site (Leonard, 1996). The search engines were tested on their accuracy of results, ease of use and provision of advanced options using 15 queries specifically composed for the evaluation. Most of the queries resemble reference questions asked in public libraries. According to the two feature tables generated by the evaluation, Alta Vista seems to be the best choice among individual search engines, while All-in-One Search Page and the Internet Sleuth achieved the highest ranking for meta- or unified search engines.
The reported findings obviously do not appear to agree with one another. The methodologies and evaluation criteria used by those studies differed as well. Can a feasible methodology be developed to help Web users select a search engine, out of the great number of choices, that is most appropriate to their specific search needs? The authors of this study are trying to do so by first evaluating the searching capabilities and performance of selected Web search engines currently available.
We are also aware that many search engines index not only Web information but also resources stored on other Internet applications such as discussion groups and Gopher. But, we chose to consider only Web databases to be consistent with the objective of our study. Moreover, we did not cover unified Web search engines such as CUSI (Configurable Unified Search Index, http://pubweb.nexor. co.uk/public/cusi/doc/list.html) since search tools of that kind do not provide anything new except putting together existing individual ones. Although some of them (e.g., MetaCrawler) have added such new features as removing duplicates, their searching mechanism remains the same.
In addition, most of the Web search engines are available to users free of charge. It seems that these free services will continue to be available to the Internet community in the foreseeable future. Given the fact that users will naturally choose search engines that can be accessed at no cost to them, our study excludes fee-based Web search services such as InfoSeek even though we understand that it may indeed perform well in retrieving Web resources.
During the process of selecting Web search engines to be evaluated, we paid particular attention to covering those representing diversity so that our choices would comprise different types of Web search engines. We applied the same criterion in choosing sample queries for our performance evaluation. Sample search queries were drawn from real reference questions.
With selected search engines, we compared their search capabilities such as Boolean logic, truncation, field searching, and word/phrase searching. Furthermore, we also evaluated the performance of the selected search engines with respect to precision and response time. Recall, the other commonly used evaluation criterion for information retrieval performance, was deliberately omitted from this study because it is impossible to determine how many relevant items there are for a particular query in the huge and ever-changing Web system. The ultimate goal of this study is, as stated earlier, to develop a feasible methodology for evaluating all Web search engines.
It indexes the full text of over 16,000,000 Web pages (by January 1996) with unspecified update frequencies. According to its documentation, Alta Vista can fetch 2.5 million pages a day following the Robots Exclusion Standard, and index 1 GB of text per hour. Alta Vista supports Boolean searching, term as well as phrase searching (i.e., proximity searching with the NEAR operator), field searching (eg, title:steelhead; url:home.html), right-hand truncation with some restriction, and case-sensitive searching if only the first letter of a word is capitalized.
Alta Vista provides three display options: compact, standard, and detailed although the latter two are the same. The display order or relevancy ranking of search results is determined by the location (e.g., in title or the body of text) of matching words, occurrence frequencies of matching words, and distance (i.e., how many words apart) between the matching words. However, only the first few words of a document found are displayed, which may limit users' ability to judge its relevancy without referring to the full version of the document. In addition, general search terms such as "computer" and "analysis" are automatically ignored in Alta Vista.
Excite allows keyword searching as well as concept searching since it is able to determine related concepts from document collections, eliminating the need for external manually-defined representations such as thesauri. An example of concept searching given by Excite is that a search query about "intellectual property rights" will retrieve all documents about the topic even if terms such as "software piracy" or "copyright law" rather than the actual matching words appear in the document. In other words, the search engine itself handles synonyms and related terms, taking the burden of vocabulary control off users' shoulders. As for keyword search, query terms are both AND'ed and OR'ed in each search, but a higher weight is given to results with terms AND'ed. However, Excite does not support at present other advanced search options than those being described already.
Equipped with automatic abstracting capability, Excite is able to generate an abstract for each of the Web pages it indexes, which is a very unique and fine feature that many of its counterparts do not have. But, there are no different formats for displaying search results. In addition, its online documentation appears somewhat unorganized.
By the end of January 1996, Lycos has indexed over 95% (ca. 19 million unique URLs including FTP and Gopher) of Web resources, making it the largest Web search engine in its family. Nevertheless, it does not index the full text of a Web page. Rather, it only extracts the title and a portion of a document (e.g., the smaller of the first 20 lines or 20% of the document). This practice has been singled out by Lycos' competitors as its most salient weakness. Around 50,000 documents are added, deleted, or updated in the Lycos index everyday.
Lycos supports Boolean logic, and furthermore, it incorporates that feature in such a way that the users do not have to type the Boolean operators when conducting a search. For example, one only needs to select the search option "Match all terms (AND)" to use the AND operator. Another search feature Lycos provides is to match query terms against Web documents at 5 different levels, namely, Loose match, Fair match, Good match, Close match, Strong match. Nevertheless, no specific explanation is given as to how the different levels of match are determined. Truncation is automatically done in Lycos during a search, which may result in some unwanted search outcome. Phrase search is not supported by Lycos so any queries with phrases cannot be appropriately executed.
On the other hand, Lycos implements a wide variety of display options. Users are given the choices of viewing 10, 20, 30, or 40 research results a time. In addition, each search result can be displayed using the summary, standard, or detailed format. The detailed format corresponds with the long abstracts Lycos prepares, which include URL, title, outline, keys, abstract, description, date, and other related information. The summary format contains what Lycos' short abstracts have: URL and descriptions. In terms of coverage, the standard format lies somewhere between the summary and detailed formats. The online documentation available at Lycos' Web site describes the composition of each output segment (e.g., outline and keys) in detail.
In summary, the three different Web search engines show diversity in their search capabilities, user interface, and quality of documentation. The next section of this paper will discuss the performance evaluation of the selected Web search engines.
| #1 | Alta Vista: volunteerism +society |
| Excite: volunteerism society | |
| Lycos: volunteerism society | |
| #2 | Alta Vista: "classical Greek philosophy" |
| Excite: classical Greek philosophy | |
| Lycos: classical Greek philosophy | |
| #3 | Alta Vista: memory +neurobiology |
| Excite: memory neurobiology | |
| Lycos: memory neurobiology | |
| #4 | Alta Vista: "sexual difference*" +"mathematical ability" |
| Excite: sexual differences mathematical ability | |
| Lycos: sexual differences mathematical ability | |
| #5 | Alta Vista: "psychological analysis" +"British artist" +"Francis Bacon" |
| Excite: British artist Francis Bacon | |
| Lycos: British artist Francis Bacon | |
| #6 | Alta Vista: violence +athlete* |
| Excite: violence athletes | |
| Lycos: violence athletes | |
| #7 | Alta Vista: computers +"learning disabilit*" |
| Excite: computers learning disabilities | |
| Lycos: computer learning disabilities | |
| #8 | Alta Vista: NAFTA |
| Excite: NAFTA | |
| Lycos: NAFTA | |
| #9 | Alta Vista: plagiarism |
| Excite: plagiarism | |
| Lycos: plagiarism | |
| #10 | Alta Vista: title:"Long Island University" |
| Excite: Long Island University | |
| Lycos: Long Island University |
Whenever there are different search options available (e.g., Alta Vista's simple search & advanced search), the simple mode is used in order to relate the findings of this study to those with little searching background. In the case of Lycos, the "Loose match" and "Match all terms (AND)" options were selected for all the queries. The latter decision was based on the rationale that none of the ten questions entails the use of other listed choices such as "Match any term (OR)" and "Match 2 terms". As for the display options, the most detailed one available is always favored since this option would provide us with more information for evaluation.
Due to the time factor, we only examined up to 10 Web records [4] for each query. As all the selected search engines display results in descending order of relevance calculated one way or another, we believe that this should not critically affect the validity of our study.
While Alta Vista and Excite always retrieved at least 10 Web records for each query, Lycos sometimes could not find anything at all on some of the topics (e.g., #4 & #5). We have omitted listing the total number of Web records retrieved by the three search engines since Excite, unlike Alta Vista and Lycos, does not provide that figure up front, and one must trace down along the path of "Next Documents" in order to get the final count.
In comparison, Alta Vista obtained the highest precision score (0.78) among the three Web search engines while Lycos had 0.55 and Excite got the lowest one (0.45). There are however very few duplicates in some 250 Web records we perused, which suggests that the constructions (e.g., spiders and indexes) of each Web search engine are diversified enough that they consequently represent different portions of the entire magnificent Web system.Table 1 Precision (P) Chart for Three Web Search Engines
Sample Query Alta Vista Excite Lycos Mean P Sum P/# P Sum P/# P Sum P/# P #1 8.5/10 0.85 8.0/10 0.8 2.0/3 0.67 0.77 #2 8.0/10 0.8 6.0/10 0.6 4.0/4 1.0 0.8 #3 10.0/10 1.0 7.5/10 0.75 9.0/10 0.9 0.88 #4 3.5/10 0.35 1.0/10 0.1 0 0 0.2 #5 1.0/10 0.1 3.0/10 0.3 0 0 0.13 #6 9.0/10 0.9 0.5/10 0.05 4.5/10/ 0.45 0.47 #7 7.5/10 0.75 3.0/10 0.3 0 0 0.35 #8 10.0/10 1.0 10.0/10 1.0 1.0/10 1.0 1.0 #9 10.0/10 1.0 6.0/10 0.6 8.0/8 [6] 1.0 0.87 #10 10.0/10 1.0 0/10 0 1.0/2 0.5 0.5 Mean P N/A 0.78 N/A 0.45 N/A 0.55 0.59
On the other hand, it is evident from Table 1 that some search queries (e.g., #4 & #5) are not really suitable for searching on the Web because of the complicated nature of those questions and the fact that sophisticated facilities such as proximity searching are yet to be developed for Web search engines.
While Excite does not support truncation, Lycos automatically truncates every possible query term (e.g., "violence" was truncated as "violence", "violenced", "violenceo", and "violences"), which inevitably brings a lot of noise into search results. Although case-sensitivity is not a significant feature for search tools, only Alta Vista offers it on a limited scale.
We deliberately tested the concept searching capability of Excite, and the results were satisfactory. On the other hand, the field searching feature provided by Alta Vista is quite unique among Web search engines, which partially explains why it did the best in Query #10 -- Long Island University, while Excite could only retrieve results about Long Island and nothing about Long Island University in its first 10 records displayed.
Among the three search engines compared, Lycos presented the greatest amount of information in a Web record including URL, outline, keys, abstract, description, and other related data. Nevertheless, a closer look at the contents reveals that a lot of the information displayed is either redundant or of little practical value. For instance, an outline always repeats the title of a Web document, and an abstract consists of the first 20 lines or 20% of the document, whichever is smaller.
In a word, Alta Vista should be the first choice, among the three Web search engines evaluated, for users who expect search results of high precision. Otherwise, the selection of a search engine for Web navigation can be based on personal preference for the documentation, interface, or other features specific to it.
Table 2 An Evaluation Methodology of Web Search Engines
Evaluation Criteria Web Search Engine Choice Alta Vista Excite Lycos Your Choice Web Indexes Coverage [7] 16 Mil. 1.5 Mil. 19 Mil. Update Unspecified Weekly Weekly Web page indexed Partial Full Partial Search Capability Boolean search Yes Yes Yes Proximity search Yes No No Truncation Limited No Automatic Field search Yes No No Case-sensitivity Limited No No Concept search No Yes No Retrieval Performance Precision 0.78 0.45 0.55 Recall Not Tested Not Tested Not Tested Response time < 3 sec. 3-5 sec. About 3 sec. Output # of formats 2 1 3 Content Extract Abstract Extract User Effort Documentation Good Poor Very good Interface Fair Good Good
In the future, we plan to apply the proposed methodology to a wider scope with the hope that our research findings will truly enable Web users to select a search engine appropriate to their specific search needs, and help Web search engine developers design even better ones for the Internet community.
Harman, Donna. (1995). Overview of the second Text REtrieval Conference (TREC-2). Information Processing & Management, 31(3), 271-289.
Lancaster, F.W., and Fayen, E.G. (1973). Information Retrieval On-Line Los Angeles, CA: Melville Publishing Co. Chapter 6.
Leighton, H. Vernon. (1995). Performance of four World Wide Web (WWW) index services: InfoSeek, Lycos, WebCrawler, and WWWWorm. http://www.winona.msus.edu/ services-f/library-f/webind.htm.
Leonard, Andrew J. (1996). Where to find anything on the net. http://www.cnet.com/ Content/Reviews/Search/.
Notess, Greg R. (July/August 1995a). Searching the World-Wide Web: Lycos, WebCrawler and More. Online, 19(4), 48-53.
Notess, Greg R. (August/September 1995b). The InfoSeek Databases. Database, 85-87.
Scoville, Richard. (January 1996). Special report: Find it on the net! PC World, also available at http://www.lycos.com.
Shirky, Clay. (October, 1995). Finding needles in haystacks. Netguide, 87-90.
Taubes, Gary. (September 8, 1995). Indexing the Internet. Science, 269, 1354-1356.
Wildstrom, Stephen H. (September 11, 1995). Feeling your web around the Web. Business Week, 22.
2. Such Web search aids take assorted names. Catalogs, indexes, directories, and search engines are some of the examples.
3. We noticed that the first letter of "Excite" should not be capitalized according to the practice of its developers. But we have intentionally altered the spelling to avoid any confusion that may arise with the actual word "excite".
4. We define a Web record as all the information displayed for a retrieved Web document. We understand that the contents of Web records differ from one search engine to another.
5. Precision scores were assigned to each retrieved item using a three-level (1 for relevant, 0.5 for somewhat relevant, and 0 for irrelevant) scoring method. The differences between the two judgements were averaged whenever needed.
6. Out of the 10 downloaded Web records, there are 2 duplicates.
7. All the figures were obtained in January 1996.
© 1996, American Society for Information Science. Permission to copy and distribute this document is hereby granted provided that this copyright notice is retained on all copies and that copies are not altered.