Automatic Ranking of Retrieval Systems in Imperfect Environments
Date
2003-07-08Source Title
SIGIR '03 Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Publisher
ACM
Pages
379 - 380
Language
English
Type
Conference PaperItem Usage Stats
217
views
views
235
downloads
downloads
Abstract
The empirical investigation of the effectiveness of information retrieval (IR) systems requires a test collection, a set of query topics, and a set of relevance judgments made by human assessors for each query. Previous experiments show that differences in human relevance assessments do not affect the relative performance of retrieval systems. Based on this observation, we propose and evaluate a new approach to replace the human relevance judgments by an automatic method. Ranking of retrieval systems with our methodology correlates positively and significantly with that of human-based evaluations. In the experiments, we assume a Web-like imperfect environment: the indexing information for all documents is available for ranking, but some documents may not be available for retrieval. Such conditions can be due to document deletions or network problems. Our method of simulating imperfect environments can be used for Web search engine assessment and in estimating the effects of network conditions (e.g., network unreliability) on IR system performance.
Keywords
Automatic Performance EvaluationIR Evaluation
Automation
Computer simulation
Correlation methods
Database systems
Query languages
Search engines
World Wide Web
Automatic performance evaluation
Information retrieval (IR) evaluation
Information retrieval systems