Automatic ranking of information retrieval systems using data fusion

Simple item page
dc.citation.epage614en_US
dc.citation.issueNumber3en_US
dc.citation.spage595en_US
dc.citation.volumeNumber42en_US
dc.contributor.authorNuray, R.en_US
dc.contributor.authorCan, F.en_US
dc.date.accessioned2016-02-08T10:19:25Z
dc.date.available2016-02-08T10:19:25Z
dc.date.issued2006-05en_US
dc.departmentDepartment of Computer Engineeringen_US
dc.description.abstractMeasuring effectiveness of information retrieval (IR) systems is essential for research and development and for monitoring search quality in dynamic environments. In this study, we employ new methods for automatic ranking of retrieval systems. In these methods, we merge the retrieval results of multiple systems using various data fusion algorithms, use the top-ranked documents in the merged result as the "(pseudo) relevant documents," and employ these documents to evaluate and rank the systems. Experiments using Text REtrieval Conference (TREC) data provide statistically significant strong correlations with human-based assessments of the same systems. We hypothesize that the selection of systems that would return documents different from the majority could eliminate the ordinary systems from data fusion and provide better discrimination among the documents and systems. This could improve the effectiveness of automatic ranking. Based on this intuition, we introduce a new method for the selection of systems to be used for data fusion. For this purpose, we use the bias concept that measures the deviation of a system from the norm or majority and employ the systems with higher bias in the data fusion process. This approach provides even higher correlations with the human-based results. We demonstrate that our approach outperforms the previously proposed automatic ranking methods. © 2005 Elsevier Ltd. All rights reserved.en_US
dc.identifier.doi10.1016/j.ipm.2005.03.023en_US
dc.identifier.issn0306-4573
dc.identifier.urihttp://hdl.handle.net/11693/23803
dc.language.isoEnglishen_US
dc.publisherElsevier Ltden_US
dc.relation.isversionofhttp://dx.doi.org/10.1016/j.ipm.2005.03.023en_US
dc.source.titleInformation Processing and Managementen_US
dc.subjectData fusionen_US
dc.subjectExperimentationen_US
dc.subjectInformation retrievalen_US
dc.subjectPerformance evaluationen_US
dc.subjectRank aggregationen_US
dc.subjectResearch and development managementen_US
dc.subjectInformation retrievalen_US
dc.titleAutomatic ranking of information retrieval systems using data fusionen_US
dc.typeArticleen_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Automatic ranking of information retrieval systems using data fusion.pdf
Size:
455 KB
Format:
Adobe Portable Document Format
Description:
Full printable version
Download

Collections

Department of Computer Engineering