• About
  • Policies
  • What is open access
  • Library
  • Contact
Advanced search
      View Item 
      •   BUIR Home
      • Scholarly Publications
      • Faculty of Engineering
      • Department of Computer Engineering
      • View Item
      •   BUIR Home
      • Scholarly Publications
      • Faculty of Engineering
      • Department of Computer Engineering
      • View Item
      JavaScript is disabled for your browser. Some features of this site may not work without it.

      Automatic ranking of information retrieval systems using data fusion

      Thumbnail
      View / Download
      455.0 Kb
      Author(s)
      Nuray, R.
      Can, F.
      Date
      2006-05
      Source Title
      Information Processing and Management
      Print ISSN
      0306-4573
      Publisher
      Elsevier Ltd
      Volume
      42
      Issue
      3
      Pages
      595 - 614
      Language
      English
      Type
      Article
      Item Usage Stats
      259
      views
      449
      downloads
      Abstract
      Measuring effectiveness of information retrieval (IR) systems is essential for research and development and for monitoring search quality in dynamic environments. In this study, we employ new methods for automatic ranking of retrieval systems. In these methods, we merge the retrieval results of multiple systems using various data fusion algorithms, use the top-ranked documents in the merged result as the "(pseudo) relevant documents," and employ these documents to evaluate and rank the systems. Experiments using Text REtrieval Conference (TREC) data provide statistically significant strong correlations with human-based assessments of the same systems. We hypothesize that the selection of systems that would return documents different from the majority could eliminate the ordinary systems from data fusion and provide better discrimination among the documents and systems. This could improve the effectiveness of automatic ranking. Based on this intuition, we introduce a new method for the selection of systems to be used for data fusion. For this purpose, we use the bias concept that measures the deviation of a system from the norm or majority and employ the systems with higher bias in the data fusion process. This approach provides even higher correlations with the human-based results. We demonstrate that our approach outperforms the previously proposed automatic ranking methods. © 2005 Elsevier Ltd. All rights reserved.
      Keywords
      Data fusion
      Experimentation
      Information retrieval
      Performance evaluation
      Rank aggregation
      Research and development management
      Information retrieval
      Permalink
      http://hdl.handle.net/11693/23803
      Published Version (Please cite this version)
      http://dx.doi.org/10.1016/j.ipm.2005.03.023
      Collections
      • Department of Computer Engineering 1561
      Show full item record

      Browse

      All of BUIRCommunities & CollectionsTitlesAuthorsAdvisorsBy Issue DateKeywordsTypeDepartmentsCoursesThis CollectionTitlesAuthorsAdvisorsBy Issue DateKeywordsTypeDepartmentsCourses

      My Account

      Login

      Statistics

      View Usage StatisticsView Google Analytics Statistics

      Bilkent University

      If you have trouble accessing this page and need to request an alternate format, contact the User and Access Services. Phone: (312) 290 1298
      © Bilkent University - Library IT

      Contact Us | Send Feedback | Off-Campus Access | Admin | Privacy