Document replication strategies for geographically distributed web search engines

Date
2013
Advisor
Instructor
Source Title
Information Processing & Management
Print ISSN
0306-4573
Electronic ISSN
Publisher
Elsevier Ltd.
Volume
49
Issue
1
Pages
51 - 66
Language
English
Type
Article
Journal Title
Journal ISSN
Volume Title
Abstract

Large-scale web search engines are composed of multiple data centers that are geographically distant to each other. Typically, a user query is processed in a data center that is geographically close to the origin of the query, over a replica of the entire web index. Compared to a centralized, single-center search engine, this architecture offers lower query response times as the network latencies between the users and data centers are reduced. However, it does not scale well with increasing index sizes and query traffic volumes because queries are evaluated on the entire web index, which has to be replicated and maintained in all data centers. As a remedy to this scalability problem, we propose a document replication framework in which documents are selectively replicated on data centers based on regional user interests. Within this framework, we propose three different document replication strategies, each optimizing a different objective: reducing the potential search quality loss, the average query response time, or the total query workload of the search system. For all three strategies, we consider two alternative types of capacity constraints on index sizes of data centers. Moreover, we investigate the performance impact of query forwarding and result caching. We evaluate our strategies via detailed simulations, using a large query log and a document collection obtained from the Yahoo! web search engine. (C) 2012 Elsevier Ltd. All rights reserved.

Course
Other identifiers
Book Title
Keywords
Web search, Distributed information retrieval, Document replication, Query processing, Query forwarding, Result caching, Object replication, Retrieval systems, Database - systems, Data allocation, Algorithms, Placement, Performance, Networks, Servers
Citation
Published Version (Please cite this version)