Data-parallel web crawling models

Date
2004
Editor(s)
Advisor
Supervisor
Co-Advisor
Co-Supervisor
Instructor
Source Title
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Print ISSN
0302-9743
1611-3349
Electronic ISSN
Publisher
Springer
Volume
3280
Issue
Pages
801 - 809
Language
English
Journal Title
Journal ISSN
Volume Title
Series
Abstract

The need to quickly locate, gather, and store the vast amount of material in the Web necessitates parallel computing. In this paper, we propose two models, based on multi-constraint graph-partitioning, for efficient data-parallel Web crawling. The models aim to balance the amount of data downloaded and stored by each processor as well as balancing the number of page requests made by the processors. The models also minimize the total volume of communication during the link exchange between the processors. To evaluate the performance of the models, experimental results are presented on a sample Web repository containing around 915,000 pages. © Springer-Verlag 2004.

Course
Other identifiers
Book Title
Citation
Published Version (Please cite this version)