Data-parallel web crawling models
Date
2004Source Title
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Print ISSN
0302-9743 1611-3349
Publisher
Springer
Volume
3280
Pages
801 - 809
Language
English
Type
ArticleItem Usage Stats
195
views
views
196
downloads
downloads
Abstract
The need to quickly locate, gather, and store the vast amount of material in the Web necessitates parallel computing. In this paper, we propose two models, based on multi-constraint graph-partitioning, for efficient data-parallel Web crawling. The models aim to balance the amount of data downloaded and stored by each processor as well as balancing the number of page requests made by the processors. The models also minimize the total volume of communication during the link exchange between the processors. To evaluate the performance of the models, experimental results are presented on a sample Web repository containing around 915,000 pages. © Springer-Verlag 2004.
Keywords
Artificial intelligenceComputers
Data parallel
Multi-constraints
Web Crawling
Web repositories
Parallel processing systems