Efficient community identification and maintenance at multiple resolutions on distributed datastores
Date
2015Source Title
Data & Knowledge Engineering
Print ISSN
0169-023X
Publisher
Elsevier BV
Volume
100
Pages
133 - 147
Language
English
Type
ArticleItem Usage Stats
142
views
views
92
downloads
downloads
Abstract
The topic of network community identification at multiple resolutions is of great interest in practice to learn high cohesive subnetworks about different subjects in a network. For instance, one might examine the interconnections among web pages, blogs and social content to identify pockets of influencers on subjects like 'Big Data', 'smart phone' or 'global warming'. With dynamic changes to its graph representation and content, the incremental maintenance of a community poses significant challenges in computation. Moreover, the intensity of community engagement can be distinguished at multiple levels, resulting in a multi-resolution community representation that has to be maintained over time. In this paper, we first formalize this problem using the k-core metric projected at multiple k-values, so that multiple community resolutions are represented with multiple k-core graphs. Recognizing that large graphs and their even larger attributed content cannot be stored and managed by a single server, we then propose distributed algorithms to construct and maintain a multi-k-core graph, implemented on the scalable Big Data platform Apache HBase. Our experimental evaluation results demonstrate orders of magnitude speedup by maintaining multi-k-core incrementally over complete reconstruction. Our algorithms thus enable practitioners to create and maintain communities at multiple resolutions on multiple subjects in rich network content simultaneously.
Keywords
Big data analyticsDistributed databases
k-Core
Algorithms
Global warming
Mining
Smartphones
Social networking
Websites
Community identification
Data analytics
HBase
Mining methods and algorithms
Big data