CoDet: Sentence-based containment detection in news corpora
Author
Varol, Emre
Can, Fazlı
Aykanat, Cevdet
Kaya, Oğuz
Date
2011Source Title
CIKM '11 Proceedings of the 20th ACM international conference on Information and knowledge management
Publisher
ACM
Pages
2049 - 2052
Language
English
Type
Conference PaperItem Usage Stats
110
views
views
80
downloads
downloads
Abstract
We study a generalized version of the near-duplicate detection problem which concerns whether a document is a subset of another document. In text-based applications, document containment can be observed in exact-duplicates, near-duplicates, or containments, where the first two are special cases of the third. We introduce a novel method, called CoDet, which focuses particularly on this problem, and compare its performance with four well-known near-duplicate detection methods (DSC, full fingerprinting, I-Match, and SimHash) that are adapted to containment detection. Our method is expandable to different domains, and especially suitable for streaming news. Experimental results show that CoDet effectively and efficiently produces remarkable results in detecting containments. © 2011 ACM.
Keywords
Corpus treeDocument containment
Duplicate detection
Similarity
Test Collection
Knowledge management
Software agents