CoDet: Sentence-based containment detection in news corpora

Date
2011
Editor(s)
Advisor
Supervisor
Co-Advisor
Co-Supervisor
Instructor
Source Title
CIKM '11 Proceedings of the 20th ACM international conference on Information and knowledge management
Print ISSN
Electronic ISSN
Publisher
ACM
Volume
Issue
Pages
2049 - 2052
Language
English
Journal Title
Journal ISSN
Volume Title
Series
Abstract

We study a generalized version of the near-duplicate detection problem which concerns whether a document is a subset of another document. In text-based applications, document containment can be observed in exact-duplicates, near-duplicates, or containments, where the first two are special cases of the third. We introduce a novel method, called CoDet, which focuses particularly on this problem, and compare its performance with four well-known near-duplicate detection methods (DSC, full fingerprinting, I-Match, and SimHash) that are adapted to containment detection. Our method is expandable to different domains, and especially suitable for streaming news. Experimental results show that CoDet effectively and efficiently produces remarkable results in detecting containments. © 2011 ACM.

Course
Other identifiers
Book Title
Citation
Published Version (Please cite this version)