Word-based compression in full-text retrieval systems

Abstract

Large space requirement of a full-text retrieval system can be reduced significantly by data compression. In this study, the problem of compressing the main text of a full-text retrieval system is addressed and performance of several coding techniques for compressing the text database is compared. Experiments show that statistical techniques, such as arithmetic coding and Huffman coding, give the best compression among the implemented; and using a semi-static word-based model, the space needed to store English text is less than one third of the original requirement.