Index Compression in Information Retrieval

38,323

38,365

To keep it in memory and for competition with other applications.

<p>Most of the bytes in the term column are wasted due to the fixed width allocation.</p> Signup and view all the answers

<p>The dictionary terms are stored as one long string of characters, with term pointers marking the end of the preceding term and the beginning of the next. The space requirement is 7.6MB.</p> Signup and view all the answers

<p>To make it small enough to keep in main memory and to reduce disk space needed for the postings file.</p> Signup and view all the answers

<p>Lossy compression discards some information, while lossless compression preserves all information.</p> Signup and view all the answers

<p>Heaps' law is represented by M = kT^b, where M is the size of the vocabulary and T is the number of tokens in the collection. It indicates that the size of the vocabulary grows with the collection size.</p> Signup and view all the answers

<p>Typical values for the parameters k and b are: 30 ≤ k ≤ 100 and b ≈ 0.5.</p> Signup and view all the answers

<p>The vocabulary will keep growing with the collection size, and there is no fixed upper bound due to the nature of the language and the increasing collection size.</p> Signup and view all the answers