- Home
- Screenshots
- Downloads
- Documentation

# Recoll: Indexing performance and index sizes

The time needed to index a given set of documents, and the resulting index size depend of many factors, such as file size and proportion of actual text content for the index size, cpu speed, available memory, average file size and format for the speed of indexing.

We try here to give a number of reference points which can be used to roughly estimate the resources needed to create and store an index. Obviously, your data set will never fit one of the samples, so the results cannot be exactly predicted.

The following data was obtained on a machine with a 1800 Mhz AMD Duron CPU, 768Mb of Ram, and a 7200 RPM 160 GBytes IDE disk, running Suse 10.1.

recollindex (version 1.8.2 with xapian 1.0.0) is executed with the default flush threshold value. The process memory usage is the one given by ps

Data | Data size | Indexing time | Index size | Peak process memory usage
--- | --- | --- | --- | ---
Random pdfs harvested on Google | 1.7 GB, 3564 files | 27 mn | 230 MB | 225 MB
Ietf mailing list archive | 211 MB, 44,000 messages | 8 mn | 350 MB | 90 MB
Partial Wikipedia dump | 15 GB, one million files | 6H30 | 10 GB | 324 MB
Random pdfs harvested on Google Recoll 1.9, idxflushmb set to 10 | 1.7 GB, 3564 files | 25 mn | 262 MB | 65 MB

Notice how the index size for the mail archive is bigger than the data size. Myriads of small pure text documents will do this. The factor of expansion would be even much worse with compressed folders of course (the test was on uncompressed data).

The last test was performed with Recoll 1.9.0 which has an ajustable flush threshold (idxflushmb parameter), here set to 10 MB. Notice the much lower peak memory usage, with no performance degradation. The resulting index is bigger though, the exact reason is not known to me, possibly because of additional fragmentation