lemire/JavaFastPFOR ↗

About lemire/JavaFastPFOR

JavaFastPFOR is a Java library designed to compress and decompress arrays of integers at extremely high speeds. It achieves decompression rates exceeding 1.2 billion integers per second (4.5 GB/s) and is significantly faster than generic codecs like Snappy and LZ4 when working with integer data. The library is particularly effective for arrays where most values use less than 32 bits or where gaps between integers are small, making it ideal for scenarios involving differential coding in databases and information retrieval systems such as inverted indexes and column stores.

The library has seen significant real-world adoption in major projects including LinkedIn Pinot, Apache Parquet, Apache Lucene, Apache NiFi, and the Terrier search engine. It is a Java port of the FastPFor C++ library and has inspired compression schemes used across multiple systems. The project includes comprehensive documentation, benchmarking tools, and is available through standard dependency management systems like Maven and Gradle. Current development versions require JDK 21 or better, though older releases supported Java 7.

The underlying research behind JavaFastPFOR has been published in several peer-reviewed papers on SIMD-based compression techniques and vectorized integer decoding, with the work recognized at top conferences like ECIR 2014. The library uses multiple compression codecs optimized for different data patterns, and most codec instances are thread-safe, though the documentation recommends using one codec per thread to avoid contention.