Gonçalo Cabrita

Fast data algorithms

#algorithms

TL;DR

Application Common Bad Performance Choices Better Performance Choices Expected Performance Gain
Trusted data hashing md5, sha2, crc32 xxhash ~10x
Untrusted data hashing md5, sha2, sha1 blake3 ~10x
Fast compression snappy, gzip (zlib) lz4 10x over gzip, ~2x over snappy
Good compression gzip (zlib) zstd ~2-10x
Best compression xz (lzma) zstd -10+ ~2-10x
Java crypto (md5, aes-gcm, etc …) Built-in JVM crypto Amazon Corretto Crypto Provider (ACCP) ~4-10x

Sources