OpenZL 0.3 decompresses 144% faster than Zstd

Meta's compression framework has been rebuilt where it matters. OpenZL 0.3 shipped on 29 September with a reworked LZ engine that its release notes say decompresses 144% faster than Zstandard at equivalent settings.

The gain comes from the entropy coder underneath the engine. OpenZL has adopted PivCo Huffman, a Huffman layout published by Marcin Żukowski that is designed around SIMD decoding, and the LZ graph now uses it as its entropy backend. Against version 0.2 at compression level 1 with a 64KB window, decompression improved by a further 33%.

The numbers, side by side

The release notes publish the comparison at those settings. OpenZL's LZ engine in 0.3 reaches 2.73x compression while decompressing at 3062 MB/s; version 0.2 managed 2288 MB/s at the same 2.74x, and Zstandard reaches 1254 MB/s at 2.74x. Compression speed barely moved — 467 MB/s against Zstandard's 419 MB/s — so this is a decompression release, and the framework now supports compression levels equivalent to Zstandard's 1 through 7 with window sizes up to 256 MiB. Meta also says some configurations now decompress faster than LZ4 at similar ratios, and a new trainer can tune an LZ profile to a specific data set and print the speed-versus-ratio frontier.

A neural selector picks the codec

The second headline change is the Compression Transformer, which builds a numeric compression graph for each input on the fly instead of requiring a codec to be chosen or trained in advance. A small neural network scores candidate codecs stream by stream until the graph is complete, and nothing changes on the decompression side. Evaluated across 868 families of numeric streams — 34,737 files and 17.9GB — Meta reports it compresses 34.9% better than zstd -19 on average and lands within 1.1% of graphs that were specifically trained for the data.

The Transformer switches on at compression level 7 and above; the default level 6 is unchanged, which means existing pipelines keep their old behaviour until someone opts in.

What else moved

Elsewhere in 0.3: a codec for numeric streams dominated by zeros, trained Zstandard dictionaries that travel with the graph as a bundle, ARM NEON and SVE2 kernels with CI coverage, an xxHash bump, and roughly 447KiB off the binaries after the removal of an older numeric model. Upgraders should note the frame format's maximum version moves to 27 from 24, so producing frames that version 0.2 can still read needs the old version to be requested explicitly.

Our opinion

Compression libraries are judged on a boring axis — how much they cost when they are switched on — and OpenZL's problem has always been the opposite of Zstandard's. Zstandard is fast, predictable and everywhere; OpenZL trades some of that predictability for ratios it can tune per data format, which is a hard sell until the tuning is automatic. A selector that gets within 1.1% of a hand-trained graph and needs no per-source work is the feature that changes that argument. The decompression jump matters just as much, because the read path is where storage and network systems actually feel the cost — and 144% is not a marginal improvement to a database or a log pipeline.