Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation.

A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.



Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27...

So interesting if they implemented and tested that and get only a marginal CRC speedup.

edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739


igzip is even faster. See also my answer that contains a quick benchmark in that linked question at the bottom.


I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: