I can see why the lzip author frustrated at the xz format, especially given that there are so many checksums and paddings around, but the lzip format is the opposite extreme.
7-zip already had a concept of multiple filters which contribute to its efficiency, and the underlying design of xz does capture them without much complication. For example, filters in the original 7-zip format (or "codecs") can have both multiple input and output streams [1]. This makes less sense for a single file compressor and xz carefully avoided them. The main problem with the xz format is not its concept but more about its concrete implementation: you don't need extensibility, you only need agility.
In comparison, lzip is too minimal. It might be technically agile by its version field, but it wouldn't if you do nothing and claim that you are open to any addition. It is not hard to pick some filters and mandate only most useful combinations of filters. The stream could have been periodically interrupted to give an early chance to detect errors before the member footer. (Unless lzip natively produces a multimember file even for a single input, which is AFAIK not the case.) The lzip author claims that a corruption in the compressed data can be detected from the decompression process, but that would mean too much redundancy in the compressed data, so this claim has been clearly misguided. And what the heck is that dictionary size coding? Compressed formats frequently make use of exponent-mantissa encodings but I have never seen an encoding where the mantissa is subtracted.
Of course, both should be avoided at this point because zstd is fast and efficient enough. Also, the file format for zstd is better than both in my opinion.
(I've posted the same comment in the older thread, and I also posted my summary of all three file formats so that you can feel what I'm talking about: https://news.ycombinator.com/item?id=39873112)
[1] https://py7zr.readthedocs.io/en/latest/archive_format.html#c...
afaik, 7-zip filters can't have multiple inputs (at the encoding stage).
multiple outputs are necessary for filters that output multiple independent data streams such as bcj2. and they are equally useful for archivers and compressors.
(I'm author of freearc, another archiver software, and multiple compression algos)
PS: thank you for format comparison, it would be great to put xz format description onto its Wikipedia page. I already used you description to understand why attackers added 8 "random" bytes to one of their scripts - probably to "fix" crc-64 value.
(Welcome to the HN, by the way! While I have also written my own compressor, I'm just a hobbyist compared to people in encode.su, to be sure :-)
> afaik, 7-zip filters can't have multiple inputs (at the encoding stage).
I don't know whether this is actually used or not, but py7zr's specification clearly mentions that complex coders have `NumInStreams` and `NumOutStreams` parameters specified.
> multiple outputs are necessary for filters that output multiple independent data streams such as bcj2. and they are equally useful for archivers and compressors.
Correct in principle, but those logical streams can still be framed and concatenated into a single physical stream. Any compressor that can handle non-stationary distributions would work well, it might be even possible to hint the logical stream boundary to guide compressors.
Several physical streams are absolutely necessary when those streams are going through different filters, but a lack of them doesn't harm too much either. Also I think xz was planning for "subblock" filters to address this use case anyway.