Risk of undetectable intended modification

[Yutaka Sawada]

 Parchive is designed to recover damaged files. To recover a damaged file, it needs to detect the damage at first. PAR1 uses MD5, and PAR2 uses MD5 & CRC-32 for the purpose. Basically, the level of PAR2 verification is same as a combination of SFV file and MD5 hash file. Even though MD5 has no cryptgraphic strength nowadays, it is fast and enough for accidental damage/error/modification. Normally, MD5 hash of a damaged file would be different from MD5 hash of the original file. I never got an incident about collision problem of PAR1/PAR2.

 Now, problem is an intended modification with forged MD5 hash. Though a theory of multi-collision of MD5 & CRC-32 was publiced long ago, I didn't think someone made those pair really. While I was searching internet about hash algorithm, I found a sample of their collision. When someone modified a file and forge their hash to be same as its original file, PAR2 clients (tested with MultiPar and QuickPar) cannot detect the modification. It treats the modified file as complete, and recovery of other damaged files will be failed. In this case, deleting the modified file is the solution, but finding the bad file requires another hash file like SHA-1/2. SFV file or MD5 file are useless, as it cannot detect the modification, too.

 This isn't a serious problem for normal usage of Parchive, because PAR files are available while backup or transport. When source files and PAR files are put on same place, it is much easier to modify PAR files than forging hash of source files. Only when you uses PAR files to prevent intended modification, you should understand the security level, PAR1/PAR2 is same as MD5 hash file. If the PAR files were not modified, they can recover damaged files still, but how to find the damage becomes the problem. Then, you must keep another secure hash files like SHA-1/2 with PAR files.

 Does someone use PAR files for the secure integrity checking usage ? Though I don't recommend that usage with PAR1/PAR2, I may change hash algorithm at next PAR3 sample. I just set MD5 in PAR3 for backword compatibility only, as it needs to calculate MD5 for PAR2 compatible packets anyway. But, as PAR3 doesn't distinguish source files by their filenames, hash collision problem may be bigger than PAR1/PAR2. I plan to use second hash for double check like; same MD5 and same 2nd hash = dupe files, or same MD5 but different 2nd hash = different files. (In this usage, 2nd hash must be stronger than MD5 for security usage.) If it uses SHA-1/2 or something latest hash as the first hash instead of MD5, it will not require strength in 2nd hash. (In this usage, 2nd hash is used only to confirm dupe files.) If many users are annoying to keep another secure hash file with PAR files, and think that PAR files should contain secure hash internally, I may change my design of PAR3.

[rnb5500]

I use the program checksum for integrity checking (it uses sha1), but for me adding sha1 (or stronger) to your Par3 would be a plus. One less program is always welcome.

It would also help to keep good habits of integrity creation/checking and PAR creation in the same time frame, rather than doing one and then the other some time (after a possible damage occurs) later.

On the other hand, if you just want to add sha1 support to Multipar it wouldn't be an annoyance to me. As I said, I'm already keeping checksum's .hash files in my folders.

[badon]

I also use corz checksum, but I much prefer using MultiPar for everything instead. For speed, I think the default hash used should be the fastest available that is also reliable for integrity checking, without a need for security. I think other hash algorithms should be optional, depending on the need for security.

For example, maybe CRC32 would be enough by default, but a user could choose to use MD5, SHA1, etc if they prefer. MultiPar should be able to record and/or detect which algorithm was used, when verifying files. In most cases, I prefer speed, but in a few cases, I will want security too, but I think MD5 will be enough. In rare cases, I might want something better than MD5.

A good example of a need for security is when distributing MultiPar. If it comes with a PAR file using SHA1 (or whatever), it could be verified using MultiPar, which would eliminate the need for all the other hash software out there - MultiPar will be the only software most people will need. I would very much like to see MultiPar become widely used for all sorts of things we can't even think of. It is very useful, and should be something everybody has, kind of like zip software.

I still think someone should tell the author of 7zip to incorporate MultiPar into it :)

[mictlan]

Once PAR3 gets final it will last as long as PAR2, even longer.
Due to the expected lifetime of PAR3 I would add SHA256 as secondary hashing algo.

[Yutaka Sawada]

 I am changing some details of new PAR3 idea, while I make sample client. When I find it is too hard to implement, I change it to easier design. I test some 256-bit hash algorithms for file hash. Then, double check of dupes will be done by checksum of the file's slices. This may be simpler than calculationg MD5 & CRC-128 (total 256-bit) for the same usage. I need to test speed with File IO later, when it runs properly. I don't want that the using hash is selectable from many hash algorithms, as my design is too complex already. At this time, I am making a function for PAR2 compatible packets, so that I can test the source block management and File IO with a PAR2 client.

 I made a simple speed tester of hash algorithms. It contains 32-bit hash for comparison; "One at a time", "FNV-1a", "MurmurHash3", and "Lookup3". I may use "Lookup3" for similarity/dupe detection of slices with segment hashes. Because I use CRC-128 for checksum of new PAR3 packet instead of MD5, it shows the speed of CRC-128. For file hash, Cryptgraphic Hash will be recommended. It tests old standard, MD5, SHA-1, SHA-256. I added three fast SHA-3 candidates from eBASH result; BLAKE, Blue Midnight Wish, and Shabal. I was surprised that BMW and Shabal are similar speed as SHA-1. They didn't selected as SHA-3 finalists, as they might be too fast to be enough secure. For someone interested in those speed, I put the tester here. As my PC is very old, the result may be different from recent PC.

Usage: test.exe [data size or file path]
When there is no file, value is treated in MB. The data is allocated on memory, so max 1GB.