Question about your usage of PAR3

[Yutaka Sawada]

 There is a restriction of 32768 source blocks in PAR2. Some users complain this is too few. While my current PAR3 supports max 65536 total blocks, some users think this is not enough still. Note, this is not about a number of source files, but a number of slices of files. Slice is a minimum unit of virtually splitted files.

 I introduced a new recoverying method (imcomplete recovery), but it is weaken against data shift. I came up with a new idea to support non-limited Input File Slices without large speed down. Just putting multiple slices in one block. For example, by putting 100 slices in a block, it is possible to support 65536 * 100 slices with ease. I may re-write my PAR3 format. Even if there is no limit in format, working application itself has a limit by memory size etc.

[badon]

If PAR3 supports directory trees, and file moving and renaming, then I'll probably need a lot. Otherwise, I'm stuck with just doing it one folder at a time.

[persicum]

Hi, Yutaka!
Dando Real assume to use FFFFFFFF00000001 prime to achieve 2^32 blocks. As to RSC32, it already supports 2^32 blocks with -tm10 and -tm11 (default) using just 32-bit arithmetic, not 64-bit as Dando Real offer. 32-bit CPU does not support 64x64=128 bit multiplication, so 64-bit would be slow on 32-bit CPU.

The problem is you may not support more than say 2000000 blocks in 32-bit environment due to memory limit of 32-bit OS. The other problem is HUGE headers (low space efficiency)!!! The last problem even having 1000000 blocks you are NOT ABLE to recover RANDOM SAND of errors, only sequent errors.

So 16-bit and incomplete recovery is the best i think
1) small headers
2) actually infinite block number and power

[persicum]

yutaka, I hope you are alive after earthshake?

Your voting has no large meaning:
1) It worth nothing to implement 2^32 blocks on 32-CPU w/o 64-bit prime such as FFFFFFFF00000001
2) It worth noting that you probably could not  use more then 2000000 blocks due to memory limitation of 32-bit OS.

[badon]

Hopefully he will come back to tell us he is OK after the earthquake.

[Yutaka Sawada]

from badon

file moving and renaming

I am not sure what is your case. If par3 client will search all other directrories on HDD to find moved/renamed files, it will becomes very slow. That is same as calculating hashes/checksums of all files on your HDD... Remember that, as it detect damaged files also, all files were possible to match. I don't know proper English words.

from persicum

So 16-bit and incomplete recovery is the best i think

I notice that definision of Block and Slice are different. In QuickPar/PAR2, slice and block is same. In current PAR3, slice and block is different. Slice is a virtually splited pieces of source files. Block is a unit of recovery.

My idea is not 32-bit Reed-Solomon. It is difficult to find the position of Partially Damaged Blocks, when the block has one checksum only, and "incomplete recovery" is weaken against shift of data. But, if the one block has multiple-checksums like 4, it is more possible to locate the damaged blocks by using the checksum of complete part. When one block has 4 checksums, total slices becomes max 65536 * 4 simply. Because recovery itself is done by 16-bit RS, memory size may not be problem so much.

yutaka, I hope you are alive after earthshake?

 I am safe. Because I was busy, I could not use internet. I live west side of Japan islands and earthquake became weaken on my city. When I felt the eathquake, I did not think so huge damage was happen on north east sea side. I was surprised by watching TV news, too. I am not sure, disaster's side effect will reach to here.

[badon]

from badon
I am not sure what is your case. If par3 client will search all other directrories on HDD to find moved/renamed files, it will becomes very slow. That is same as calculating hashes/checksums of all files on your HDD... Remember that, as it detect damaged files also, all files were possible to match. I don't know proper English words.

The PAR3 client ONLY needs to search for MISSING files, NOT all files, and ONLY in the subdirectories of the root directory the PAR3 files are protecting, NOT the whole HDD.

[Yutaka Sawada]

from badon

ONLY needs to search for MISSING files, NOT all files

 If you want to introduce an unknown feature, you should explain details of it as possible as you can. OK, it may search missing (moved to somewhere) files only. Then, how will be the status of moved files ? They may be broken, short size (deleted last part), longer size (appended some), or complete. At this time, my PAR2 client searchs only misnamed complete files in same directory. What do you want, is searching of moved/renamed complete files under the directory-tree enough ? (Basically moving is same as renaming of sub-directory.)

[badon]

Yes, that is enough. It would be hard to search outside the PAR protected directory tree.

It sounds like it will require some user interaction to tell MultiPar what to do when it finds files in the directory tree that don't make sense. It should be easy to do automatically if:

* the file has been moved, but not renamed
* the file has been renamed, but not moved

It should be easy for MultiPar to figure out which file it is by comparing the checksum if:

* the file has been moved and renamed

It may require the user to provide information if:

* the file has been moved and renamed, and it is damaged
* there are many files that have been moved and renamed, and are damaged

[dvasdekis]

Hi Yukata,

Thank you for giving this careful thought. As you are aware, the 16-bit slice limit is a problem for Usenet users presently, and is becoming a larger problem all the time. We are all looking for a permanent solution.

Thanks again,

Dimitri