Release note of v1.1.7 tree

[persicum]

Dando Real
PAR2 is an ordinary vehicle, but PAR3/RSC32 is a hyperspace warp starship, don't use the latter to go to country cottage. =))) Swapping is warping, and FFT is a magic dust

[Yutaka Sawada]

from persicum, 2011 Feb 27, 10:35:33 PM ;

If you use FileWrite() to write outside a file never do that because this doesnot work for files larger than 30G.

 I am not sure, but FileWrite() seems to be a command of Delphi. I use WriteFile() Win32API by VC. Because I can not treat so large 30GB over files, I never tested that problem.

 I don't know, hyperspace warp by startship is well-known technology... As easy simile, RSC32 is like an air-plain. It is faster than cars, but you need to go air-port at first. Swapping to temp file is same as going to air-port. The total time depends on, how far you want to go.

[persicum]

I use WriteFile() Win32API by VC

Everythink is OK with WriteFile() WinAPI.

But for very large files you should allocate a disk space first by using Seek() (I don't know how this winAPI is called exactly), then

BOOL WINAPI SetEndOfFile(
  __in  HANDLE hFile
);

[Dando Real]

Hi Yutaka, I read your message only now.

I made some debug version to test multi-thread. They are public at "Debug version for test" topic on this forum. Please test them, and compare which is fast for when.

OK, I'll try to do some tests

"read/write all at once" style IO is used for FFT. For example, FFT requires a set of data [ A ][ B ][ C ]. RSC32 may rearrange file data by using temporary file. The data of source file is like; [ A ][ A ][ A ][ B ][ B ][ B ][ C ][ C ][ C ]

I read a little about Galois Fields. You use p = 2^16 + 1 and you are bounded to an FFT of dimension 2^16, on blocks of 16 bits. So you have to compute many times the FFT.
Why not use instead GF(2^N), there is a library to work on fields of characteristic 2. You can then use FFT of greater order, requiring less (slow) HD seeking.
Otherwise, as GF(p) there is p = 2^64 - 2^32 + 1 which allows an FFT up to 2^32, without divisions, as described here:
http://www.craig-wood.com/nick/armprime/math.html

I think the following. The relative FFT speed/efficiency is (Data_Buffer_Size)/(Data_Buffer_Size+Recovery_Buffer_Size). Normally it is about 80-90%.

But you want to use say 256M RAM for Data_Buffer_Size and say 8G RAM for Recovery_Buffer_Size,
so FFT usability will be 1/32. So this slowdown recovery creation in 32 times as compared with the temp file case.

persicum,
sorry I don't understand why use 256 MB for data and 8GB for recovery, instead for example of 4 GB each.
And why should FFT slowdown recovery instead of speeding it up?

Why you are so hate a swapping?

The problem is not swapping itself - it is that I'm bound on low free space, I fullfill whatever HD I buy.

[persicum]

sorry I don't understand why use 256 MB for data and 8GB for recovery, instead for example of 4 GB each. And why should FFT slowdown recovery instead of speeding it up?

PAR2 uses a large buffer only for recovery, does not it? Why use huge buffer for data if they available on HDD? You assume use 4G for data and 4G for recovery, but this would slowdown FFT (4+4)/4 = 2 times.

[persicum]

p = 2^64 - 2^32 + 1

   
for FFFFFFFF00000001 prime see "Question about your usage of PAR3"

[Dando Real]

PAR2 uses a large buffer only for recovery, does not it? Why use huge buffer for data if they available on HDD? You assume use 4G for data and 4G for recovery, but this would slowdown FFT (4+4)/4 = 2 times.

Because if problem is random seeking, you load as much as possible by sequential ( relatively fast ) reading from files to memory, then you can have a fast random access to data

What I don't understand is what makes the FFT slow, you mean the time needed to load data in memory?

[persicum]

Because if problem is random seeking, you load as much as possible by sequential ( relatively fast ) reading from files to memory, then you can have a fast random access to data

Seeking or sequential reading does not matter until you read [A][][][][][] and need only [A] but [][][][][][] leave HDD cache.

What I don't understand is what makes the FFT slow, you mean the time needed to load data in memory

I meant that instead say FFT([AAAAAAAAAAAAAABB]) you assume to make FFT([AABB]) seven times.
But first FFT utilizes 16 symbols, but seven last FFTs utilize 7*4=28 symbols. So slowdown will be 28/16=1.75 time.

I may making a mistake here, but I can say definitely - Yutaka could not implement PAR3 w/o swapping original data into strips as RSC32 already does. So you need a Solid State Drive or Hybrid HDD+SDD rather than a lot of RAM and 64-bit which cannot help to PAR3, only SDD can speedup. =)))

As to FFFFFFFF00000001 it is cool prime but will slowdown PAR3 two times under 32-bit environment. Four 32-bit mults/64-bit_data_access will slowdown two times. But under 64-bit it would be a very cool choice.

[persicum]

Dondo Real

Persicum, I have already seen your utility in the past. But, as you admit, it is not efficient with small recovery, and needs a lot of hd temp space.As for now, I prefer something which is slower in generating, and faster on recovering, because the former is made only once, the second can happen multiple times

See benchmark on the RSC32 forum. With a second physical non-RAIDed HDD for swapping the speed of RSC32 repairing (regardless to data and recovery block count) is the same as MultiPAR2 single byte repairing (depends strongly on data block count).