Live Business Chat
2014 Apr 20, 07:17:07 am *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
News: Some accounts got accidentally deleted by anti-spam software during a spammer attack on this forum. Please re-register. If you have trouble, contact badon or tamo42 in the chat. This is a friendly non-profit discussion group about making money. You won't be able to see all forums at first. You have to register to see more forums. Click the "NOTIFY" button every chance you get to receive instant alerts about new information.
 
   Home   Help Search Calendar Login Register  
Poll
Question: How many slices will you set in PAR3 ?
65536 is enough (16-bit, same as PAR2) - 1 (16.7%)
I want 262144 (18-bit) - 0 (0%)
I want 1048576 (20-bit, same as RSC32) - 0 (0%)
I want 16777216 (24-bit) - 0 (0%)
I want more (32-bit) - 5 (83.3%)
Total Voters: 6

Pages: 1   Go Down
  Print  
Author Topic: Question about your usage of PAR3  (Read 3035 times)
0 Members and 1 Guest are viewing this topic.
Yutaka Sawada
Moderator
Mogul
*****

Karma: +9/-0
Offline Offline

Posts: 427


« on: 2011 Mar 07, 07:21:04 pm »

 There is a restriction of 32768 source blocks in PAR2. Some users complain this is too few. While my current PAR3 supports max 65536 total blocks, some users think this is not enough still. Note, this is not about a number of source files, but a number of slices of files. Slice is a minimum unit of virtually splitted files.

 I introduced a new recoverying method (imcomplete recovery), but it is weaken against data shift. I came up with a new idea to support non-limited Input File Slices without large speed down. Just putting multiple slices in one block. For example, by putting 100 slices in a block, it is possible to support 65536 * 100 slices with ease. I may re-write my PAR3 format. Even if there is no limit in format, working application itself has a limit by memory size etc.
Logged
badon
Administrator
Capitalist Pig
*****

Karma: +57/-38
Online Online

Posts: 8418



« Reply #1 on: 2011 Mar 10, 12:54:39 am »

If PAR3 supports directory trees, and file moving and renaming, then I'll probably need a lot. Otherwise, I'm stuck with just doing it one folder at a time.
Logged

Do not PM questions. Answers should be publicly available.
Backup is not enough. Protect your data with MultiPar.
Writer of LBC Chinese coin investment articles.
Founder of the Coin Compendium (forum).
I type faster on a TypeMatrix.
Use my work. Give credit.
china-mint.info forum.
LBC makes you rich.
FreeArc is amazing!
Donate.
persicum
Serious Business
***

Karma: +2/-1
Offline Offline

Posts: 144


« Reply #2 on: 2011 Mar 11, 12:17:24 am »

Hi, Yutaka!
Dando Real assume to use FFFFFFFF00000001 prime to achieve 2^32 blocks. As to RSC32, it already supports 2^32 blocks with -tm10 and -tm11 (default) using just 32-bit arithmetic, not 64-bit as Dando Real offer. 32-bit CPU does not support 64x64=128 bit multiplication, so 64-bit would be slow on 32-bit CPU.

The problem is you may not support more than say 2000000 blocks in 32-bit environment due to memory limit of 32-bit OS. The other problem is HUGE headers (low space efficiency)!!! The last problem even having 1000000 blocks you are NOT ABLE to recover RANDOM SAND of errors, only sequent errors.

So 16-bit and incomplete recovery is the best i think
1) small headers
2) actually infinite block number and power
Logged
persicum
Serious Business
***

Karma: +2/-1
Offline Offline

Posts: 144


« Reply #3 on: 2011 Mar 11, 07:11:22 am »

yutaka, I hope you are alive after earthshake?

Your voting has no large meaning:
1) It worth nothing to implement 2^32 blocks on 32-CPU w/o 64-bit prime such as FFFFFFFF00000001
2) It worth noting that you probably could not  use more then 2000000 blocks due to memory limitation of 32-bit OS.
Logged
badon
Administrator
Capitalist Pig
*****

Karma: +57/-38
Online Online

Posts: 8418



« Reply #4 on: 2011 Mar 11, 08:46:27 am »

Hopefully he will come back to tell us he is OK after the earthquake.
Logged

Do not PM questions. Answers should be publicly available.
Backup is not enough. Protect your data with MultiPar.
Writer of LBC Chinese coin investment articles.
Founder of the Coin Compendium (forum).
I type faster on a TypeMatrix.
Use my work. Give credit.
china-mint.info forum.
LBC makes you rich.
FreeArc is amazing!
Donate.
Yutaka Sawada
Moderator
Mogul
*****

Karma: +9/-0
Offline Offline

Posts: 427


« Reply #5 on: 2011 Mar 14, 08:13:34 pm »

from badon
Quote
file moving and renaming

I am not sure what is your case. If par3 client will search all other directrories on HDD to find moved/renamed files, it will becomes very slow. That is same as calculating hashes/checksums of all files on your HDD... Remember that, as it detect damaged files also, all files were possible to match. I don't know proper English words.

from persicum
Quote
So 16-bit and incomplete recovery is the best i think

I notice that definision of Block and Slice are different. In QuickPar/PAR2, slice and block is same. In current PAR3, slice and block is different. Slice is a virtually splited pieces of source files. Block is a unit of recovery.

My idea is not 32-bit Reed-Solomon. It is difficult to find the position of Partially Damaged Blocks, when the block has one checksum only, and "incomplete recovery" is weaken against shift of data. But, if the one block has multiple-checksums like 4, it is more possible to locate the damaged blocks by using the checksum of complete part. When one block has 4 checksums, total slices becomes max 65536 * 4 simply. Because recovery itself is done by 16-bit RS, memory size may not be problem so much.

Quote
yutaka, I hope you are alive after earthshake?

 I am safe. Because I was busy, I could not use internet. I live west side of Japan islands and earthquake became weaken on my city. When I felt the eathquake, I did not think so huge damage was happen on north east sea side. I was surprised by watching TV news, too. I am not sure, disaster's side effect will reach to here.
Logged
badon
Administrator
Capitalist Pig
*****

Karma: +57/-38
Online Online

Posts: 8418



« Reply #6 on: 2011 Mar 15, 10:24:19 am »

from badon
I am not sure what is your case. If par3 client will search all other directrories on HDD to find moved/renamed files, it will becomes very slow. That is same as calculating hashes/checksums of all files on your HDD... Remember that, as it detect damaged files also, all files were possible to match. I don't know proper English words.

The PAR3 client ONLY needs to search for MISSING files, NOT all files, and ONLY in the subdirectories of the root directory the PAR3 files are protecting, NOT the whole HDD.
Logged

Do not PM questions. Answers should be publicly available.
Backup is not enough. Protect your data with MultiPar.
Writer of LBC Chinese coin investment articles.
Founder of the Coin Compendium (forum).
I type faster on a TypeMatrix.
Use my work. Give credit.
china-mint.info forum.
LBC makes you rich.
FreeArc is amazing!
Donate.
Yutaka Sawada
Moderator
Mogul
*****

Karma: +9/-0
Offline Offline

Posts: 427


« Reply #7 on: 2011 Mar 25, 07:30:44 pm »

from badon
Quote
ONLY needs to search for MISSING files, NOT all files

 If you want to introduce an unknown feature, you should explain details of it as possible as you can. OK, it may search missing (moved to somewhere) files only. Then, how will be the status of moved files ? They may be broken, short size (deleted last part), longer size (appended some), or complete. At this time, my PAR2 client searchs only misnamed complete files in same directory. What do you want, is searching of moved/renamed complete files under the directory-tree enough ? (Basically moving is same as renaming of sub-directory.)
Logged
badon
Administrator
Capitalist Pig
*****

Karma: +57/-38
Online Online

Posts: 8418



« Reply #8 on: 2011 Mar 25, 09:37:59 pm »

Yes, that is enough. It would be hard to search outside the PAR protected directory tree.

It sounds like it will require some user interaction to tell MultiPar what to do when it finds files in the directory tree that don't make sense. It should be easy to do automatically if:

* the file has been moved, but not renamed
* the file has been renamed, but not moved

It should be easy for MultiPar to figure out which file it is by comparing the checksum if:

* the file has been moved and renamed

It may require the user to provide information if:

* the file has been moved and renamed, and it is damaged
* there are many files that have been moved and renamed, and are damaged
Logged

Do not PM questions. Answers should be publicly available.
Backup is not enough. Protect your data with MultiPar.
Writer of LBC Chinese coin investment articles.
Founder of the Coin Compendium (forum).
I type faster on a TypeMatrix.
Use my work. Give credit.
china-mint.info forum.
LBC makes you rich.
FreeArc is amazing!
Donate.
dvasdekis
Dreamer
*

Karma: +0/-0
Offline Offline

Posts: 6


« Reply #9 on: 2011 Apr 19, 07:18:42 pm »

Hi Yukata,

Thank you for giving this careful thought. As you are aware, the 16-bit slice limit is a problem for Usenet users presently, and is becoming a larger problem all the time. We are all looking for a permanent solution.

Thanks again,

Dimitri
Logged
Pages: 1   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!
Page created in 0.194 seconds with 22 queries.