Multiple Instances

[jonnyboy]

@persicum

Is there are problem with running multiple instance of RSC32? I'm thinking new instances upto 50% available memory or 75% processor? I'm testing now. But, if you know of a problem, it could save me some time.

Thank you

[persicum]

if they have the same drive for swap files then may be some interference occure... The are no special troubles excepting swapfiles time access...

[jonnyboy]

I ran test with 4 instances, all using separate swap, destination and log files. I did not see any problems. Also, the cpu never went over 50% with the 4 instances.
Since I have 8 cores and 32GB ram in test system, I can try, pretty much, anything.

[jonnyboy]

There does seem to be a problem. When running more than one instance, the swapping pausees a really long time while it clears the ram and finishes writing to disk.

Is there a way to determine what stage RSC32 is currently in?

[persicum]

Maybe better to use 750 or 1000 MB per instance to leave more RAM to OS cache?

[jonnyboy]

Yes. When I first run the tests, I used 2 threads and -mu250m, it ran with no problems. Then, when tested second run with 4 threads and -mu1g, there were lots of pauses, not freezes, it appeared as if each instance had hung.

I will run some more tests, namely 1 instance 4 times, 2 instances 2 times and 4 instances 1 time. Then, I'll compare the times. If the speed is not increased by running multiple instances, then there is no point continuing down that path.

[persicum]

Would you post the results after tests finished, please? I believe that you definitely can obtain much speed from instances because your PC is very powerful indeed. As to me, even two instances interfere deadly each other on my PC =]]

[jonnyboy]

I apologize for the delay, I have busy with other projects. I ran a few tests and here are the results:
My test server 8 cores 3.0 GHz, 32 GB ram, 6 physical 2TB hdd's, non-raided.
Each instance is:

Code:

Encoding Method..............: RS32_DFT
Number of Data Volumes.......: 475620
Number of Recovery Volumes...: 47562
Total Size...................: 14,744,202,241
Recovery Size................: 1,523,576,916
Overall Size.................: 16,267,779,157
Size of Volume...............: 31,000
Size of Buffer...............: 1,736 (5.6%)
Volume Alignment.............: Not Available
Redundancy...................: 10%
Header Redundancy............: 5
Efficiency...................: 96.77%
Transform Efficiency.........: 99.79%
Memory Required..............: 991,966,360
Number of Threads............: 4
SSE2 Support.................: Enabled
Sizing Scheme................: 10 Equal Files

Run 1 instance 4 times:      37min 36 sec, fastest individual time was 9 min, 20 sec
Run 2 instances 2 times:     21min 2 sec, fastest individual time was 10 min, 19 sec
Run 4 instances 1 time:      14min 2 sec, fastest individual time was 13 min, 34 sec

These times do not include creating the hash tables.
Using the same drive for multiple swap stores was devastating to the overall time. It stalls many times. If, more than 1 additional physical drive is not available, then do not run more than 1 instance.
I tested only physical drives. It would be interesting to test on zfs or hardware raid.
With my server I could probably run 5 instances, that would put utilization at 100%, any more would probably not show any increase in overall speed.


@persicum:
If you were to rebuild your app that would separate the tasks to separate exe's, (or add arguments) then my gui could limit the swapping to 1 instance and up to 1 instance per 2 cores. That might make it possible to run multiple instances with only 1 additional drive.
I still need to figure out how to redirect stdoutput, then I will have something for you.

[jonnyboy]

Run 1 instance 4 times:      43 min, 48 sec
Run 2 instances 2 times:   25 min, 25 sec
Run 4 instances 1 time:      16 min, 13 sec


This is the as previous tests, but counts the time to create the hash tables.

[persicum]

I am shocked... Did your test require 5 separate physical HDD's? o:
May be one 100-200G Solid State Drive to be a solution??