Multiple Instances

[jonnyboy]

I used 4 separate drives for the swap files. I will be setting up a server with zfs and at least 12 drives and I will test it with a single drive, but separate folder. I do not own any SSD's but that might work, depends on read speed.

Using the same drive for the swap files caused it to stall several times, at least it 'appeared' to stall.

[jonnyboy]

this 3.03 is for testing purposes only. This creates a dummy file to show that HDD is busy by an instance. After portion swapped this deletes the dummy file so other instances can continue.

This affects currently swap stage only, encoding and restoring could stall as before.

It appears that the dummy file is being created, then a file fragment is being written, then the dummy file is removed and the process continues until the entire swap file has been written. If that is correct, running multiple instances would produce severely fragmented filesets, making them more IO intensive and slower.

I am testing 2 instances and both are running concurrently. I expected 1 process to wait for the other to complete, it did not. The dummy file is only visible in the swap folder occasionally.
I did notice something  I had not noticed before. The hash table only appears to be calculated by 1 process. The other processes just zip through it, must be able to access the results in memory.

[persicum]

It appears that the dummy file is being created, then a file fragment is being written, then the dummy file is removed and the process continues until the entire swap file has been written. I am testing 2 instances and both are running concurrently. I expected 1 process to wait for the other to complete, it did not. The dummy file is only visible in the swap folder occasionally.

Yes, that’s right. Does this behavior have any effects as compared to older version? Is elapsed time better or worse?

If that is correct, running multiple instances would produce severely fragmented filesets, making them more IO intensive and slower.

The prog always allocates free space before filling swaps with actual bytes.

I expected 1 process to wait for the other to complete, it did not.

Would it be a queue behavior? You may run instances one after one from a GUI.
BTW, besides swapping other stages as encoding and restoring are also very disk intensive, too.
If one wants multiple instances he must have multiple HDDs as well =))

I did notice something  I had not noticed before. The hash table only appears to be calculated by 1 process. The other processes just zip through it, must be able to access the results in memory.

Sorry, I cannot understand what you mean.

[jonnyboy]

What I was getting at is:

If I start 1 instance and it progresses through the list of files creating the hash table and 2 minutes into it I start 3 more instances, within 30 seconds, the other 3 instances are processing the exact same file as the first instance. It did not slow down, the others caught up.

[jonnyboy]

Is elapsed time better or worse?

It did not appear to be faster than individual instances and the processes never completed. I tried 3 times and it crashed 3 times, never completing.

[persicum]

Did you give a different -bn to each instance while testing upon same fileset?

[persicum]

I have tested three instances in a common folder both for -wt and -wrr switches. I have not found any troubles except severe HDD head seek. They worked concurrently. Of course, I gave them three different -bn, namely -bntest1, -bntest2 and -bntest3.

If one has only one extra drive, which is HardDiskDrive and is not SolidStateDrive or RAMDisk, I don't know how prevent stall except simple queue. But a queue is not a concurrent execution.