Unofficial Faster Preclear


Recommended Posts

I have developed a way to do the post-read and verify step of preclear dramatically faster than the current method (more than twice as fast). Functionally it is the same. I am looking for some folks that are planning to do some preclears this weekend, optimally someone planning to preclear 2 identical drives so we get a speed comparison.

 

Drop me a PM if interested. Hoping to get 2-3 volunteers. I will be preclearing a new 4T drive. I am on 32-bit 5.0.5 but hoping a 64-bit person running a 6.0 beta will help test it too, and help me resolve any incompatibility.

 

I am out of town and not able to access my array, but will be able to send the enhanced Preclear Friday.

 

Edit:  This is the link to the latest version:  http://www.mediafire.com/download/xu2cel2zkb7475q/FastPreclear_FIX.zip

  • Upvote 2
Link to comment

Does this new method reduce the amount of work that the drive is asked to do? I was under the impression that the slowness of the preclear was on purpose to thoroughly test all parts of the drive possible in a reasonable period of time. Pseudo random seeks and other stuff to really work it hard.

 

Here is a quote directly from Joe.

When pre/post reading the disk, I intersperse reading the beginning block, random blocks of data, the linear set of blocks of the entire drive, and the last block on the device.  I purposely keep the disk head moving a LOT more than if just reading each block in turn.  This is to identify hardware that is marginal.  If your disk or cables, or controller, or power supply cannot cope with constant activity... you probably want to know it before you assign the disk to a spot in your array.
Link to comment

I write this a couple years ago when I was very active here, but never shared it with the community. It is a minor change to the preclear script to use a new component. It works the disk exactly the same. In fact you could argue that it works it harder because it is running faster. The pre-read and zero fill are identical (same preclear script) but the post-read and verify is done more efficiently.

Link to comment

I have almost everything together to build my backup server.

 

I'm planning to preclear some drives but I can start by next week only.

 

I will have 3x 4TB drives (to be harvested) - will post the specs if I have them ready.

Also 2x 500GB (identical) but I expect one to have issues.

1x 3TB drive + 1x 2TB drive

 

I intend to run 3 cycles - so I could run it with 5.x and 6.x

 

Prepare the test plan and provide the script and I will let it run for you.  ;)

 

Would you mind sharing the differences (in layman terms) between Joe's script and your's?

Link to comment

Are you planning on working with Joe to integrate this with his work, or are you going to rewrite the whole thing to be your supported version? There have been a whole lot of people wanting a browser integrated plug in version of preclear for a long time.

 

+1 to this

Link to comment

Are you planning on working with Joe to integrate this with his work, or are you going to rewrite the whole thing to be your supported version? There have been a whole lot of people wanting a browser integrated plug in version of preclear for a long time.

1+

 

I would be willing to test this.

 

Sent from my SCH-I545 using Tapatalk

 

 

Link to comment

Are you planning on working with Joe to integrate this with his work, or are you going to rewrite the whole thing to be your supported version? There have been a whole lot of people wanting a browser integrated plug in version of preclear for a long time.

 

I am making the changes to the Preclear script (I have made changes in the past - if you look in there you will see comments where bjp999 contributed). I have discussed this with Joe L. and I will share my changes with him and hope that he will accept this change in the official release. There is a second component needed with my version that will need to be distributed to do the faster post-read. This will be a small headache for him.

 

As far as launching preclear from within a GUI, I will consider it as a myMain enhancement. MyMain already monitors preclears (is this well known?).

Link to comment

As far as launching preclear from within a GUI, I will consider it as a myMain enhancement. MyMain already monitors preclears (is this well known?).

The ability to safely launch it from MyMain would be great. I use a combination of the email notifications and the MyMain page to check preclear status currently.
Link to comment

I'm getting 3 3TB Red's from newegg today or tomorrow - hope they aren't doing there crap-tastic packaging with multiple drive again.  So I can test it.  I have two preclear stations I could run the preclears on.  They are not identical boxes however.  Or just run the preclears on one box one at a time.  Assuming I don't have to RMA a drive or two from the shipment anyway.

Link to comment

I'm doing preclears right now and for the next week... 

 

I've got a pair of new Seagate NAS 4TB drives to burn in and a pair of Seagate Desktop 4TB drives to retest and a pair of old WD 2TB greens that I'm taking out of service (so I'm preclearing them to erase and test). 

 

The 4tb NAS drives are just at 26% complete on the post-read of the first of 2 cycles and have taken about 26 hours so far, they'll probably take about 34-38 hours if I recall correctly.

 

So I'd be interested in running the new beta.

 

Regards,

 

Stephen

Link to comment

The quicker post-read phase would be nice; but a clean integration with MyMain that allowed pre-clears to be done entirely from the GUI would be even nicer.    Given a choice of MyMain control or the increased speed, I'd definitely prefer the MyMain control ... and I suspect most others would also.

 

... Of course BOTH would be even better  8)

Link to comment

Are you planning on working with Joe to integrate this with his work, or are you going to rewrite the whole thing to be your supported version? There have been a whole lot of people wanting a browser integrated plug in version of preclear for a long time.

 

As far as launching preclear from within a GUI, I will consider it as a myMain enhancement. MyMain already monitors preclears (is this well known?).

 

I wasn't ware about monitoring preclears (I'm using MyMain). I was always worried about preclearing on the same box as the my "production" data. But if it can be done safely this is pretty cool!

Link to comment

Running preclears on a production server is not a problem. Personally I do it often. Click the link in my sig on monitoring preclears to see what it looks like. You could also run myMain on the test server.

 

On the topic of myMain integration to start preclears from the GUI, I will think about it. But starting up screen and running a simple command doesn't seem nearly as valuable as saving 10+ hours on the preclear!

Link to comment

Saving 10+ hours would be for a 4T disk? Interesting.

 

The last 3T drive I precleared took 13h 35m to post read. The drive wrote zero's at 147 MB/s and my typical parity check speeds are this fast too (I have all the same drives) so I suspect this would be the fastest average sequencial read/write speed to read the whole drive. It took 5h 40m to do this clearing, so I'd expect any post read improvement to have a similar read time as a minimum. It would appear just under 8 hours of time could be saved at the most, which is a good improvement.

 

However, if you add random seek testing then I'd expect the average read speed to go down and the time saved to be less.

 

FYI, pre-read was 6h 33m.

Link to comment

Saving 10+ hours would be for a 4T disk? Interesting.

 

The last 3T drive I precleared took 13h 35m to post read. The drive wrote zero's at 147 MB/s and my typical parity check speeds are this fast too (I have all the same drives) so I suspect this would be the fastest average sequencial read/write speed to read the whole drive. It took 5h 40m to do this clearing, so I'd expect any post read improvement to have a similar read time as a minimum. It would appear just under 8 hours of time could be saved at the most, which is a good improvement.

 

However, if you add random seek testing then I'd expect the average read speed to go down and the time saved to be less.

 

FYI, pre-read was 6h 33m.

 

10h was an estimate. Maybe a little high, but we'll have to see with some real testing. Yes, it was based on preclearing a 4T drive.

 

I was looking at the code (I wrote this is 2011 so is taking some time to get back into it) and realized that for the post read I did accidentally (it seems) disable the overlapping randomized I/O. I am on the road with no array - all I can do is edit the code but not test it. So I added it the randomized I/O back in. The time for the post read should be close to the time for the pre-read with my enhancement, which in your example (with a 3T drive), would save some 7 hours, and extrapolating to a 4T drive, 8 3/4 hours. But we'll have to see what comes out of the testing. Hope community agrees this is pretty dramatic.

 

I have gotten several volunteers. Some 32 bit some 64 bit. I expect to be able to send something out Friday night (if all goes well) or Saturday (if it doesn't). I'll be running it through its paces on a test array with some small 20G drives I use for preclear testing.

 

Wish me luck!

 

Contributions will be gratefully accepted and encourage more enhancements. ;)

Link to comment

Are you planning on working with Joe to integrate this with his work, or are you going to rewrite the whole thing to be your supported version? There have been a whole lot of people wanting a browser integrated plug in version of preclear for a long time.

1+

 

I would be willing to test this.

 

Sent from my SCH-I545 using Tapatalk

 

+2

Link to comment

Contributions will be gratefully accepted and encourage more enhancements. ;)

 

It sounds great and there will be many people who will love the quicker speed you've found.

 

Now I wonder if there was a good reason you disabled the randomized IO that you've forgotten about?

 

Two theories - I don't remember exactly:

 

1 - I figured we'd already done that through the entire preread and the post read was more about comparing the disk to zeroes and checking for induced errors.

 

2 - I was lazy and doing it quick and dirty.

 

Or some combination therein!

Link to comment

If you run myMain within unmenu, it will allow you to monitor progress. When complete it will show you the preclear, zeroing, and post clear times. Not sure it handles multi-pass - been a while since I created it and I don't run multiple cycles.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.