Preclear plugin


Recommended Posts

What does everyone think about cleaning up the installation of this whole thing by having the plugin install the script?

 

Also it should check for disk.cfg and warn to create one for new servers without any array assignments, fastest way is click new config.

 

Definitely.  These two things would solve a ton of problems.

Link to comment

Thanks for the redux on what I accidentally ended up doing right, Walter! And for the link to the faster version.

 

Just as a data point, I've finished the Pre-Read stage: about 19.5 hours for 8TB, at about 120 MB/s. Now at Zeroing at 7% complete at 182 MB/s (so far; I'm sure that will drop).

 

I suppose I should have hooked up the two other 8TB drives I want to use before I started pre-clearing the first one, so I could do them all at the same time. Oh, well. I'll get this one done, then hook up the other two, and do those two together. Actually, I guess this way is more scientific: I'll get to see what the difference is between solo and multiple pre-clears, on my own system.

Link to comment

OK: 72 hours later, first preclear done.

 

Now, is there a guide for how to interpret the report the preclear script provides?

 

If I've got a few "near_thresh" values, but there are no SMART attributes that are FAILING_NOW, and there are 0 sectors that have been or have ever been pending re-allocation, am I good to go?

 

(Also, as a recent new user somewhat confounded by the setup steps needed to start his initial run of the preclear script, I agree that having the plugin install the script, and having the 'disk.cfg' check and solution—i.e. "fastest way to create a 'disk.cfg' file is to click 'New Config'"—would be very helpful!)

Link to comment

Since I can't get authorization of Joe L. to distribute the preclear script, I'm writing a new script for it. Right now I have all the basic functionality working, aka preclear and test preclear. Since support all legacy functions takes time, I want to know if anyone is preclearing disks with partition starting on block 63, and if anyone have converted from block 64 to block 63 or vice versa in the last year.

Link to comment

Since I can't get authorization of Joe L. to distribute the preclear script, I'm writing a new script for it. Right now I have all the basic functionality working, aka preclear and test preclear. Since support all legacy functions takes time, I want to know if anyone is preclearing disks with partition starting on block 63, and if anyone have converted from block 64 to block 63 or vice versa in the last year.

 

He won't give you the authorisation or you can't get hold of him!? Haven't you incorporated both scripts (Original and Faster) into the Plugin already? Have you been asked to remove them? If it is a matter of contact I am SURE someone can help facilitate that. If it is the latter then I would be SHOCKED. Your Plugin has added much needed front end to an almost ESSENTIAL task. I won't speculate further until you confirm.

 

Is there perhaps another reason WHY you are re-writing it too?  Are you looking to add further functionality? If so what is this functionality?

 

If at all possible (if you really don't have to) I think given the exposure your plugin gets via Community Applications - not to mention how well you have done at making a CLI exercise task so easy and user friendly via the GUI - I think it should only use one of the two very well tested scripts.

 

Although it is officially unsupported by LT, a preclear is a task that almost every user of unRAID does. All new users of unRAID are advised to do it by Staff, Mod's and Experienced Users alike.  At the moment the ability to do it is STABLE. Your Plugin is great as all it does is add a front end to those stable scripts whose (once executed) outcomes are very assured.

 

My issue is that by introducing an untested scripts into the mix does a few things:

 

- Removes confidence in the preclear script;

- Introduces possibilities for error / bugs etc;

- Add's confusion etc

 

Now, like I said above, IF you are being told you can't use the scripts then I guess you have no choice BUT I think that would be a big step back from where we are today. If it isn't broken don't fix it.

 

Can you add some more context to your decision?

Link to comment

Since I can't get authorization of Joe L. to distribute the preclear script, I'm writing a new script for it. Right now I have all the basic functionality working, aka preclear and test preclear. Since support all legacy functions takes time, I want to know if anyone is preclearing disks with partition starting on block 63, and if anyone have converted from block 64 to block 63 or vice versa in the last year.

 

He won't give you the authorisation or you can't get hold of him!? Haven't you incorporated both scripts (Original and Faster) into the Plugin already? Have you been asked to remove them? If it is a matter of contact I am SURE someone can help facilitate that. If it is the latter then I would be SHOCKED. Your Plugin has added much needed front end to an almost ESSENTIAL task. I won't speculate further until you confirm.

 

Is there perhaps another reason WHY you are re-writing it too?  Are you looking to add further functionality? If so what is this functionality?

 

If at all possible (if you really don't have to) I think given the exposure your plugin gets via Community Applications - not to mention how well you have done at making a CLI exercise task so easy and user friendly via the GUI - I think it should only use one of the two very well tested scripts.

 

Although it is officially unsupported by LT, a preclear is a task that almost every user of unRAID does. All new users of unRAID are advised to do it by Staff, Mod's and Experienced Users alike.  At the moment the ability to do it is STABLE. Your Plugin is great as all it does is add a front end to those stable scripts whose (once executed) outcomes are very assured.

 

My issue is that by introducing an untested scripts into the mix does a few things:

 

- Removes confidence in the preclear script;

- Introduces possibilities for error / bugs etc;

- Add's confusion etc

 

Now, like I said above, IF you are being told you can't use the scripts then I guess you have no choice BUT I think that would be a big step back from where we are today. If it isn't broken don't fix it.

 

Can you add some more context to your decision?

Possibly related to this:

http://lime-technology.com/forum/index.php?topic=26121.msg228146#msg228146

Link to comment

Since I can't get authorization of Joe L. to distribute the preclear script, I'm writing a new script for it. Right now I have all the basic functionality working, aka preclear and test preclear. Since support all legacy functions takes time, I want to know if anyone is preclearing disks with partition starting on block 63, and if anyone have converted from block 64 to block 63 or vice versa in the last year.

 

He won't give you the authorisation or you can't get hold of him!?

Last impression I got, Joe was not real thrilled with others modifying his work. I think he was trying to force limetech to write a preclear function into unraid, instead of relying on the community for a core function.

 

http://lime-technology.com/forum/index.php?topic=26121.msg228146#msg228146

Link to comment

He won't give you the authorisation or you can't get hold of him!? Haven't you incorporated both scripts (Original and Faster) into the Plugin already? Have you been asked to remove them? If it is a matter of contact I am SURE someone can help facilitate that. If it is the latter then I would be SHOCKED. Your Plugin has added much needed front end to an almost ESSENTIAL task. I won't speculate further until you confirm.

 

I can't get in touch with him. He won't respond to PM or topic questions. I think he knows what I'm doing, but he won't participate. Either way, it's a bummer.

 

Is there perhaps another reason WHY you are re-writing it too?  Are you looking to add further functionality? If so what is this functionality?

 

The original script was written while unRAID used BASH 3.2., I guess. The code is linear, and it's difficult to modify one thing without borking others. It's more a rework of the original code, incorporating things like associative arrays and trying to contain the variables inside functions, avoiding the disseminated use of global variables.

 

No other functionality is planned beyond the actual scope.

 

If at all possible (if you really don't have to) I think given the exposure your plugin gets via Community Applications - not to mention how well you have done at making a CLI exercise task so easy and user friendly via the GUI - I think it should only use one of the two very well tested scripts.

 

They are basically the same code. When bjp999 took the code to improve it, he was hands tied. He couldn't modify it extensively, so he opted to use a binary executable to speedup the post-zeroing verification. The problem in the original code is that post-zeroing read executes two reads on the same area, so it actually reads twice the disk. It's not necessary and it makes the code confusing. I removed that part. Other problem I corrected is that some of the last blocks of the disk eventually weren't read, and that I got fixed too.

 

I fixed that disks.cfg and super.dat problem too.

 

My issue is that by introducing an untested scripts into the mix does a few things:

 

- Removes confidence in the preclear script;

- Introduces possibilities for error / bugs etc;

- Add's confusion etc

 

I'm not removing the support to the original script right now. I'm proposing a new code that would eventually meet the same confidence we have in the current one. It need a lot more testing before we can pull the plug.

 

And the confusion always exists. We can't make a omelet without breaking some eggs.

 

 

Don't worry, afterall it's only been 3 years since that attempt. I'm sure unraid 6.2 soon™ edition will address it.

 

Have you any real news about that?

 

 

Link to comment

 

I also hope any new developments have an open source license indicated on it from the very beginning.

 

I'll license it under GPL like I did on Unassigned Devices. Developing a plugin is one task, it's concentrated in time, but bugfixes, improvements and support are long term tasks that may need some succession in development.

 

Guys, I need some people to test/validate this new script. I'll release a testing version of the new Preclear Plugin within the next week or so.

Link to comment

I've got a retired 2 TB ears I can hammer ?

 

I have several smaller drives I'm not using anymore and 2 servers each with a spare port. Be happy to give it a workout.

 

I have 2 new drives needing preclearing that I can test on.

 

i have WD 2TB blacks Enterprise that i can test.

 

I appreciate if you all can test it soon.

Link to comment

Preclear not working on 6.2 beta, any idea what this error is about?

 

This is from the script, but the plugin returns same error:

 

root@Testv6:/boot# preclear_disk.sh -n /dev/sdb
sfdisk: invalid option -- 'R'

Usage:
sfdisk [options] <dev> [[-N] <part>]
sfdisk [options] <command>

Display or manipulate a disk partition table.

Commands:
-A, --activate <dev> [<part> ...] list or set bootable MBR partitions
-d, --dump <dev>                  dump partition table (usable for later input)
-J, --json <dev>                  dump partition table in JSON format
-g, --show-geometry [<dev> ...]   list geometry of all or specified devices
-l, --list [<dev> ...]            list partitions of each device
-F, --list-free [<dev> ...]       list unpartitions free areas of each device
-s, --show-size [<dev> ...]       list sizes of all or specified devices
-T, --list-types                  print the recognized types (see -X)
-V, --verify [<dev> ...]          test whether partitions seem correct

--part-label <dev> <part> [<str>] print or change partition label
--part-type <dev> <part> [<type>] print or change partition type
--part-uuid <dev> <part> [<uuid>] print or change partition uuid
--part-attrs <dev> <part> [<str>] print or change partition attributes

<dev>                     device (usually disk) path
<part>                    partition number
<type>                    partition type, GUID for GPT, hex for MBR

Options:
-a, --append              append partitions to existing partition table
-b, --backup              backup partition table sectors (see -O)
     --bytes               print SIZE in bytes rather than in human readable format
-f, --force               disable all consistency checking
     --color[=<when>]      colorize output (auto, always or never)
                             colors are enabled by default
-N, --partno <num>        specify partition number
-n, --no-act              do everything except write to device
     --no-reread           do not check whether the device is in use
-O, --backup-file <path>  override default backup file name
-o, --output <list>       output columns
-q, --quiet               suppress extra info messages
-X, --label <name>        specify label type (dos, gpt, ...)
-Y, --label-nested <name> specify nested label type (dos, bsd)

-L, --Linux               deprecated, only for backward compatibility
-u, --unit S              deprecated, only sector unit is supported

-h, --help     display this help and exit
-v, --version  output version information and exit

Available columns (for -o):
gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S
      Start-C/H/S
bsd: Slice Start End Sectors Cylinders Size Type Bsize Cpg Fsize
sgi: Device Start End Sectors Cylinders Size Type Id Attrs
sun: Device Start End Sectors Cylinders Size Type Id Flags

For more details see sfdisk(.
Sorry: Device /dev/sdb is busy.: 1

 

Link to comment

Preclear not working on 6.2 beta, any idea what this error is about?

 

This is from the script, but the plugin returns same error:

 

Seem like the timming is almost perfect.  ::)

 

The new script is mostly completed. The only significant missing feature is the SMART relatories. Will try to speedup things.

Link to comment

Here it is, guys: https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/plugins/preclear.disk.beta.plg

 

It will add a Preclear Disk Beta icon under Settings.

 

How to test it:

 

1) Pick a formatted disk or format it with Unassigned Devices and copy some files to it;

 

2) Go to Settings/PreclearBeta and start a Clear operation;

 

3) Once it's over, please run a Verify All the Disk operation;

 

4) If successful, please run both Test and Run the post-read verify operations from the older plugin;

 

5) If possible, assign the precleared disk to the array and start a Check Parity operation.

 

All feedbacks are pretty much appreciated.

 

Link to comment

Here it is, guys: https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/plugins/preclear.disk.beta.plg

 

It will add a Preclear Disk Beta icon under Settings.

 

How to test it:

 

1) Pick a formatted disk or format it with Unassigned Devices and copy some files to it;

 

2) Go to Settings/PreclearBeta and start a Clear operation;

 

3) Once it's over, please run a Verify All the Disk operation;

 

4) If successful, please run both Test and Run the post-read verify operations from the older plugin;

 

5) If possible, assign the precleared disk to the array and start a Check Parity operation.

 

All feedbacks are pretty much appreciated.

 

I think I was a little hasty in my apprehension about this work, so for that I am Sorry. After taking in all the facts (and also TBH some surprising behaviour by some of the longer serving members of the community) I would like to do my bit to support. I will use your BETA preclear version throughout my testing of unRAID 6.2 soon™ Edition through BETA -> RC -> FINAL:

 

http://lime-technology.com/forum/index.php?topic=47427.0

 

I will work into what I am doing the ability to test as much as I can (e.g. between testing rebuilds etc) the following disks. And I have started them ALL off on a preclear run right now ....

 

Parity 1: Western Digital Blue WD800AAJS (00PSA0) 80GB 7200 RPM 8MB Cache SATA 3.0Gb/s

Parity 2: Western Digital Blue WD800AAJS (00PSA0) 80GB 7200 RPM 8MB Cache SATA 3.0Gb/s

 

Data 1: Seagate Barracuda 7200.10 ST380815AS (9CY131-020) 80GB 7200 RPM 8MB Cache SATA 3.0Gb/s

Data 2: Western Digital WD Blue WD800JD (60LSA5) 80GB 8MB Cache SATA 3.0Gb/s

Data 3: Western Digital WD Blue WD800JD (60LSA5) 80GB 8MB Cache SATA 3.0Gb/s

 

6.2.0-beta18:

 

*For those who notice, don't worry about the temperatures. As it is my test server with test disks I didn't double check the fans. They had failed. Replaced them as of the verify so temperatures are in order.

 

Links below are screenshots from the Plugins' setup page and from Unassigned Devices on the Main Tab (if you're checking compatibility).

 

Screen_Shot_2016_03_12_at_3_56_07_PM.png

 

Screen_Shot_2016_03_12_at_3_56_28_PM.png

 

 

UPDATE 1: Preclear on ALL drives was successful. Links below indicating status are once again from Unassigned Devices and Main Tab for results (including eye popup at finished) indication and indicating Start of an ALL disk Verify (including eye popup during).

 

Screen_Shot_2016_03_12_at_5_31_32_PM.png

 

Screen_Shot_2016_03_12_at_5_30_46_PM.png

 

Screen_Shot_2016_03_12_at_5_31_59_PM.png

 

Screen_Shot_2016_03_12_at_5_35_21_PM.png

 

Screen_Shot_2016_03_12_at_5_35_45_PM.png

 

Screen_Shot_2016_03_12_at_5_44_48_PM.png

 

 

UPDATE 2: ALL Disk Verify on ALL drives was successful. Message reports back that the drive is now precleared.

 

Screen_Shot_2016_03_12_at_6_05_45_PM.png

 

Screen_Shot_2016_03_12_at_6_06_01_PM.png

 

Screen_Shot_2016_03_12_at_6_06_19_PM.png

 

 

UPDATE 3: Then I tried to follow the instructions to use the exisiting preclear plugin to run the Test and Postread Verify from the older plugin. Unfortunately I could not. After placing the Joe.L version (original) version of the script on the flash, here is what happened:

 

Firstly I note that once I pressed the X it cleared the status BUT now no-where does it indicate that the drive is preceared. We should probably preserve that pre cleared flag and indicate it somewhere on the GUI.

 

Screen_Shot_2016_03_12_at_6_14_49_PM.png

 

Then I tried to run the Test and I got a blank status field in the GUI (which was not helpful). I had to click the eye to see what had happened. Of course, the stupid disk.cfg issue. Can we resolve this?

 

Screen_Shot_2016_03_12_at_6_16_15_PM.png

 

Screen_Shot_2016_03_12_at_6_16_34_PM.png

 

After adding a disk to a slot, starting the Array, stopping the Array and then removing the disk (which meant I now have a disk.cfg) I tried to run a Test again. I was once again I got a blank status field in the GUI (which was not helpful). I had to click the eye to see what had happened. This was the -R issue reported by johnnies.black and BRiT indicating that the version of "sfdisk" included in unraid 6.2.0 beta 18 is different, and doesn't seem to support the uppercase R option.

 

Screen_Shot_2016_03_12_at_6_18_08_PM.png

 

Screen_Shot_2016_03_12_at_6_18_41_PM.png

 

Not knowing where to go from here I decided to try the next step, which was the Run Post Read Verify. Unfortunately I was not able to complete this either. The popup window opened, it allowed me to select "Run post read verify" but when I clicked Start nothing happened. It just sat there. At which point the GUI also hung for a while.

 

Screen_Shot_2016_03_12_at_6_21_35_PM.png

 

At this point. Mainly due to the HANG I went to the CLI and looked at the syslog. It appeared unRAID was spamming my Syslog with messages about the disks. I wasn't sure if what was happening was due to the preclear or something going on in unRAID. I made an educated (although it might have been a poor one) decision to decide it was something happening in unRAID (don't yet know if it is harmless) so I grabbed the diagnostics file and posted about it here:

 

https://lime-technology.com/forum/index.php?topic=47408.msg453990#msg453990

 

With only assigning the disks to the Array left and doing a Check Parity I decided to take stock at what the failed attempts at the above operations had done to the status in the Preclear GUI and the Unassigned Devices GUI and this was the result. Messages and Status' were confused. It also didn't seem to grasp that the Seagate had been precleared either.

 

Screen_Shot_2016_03_12_at_6_33_11_PM.png

 

Screen_Shot_2016_03_12_at_6_33_23_PM.png

 

 

UPDATE 4: I started the Array fine. Formatted the disks (nice to see the default is XFS now as I didn't check this) and started a Parity Sync.

 

Screen_Shot_2016_03_12_at_7_02_56_PM.png

 

Screen_Shot_2016_03_12_at_7_03_46_PM.png

 

Screen_Shot_2016_03_12_at_7_05_21_PM.png

 

 

UPDATE 5: Finished (without  step 4 as indicated above). Parity Sync Complete. unRAID 6.2-beta18 seems to have been fine with the work that the Plugin has done. Dual Parity Protected Array**

 

Screen_Shot_2016_03_12_at_7_33_25_PM.png

 

**Note: For those that are 6.2 conscious, I have noted how the 2nd Parity Disk appears in the Unassigned Devices area and have posted this in the Unassigned Devices support thread here:

 

https://lime-technology.com/forum/index.php?topic=45807.msg453974#msg453974

 

 

gfjardim: Please let me know if you would like anything else tested / done. Hope this helps.

Link to comment

Trying to start a preclear but getting assigned disk error.

 

Please run the command below and send the the file preclear_beta.txt created on your flash drive:

 

bash -x /usr/local/emhttp/plugins/preclear.disk.beta/script/preclear_disk.sh /dev/sde >/boot/preclear_beta.txt

Link to comment
  • Squid featured, unfeatured and pinned this topic
  • Squid unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.