Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add

Simple enough.  Thanks.

 

The next issue is that a new WD WD20EARX 2TB hard drive is not completing the preclear script.  It has Advanced Format capability and I have tried preclear using both the -A option and without (Disk Settings, Default partition format: MBR: 4K aligned has been applied through the interface).  In one of the WD brochures I found for the hard drive, it indicated a jumper could be used to enable/force the AF capability.  I am not using a jumper on the hard drive.  No reports are generated so I can't post anything.  The preclear script makes it past the ten Steps, but then indicates the disk could not be precleared.  Time elapsed was a bit under 7 hours, which from what I have read doesn't seem very long.  (That sounds like the writes to the disk failed)

 

Any suggestions on how to proceed from here?  I was able to successfully clear an older HD.

 

Thanks.

The jumper ONLY applies for an EARS drive, yours is a different model, it is an EARX drive.  No jumper is used with that drive, ever.  (so you are good for that)

 

If the drive can not pre-clear, try a different cable, a different disk controller port, and if nothing works, RMA it.

You can use the -A option.  It is as good as any for your drive, although it would work just fine with either partition alignment.  The setting has absolutely nothing with the inability to be cleared. 

 

As far as how to proceed, use the older disk that was able to be pre-cleared.

 

Joe L.

Can I preclear a 3TB drive under unRAID 4.7 so it will be ready to use in its fullest under unRAID v5.0 latest beta?

 

Can I preclear a 3TB drive under unRAID 4.7 so it will be ready to use in its fullest under unRAID v5.0 latest beta?

yes.    Just make sure you use the latest version of the preclear script (1.13 is the newest at this time)

 

Joe L.

I have just completed preclear v1.13 on two Seagate 3TB 7200.12 drives.  One completed with no errors, and the other reported a problem.  The issue seems to be this line in the postread_errorssdb file (copied here from the preclear output screen):

  ========================================================================1.13

  ==  ST3000DM001-9YN166    S1F025HB

  == Disk /dev/sdb has NOT been precleared successfully

  == skip=80600 count=200 bs=8225280 returned instead of 00000

  ============================================================================

I then ran "dd if=/dev/sdb count=200 bs=8225280 skip=80600 | sum", and got this:

  200+0 records in

  200+0 records out

  00000 1606500

  1645056000 bytes (1.6 GB) copied, 17.3523 s, 94.8 MB/s

If I'm reading this output properly, it seems that the blocks (and disk) are ok.

 

One of my two drives is installed poorly (outside the box with cables snaking inside).  I didn't get a chance to confirm that this is the one with the problem, but  I wonder if I had some kind of glitch during the postread.

 

Here is the short preclear smart report for this drive at the end of the run (full smart report is attached)

** Changed attributes in files: /tmp/smart_start_sdb  /tmp/smart_finish_sdb

                ATTRIBUTE  NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS      RAW_VALUE

      Raw_Read_Error_Rate =  114    116            6        ok          63043384

        Spin_Retry_Count =  100    100          97        near_thresh 0

        End-to-End_Error =  100    100          99        near_thresh 0

          High_Fly_Writes =    98    100            0        ok          2

  Airflow_Temperature_Cel =    73      72          45        ok          27

      Temperature_Celsius =    27      28            0        ok          27

No SMART attributes are FAILING_NOW

Update:  preclear_disk.sh -t /dev/sdb reports the drive as precleared...

 

Am I safe to continue?  Will unraid even let me add this disk to my array?

 

smart_sdb.txt

I think you'll be fine. 

 

It appears as if your disks did not respond at all at that point in the post-read-verify, possibly due to an intermittent cable.  The discrete command you issued did work as expected.  It showed the set of blocks as zero.

 

You can always re-run just the post-read verify process to confirm, but I suspect your cabling issue to be the root cause.

preclear_disk.sh -V /dev/sdX

 

Joe L.

 

 

 

Great - thanks.

I had a third drive that somehow the preclear stopped working at 80%.  Since I'm restarting that '-V' run I decided to verify this drive too.

Cheers,

Tim

- wrong thread  -

There was a discussion over on another thread about running out of memory while running preclear. This happened to me yesterday. It is my first time trying preclear on a 3TB disk. I think it probably finished the pre-read but sometime later the webgui was unresponsive. I couldn't get powerdown to finish until I killed dd. After rebooting everything seems fine so I started preclear again using the parameters in Joe L.'s post above but it is only 35% finished with pre-read after nearly 12 hours. I am sure the real problem is all the plugins I had running.

 

I have 4GB memory. If I disable my plugins should I be able to preclear 3TB using the default parameters so I can get this done sooner? Are other users able to preclear 3TB using the defaults without running out of memory?

 

There was a discussion over on another thread about running out of memory while running preclear. This happened to me yesterday. It is my first time trying preclear on a 3TB disk. ...

Could you identify that 3TB disk (by device-name: /dev/sdX)

and report the output of:

fdisk -l /dev/sdX

replace X appropriately, and that -l is  "dash (lower-case) ell"

[Feel free to mask out the (reported) Serial #]

 

I suspect this whole problem derives from a misconception about disk geometry, and (maybe) the fix is simple, and efficient.

 

Are other users able to preclear 3TB using the defaults without running out of memory?

Yes, i have precleared a Toshiba 3T with the standaard preclear command. This was done in 25 hours (i3-2100t processor and 16 GB RAM on a sata-600 interface )

Could you identify that 3TB disk (by device-name: /dev/sdX)

and report the output of:

fdisk -l /dev/sdX

replace X appropriately, and that -l is  "dash (lower-case) ell"

[Feel free to mask out the (reported) Serial #]

 

I suspect this whole problem derives from a misconception about disk geometry, and (maybe) the fix is simple, and efficient.

Well I am currently preclearing the disk again using preclear_disk.sh -w 65536 -r 65536 -b 200 /dev/sdX

and it took more than 24 hours just for the pre-read. Now it is writing zeros (11% done). Interestingly, this step seems to be going a lot faster than the pre-read. I did not disable any of my plugins so we'll see if it crashes again but so far I can still get to the webgui.

 

Here is fdisk -l output:

Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1      267350  2147483647+   0  Empty
Partition 1 does not end on cylinder boundary.
Partition 1 does not start on physical sector boundary.

 

Yes, i have precleared a Toshiba 3T with the standaard preclear command. This was done in 25 hours (i3-2100t processor and 16 GB RAM on a sata-600 interface )

I guess with 16GB anything is possible.

 

Well I am currently preclearing the disk again using preclear_disk.sh -w 65536 -r 65536 -b 200 /dev/sdX

and it took more than 24 hours just for the pre-read. Now it is writing zeros (11% done). Interestingly, this step seems to be going a lot faster than the pre-read. I did not disable any of my plugins so we'll see if it crashes again but so far I can still get to the webgui.

 

Here is fdisk -l output:

 ...
Units = cylinders of 16065 * 512 = 8225280 bytes
... 

Thanks for that info. Unfortunately, it does not support my theory. Since the only reports of the "out of memory" condition seemed to occur with 3TB+ drives, (and since drives > 2TB might not fit into fdisk's notion of "geometry"), I was expecting to see a "Units=" value much greater than that 8225280, which would have caused preclear to suck up a correspondingly greater amount of memory. But, alas ...

 

It is still a fact that no good will come from preclear using that "Units=" value for determining the block-size it will pass to dd using the bs=N argument. The whole notion of disk geometry is meaningless in any post-DOS (& CP/M ) OpSys.

 

Now, as for your result using the override parameters. By specifying a 64KB block-size, you will definitely reduce dd's memory usage, and it should have only a negligible impact on the performance of dd itself. *BUT*, because you were advised to use "-b 200" (in conjunction with that 64KB block-size), that has caused your (overall) read performance to deteriorate. That is because each invocation of dd (by preclear) is only transferring ~12.8 MB (200*64KB) instead of the previous 1.6GB (200 * 8225280). That is the (household) equivalent of emptying your bathtub with a tablespoon (1/2 oz) instead of a pitcher (~64oz). The end result is the same (finished read cycle/empty tub), but not the time to perform it.

 

The fact that the write cycle did not seem to suffer also makes sense, but is harder (for me) to explain (to you) [my shortcoming!] If I were talking to a colleague, I would say "the write cycle doesn't suffer because sufficient write-behind data is pending to absorb the added [dd] invocation overheads."

 

You can get the safety of the (reduced) "-w 65536 -r 65536" and not have any performance penalty by increasing the -b parameter to  compensate. Ie, -b (block-count) should be increased to ((8225280 / 65536) * 200), which is (~) 25600 (ie, 128x).

So, "-w 65536 -r 65536 -b 25600" should do it. (Then you'll be emptying by the pitcherful again.)

 

 

..., "-w 65536 -r 65536 -b 25600" should do it. (Then you'll be emptying by the pitcherful again.)

Thanks. I'll keep that in mind for next time.

 

Finished writing zeros and am now on the post-read, 41+ hours so far. I think 30+ of that was all pre-read so I guess I can expect another 30+ for post-read. Doesn't seem to be an option to let me start over and only do post-read using the larger -b, but I guess I will just count it as further burn-in/testing.

 

 

 

..., "-w 65536 -r 65536 -b 25600" should do it. (Then you'll be emptying by the pitcherful again.)

Thanks. I'll keep that in mind for next time.

 

Finished writing zeros and am now on the post-read, 41+ hours so far. I think 30+ of that was all pre-read so I guess I can expect another 30+ for post-read. Doesn't seem to be an option to let me start over and only do post-read using the larger -b, but I guess I will just count it as further burn-in/testing.

To run the post-read-verify only on a drive.

      preclear_disk.sh -r 65536 -b 25600  -V /dev/sdX

The option is there to just do the post-read verify. ( -V )

To run the post-read-verify only on a drive.

      preclear_disk.sh -r 65536 -b 25600  -V /dev/sdX

The option is there to just do the post-read verify. ( -V )

I had even put the -? output in a file so I could study it. Don't know how I missed that option.

 

Post-read 2% complete in <18 minutes, so about 15 hours to go instead of 30.

 

Thanks

 

I recently had a backplane fail, and as a result ended up with redballed drives.  I have removed them from the array, replaced the back backplane and now want to add the drives back in.  Luckally, there was no data on this group...

 

My question, can / is there a command to just quickly preclear the drive without having to do a full preclear?  I know the drives are good, I have already precleared them several times...

 

What exactly does the -Z option do?  Will this preclear a drive and allow me to readd them to the array quickly?

 

Thanks,

 

Tony

I recently had a backplane fail, and as a result ended up with redballed drives.  I have removed them from the array, replaced the back backplane and now want to add the drives back in.  Luckally, there was no data on this group...

 

My question, can / is there a command to just quickly preclear the drive without having to do a full preclear?  I know the drives are good, I have already precleared them several times...

Sorry, but the best you can do is skip the pre-and post read phases. ( -n option)

 

What exactly does the -Z option do?  Will this preclear a drive and allow me to readd them to the array quickly?

The -z option writes zeros to the first  512 bytes of the disk. (usually referred to as the master-boot-record)    It will make the drive look completely un-partitioned.

 

Joe L.

Be careful.  I'm not sure if the intent was to salvage the data on the drives or not.  The above will get it working again, but won't that wipe all data?  It seems that if the backplane failed the data is ok still and the ask is about how to tell Unraid to trust the drives (?)

Is there a quicker way?  The BP failed on a Norco 4224.  One of them in the middle.  So Unraid redballed the drives, but there is NO data on them yet.  Luckally, I had not pushed enough TB of data over to actually hit that row of disks.

 

The Parity disk was not in the group.

 

I have removed the drives and put them into another pc with the intent to preclear them and readd them to the array...

 

I was hoping there was some preclear script command to clear enough of the disk to fool unraid into thinking it is empty and not do another full preclear on the disk.

 

Outside of that I am using the -n option to skip pre/post read...

I was hoping there was some preclear script command to clear enough of the disk to fool unraid into thinking it is empty and not do another full preclear on the disk.

If you did add the preclear signature without clearing the disk, you would definitely corrupt any subsequent disk rebuilds until a full correcting parity check was run. Is the data on your other disks important to you?

At this point, I think the best solution is to drop parity from the array temporally, preclear the drives, readd them to the array and then restart parity again.  I saw some procedures regarding this.  I have 6 out of 23 drives with data on them at the moment.  I do NOT want to lose that data at all!  I spent close to 2 months converting blu-rays to iso's to store on the  server...

 

I believe, I print a copy of the current drive config, clear the config, readd the drives in the same order along with the now precleared drives and after the array comes up I'll restart parity.  I'm dropping the Norco case all together, my new Supermicro case is on it's way.  Just want to be ready for when it arrives.

 

Thanks for everyones help.

Is there a quicker way?  The BP failed on a Norco 4224.  One of them in the middle.  So Unraid redballed the drives, but there is NO data on them yet.  Luckally, I had not pushed enough TB of data over to actually hit that row of disks.

 

The Parity disk was not in the group.

 

I have removed the drives and put them into another pc with the intent to preclear them and readd them to the array...

 

I was hoping there was some preclear script command to clear enough of the disk to fool unraid into thinking it is empty and not do another full preclear on the disk.

 

Outside of that I am using the -n option to skip pre/post read...

If you do not have a parity disk assigned, and have already gone through a preclear cycle, then just assign the disks and set a new disk configuration, formatting any that need formatting.

 

When you add the parity disk it will be calculated on whatever is on the disks.

 

Oh yes... no-files != all zeros.  When unRAID formatted a disk there are a lot of sectors written, even though you've not yet written any files to the disk.

When I've formatted disks, the disk says ~90MB have been written if I recall correctly.

I haven't precleared a disk in over a year and I received a 2TB WD Red drive from my brother as a Xmas gift. Has anything changed that I should be aware of since then? It's been so long that I've forgotten all the commands and steps to doing a preclear too.

If I install a Hot Swap bay on my server can I preclear (without adding to array) a newly inserted disk without stopping the server? Will it be recognized?