[SOLVED] VERY slow preclear on disk(s) attached to PCIe SATA card

No joy!

 

I tried adding both irqpoll and irqfixup to the syslinux.cfg by modifying the line "append initrd=bzroot rootdelay=10 irqpoll" to try each option.

 

The only difference is abysmal boot times (dare I say, almost Windows like?) and a noticeable performance lag.

 

Right before the login prompt appears is says disabling IRQ #16 regardless of what I add to syslinux.cfg

 

Does my syslog indicate anything else I might try?

 

Try any and/or all of these options.  Do a search on the net to learn how they work.

I think I had to use them on one of my older motherboards. along with iqpoll.

 

acpi=off noapic nolapic

Added acpi=off noapic nolapic to syslinux.cfg.  There appears to be some progress in resolving the issue.  USB1, sata_mv and eth0 are all now on IRQ #11.  A preclear is running on the disk attached to the PCIe controller.  It is currently running at > 100 MB/s on the pre-read.  I have started unRAID web and run unMenu and unRAID Main from within unRAID Web.  IRQ #11 was not disabled.  Preclear continues to run and I have bounced back and forth between several unRAID web pages with no adverse affects other than the web access to the unRAID box appears to be a bit sluggish compared to when apic was enabled.

The one undesirable side affect of adding "acpi=off noapic nolapic" to syslinux.cfg is that when unRAID booted I could not login to the server from the keyboard attached to the server (it's been there for convenience in all my testing).  The keyboard input is dead.  I figured it was because it is a PS/2 keyboard, so. I plugged in a USB keyboard.  BOOM! IRQ #11 disabled 

With the USB keyboard plugged in USB 2.0 port, IRQ #11 gets disabled when the server boots. I plugged it into one of the USB 3.0 ports (what a waste! not that I need the USB 3.0 ports on this box for something else) and it booted without disabling IRQ #11.  However, a few minutes later, IRQ #11 is disabled. So, I went back to the PS/2 keyboard; server boots, no keyboard input, no IRQ #11 disabled and I can sill telnet and access the server from the keyboard on my Windows box.  Preclear is running in a Telnet session.

I tried to output the latest cat /proc/interrupts results to a file and now it appears I cannot write anything to the flash disk as it says I have a "read-only filesystem."  On to troubleshooting the next issue.  Apparently, I can't write to the array either everything is read-only.  One step forward, one step back.

Run check disk in the flash in a PC or Mac.

you should be able to capture the /proc/interrupts from the telnet screen.

It seems you have a conflict with the USB ports. No dua...

Disable all USB 3.0 and XHCI, see what happens.

try moving the keyboard and/or usb key into other ports.

The idea is to separate as many interfaces as possible.

I am surprised that the ps/2 keyboard had issues.

Try turning off ACPI in the bios. Sometimes that hangs off an IRQ.

You can also experiment with adding IRQPOLL to the list of options appended.

You may need to try all sorts of variations on these options.

I have the feeling that the lan adapter driver does not share interrupts well.

As far as laggy emhttp interface while preclearing. That's normal from my experience.

you should be able to capture the /proc/interrupts from the telnet screen.

It seems you have a conflict with the USB ports. No dua...

 

Disable all USB 3.0 and XHCI, see what happens.

 

try moving the keyboard and/or usb key into other ports.

The idea is to separate as many interfaces as possible.

I am surprised that the ps/2 keyboard had issues.

 

Try turning off ACPI in the bios. Sometimes that hangs off an IRQ.

After running it twice, chkdsk on unRAID flash drive appears to have resolved the "read-only" issues.

I have disabled USB 3.0 and XHCI. PS/2 keyboard now works; not sure how those are related.

Preclear started on unRAID console.  Web interfaces all functioning without LAN traffic causing problems.  All looked good.  Switched monitor input to unRAID server - BOOM, IRQ #11 disabled (maybe just poor timing - or switching video signal also messes with IRQs).

Here is the latest cat/proc/interrupts with USB 3.xhci disabled:

           CPU0       
  0:      63525    XT-PIC-XT-PIC    timer
  1:        175    XT-PIC-XT-PIC    i8042
  2:          0    XT-PIC-XT-PIC    cascade
  3:        322    XT-PIC-XT-PIC    ahci
  5:         26    XT-PIC-XT-PIC    ehci_hcd:usb2
11:     400405    XT-PIC-XT-PIC    ehci_hcd:usb1, sata_mv, eth0
12:          3    XT-PIC-XT-PIC    i8042
NMI:          0   Non-maskable interrupts
LOC:          0   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RES:          0   Rescheduling interrupts
CAL:          0   Function call interrupts
TLB:          0   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:          3   Machine check polls
ERR:          0
MIS:          0

Next step, play with all the irqpoll, apic options in different combinations.

Yeah, I think the NIC driver is a problem.

I'm off to church now, maybe praying is my only hope 

I think I'll order the Intel DQ67EP board just to see if the Intel NIC/BIOS on this board will play nicely, although even it requires unRAID v5 beta.  The Supermicro mini-ITX MBs I mentioned are not an option as they use the G2 mobile CPU socket not the regular 1155 desktop CPU socket.  I can't even find any of the mobile i3 CPUs in retail channels.

Do you have any other PCIe SATA controllers to test in that slot?

It would help isolate if it's the MV8 driver or the realtek driver.

Also at some point limetech used the realtek driver, then found issues and used the linux driver for the realtek card.

Perhaps trying different versions of the 5.x series.

See this post

http://lime-technology.com/forum/index.php?topic=16125.msg149210#msg149210

And this

http://lime-technology.com/wiki/index.php?title=UnRAID_Server_Version_5.0-beta_Release_Notes

I think in Beta 10 there was this

use Realtek r8168 driver instead of linux r8169 driver

Of course, with APIC disabled, Shutdown from unRAID web page does not shutdown the server.  I am going to try one of the betas with the Realtek driver and APIC enabled and see what happens.

It looks like both beta 10 and beta 12a have the native Realtek driver?

Changes from 5.0-beta12 to 5.0-beta12a
--------------------------------------
- emhttp: inventory up to 26 total storage devices
- linux: remove built-in r8169 driver
- linux: add Realtek r8168 driver version r8168-8.025.00 (for Realtek PCI-Express NIC's)
- linux: add Realtek r8169 driver version r8169-6.015.00 (for Realtek PCI NIC's)
- linux: include /lib/firmware directory for e100, bnx2, tigon drivers
- linux: increase scsi command time-out from 10sec to 30sec

or

Changes from 5.0-beta9 to 5.0-beta10
------------------------------------
- afp: upgraded netatalk to version 2.2.0-p6
- afp: added "TimeMachine volume size limit" parameter for AFP
- afp: always mount array/cache disks with extended attributes enabled
- linux: upgraded to version 2.6.39.3
- linux: use Realtek r8168 driver instead of linux r8169 driver
- linux: added Adaptec AIC79xx U320 support
- linux: added Adaptec AIC94xx SAS/SATA support
- linux: added UDF file system support
- shfs: implement persistent inode numbers (AFP fix)

I'll try 12a first then 10.

Well, that was quick. 

- Loaded v5 beta 12a on a USB stick.  (this version contains the Realtek NIC driver from Realtek)

- No extras at all, not even unMenu. 

- Default syslinux.cfg without the noapic line

- Booted tower

- Log in as root

- Switch monitor to Windows 7 desktop

- Access //tower/main from browser

- Assign a couple of disks

- Hit monitor switch to return to console on Tower - "disabling IRQ #16" appears immediately.  Monitor signal switching seems to affect interrupts.

Next test - various irqpoll, irqfixup, noapic combinations in syslinux.cfg

At least I have eliminated the slight possibility of any add-ons contributing to the problem.

why not try without the KVM in there to switch the monitor back and forth.  Granted this should not be happening anyway, but it seems it is possible that that KVM might have something to do with this.

why not try without the KVM in there to switch the monitor back and forth.  Granted this should not be happening anyway, but it seems it is possible that that KVM might have something to do with this.

I agree. every now and then my KVM does weird things and sends all sorts of spurious or repeated characters to the machine.

why not try without the KVM in there to switch the monitor back and forth.  Granted this should not be happening anyway, but it seems it is possible that that KVM might have something to do with this.

I am not using a KVM.  A separate keyboard is connected to the unRAID server.  My desktop monitor has several video inputs and I have the tower connected to the monitor via a DVI cable.  I am just switching inputs on the monitor.  I don't have a spare monitor at the moment that I can dedicate to the unRAID server and, of course, I need video when I am playing with the BIOS settings.  Maybe I can round up a monitor for the tower.

Oh, if that is the case then the monitor is probably not the issue.  I assumed KVM when you said that you were swtiching back and forth.

If the USB keyboard / monitor is the issue, why have them connected at all?

If the USB keyboard / monitor is the issue, why have them connected at all?

Because I need a monitor and keyboard attached for tweaking the BIOS.  I do not believe the monitor is the issue as I used to be switching back and forth without IRQ #16 being disabled by a video switch.  That is a recent and now every time occurrence and must have been triggered by a BIOS tweak that I am trying to track down.  Before that became a trigger, it appeared to be NIC traffic that would trigger the disabling of IRQ #16 (or #11 when using nopic switch).  I have been playing with BIOS settings to see if any of them affect IRQ assignments/APIC.

What I have is a moving target right now.  I have ordered the Intel DQ67EP board to see if its BIOS handles IRQ better than the Biostar.

They don't need to be connected to run the server.  Change the bios and test to see if things are working after removing them, as this test should be closer to the actual usage.

There's a problem with the IRQ assignments and a driver not sharing the IRQ's nicely.

Whenever there is input on the USB port and network traffic, the machine has issues.

This means if you happen to update the flash by writing to it while heavy network traffic is occuring, the same problem will manifest.

Go with a tested, tried and true motherboard. Time is money and you've already spent allot of time on this.

There's a problem with the IRQ assignments and a driver not sharing the IRQ's nicely.

Whenever there is input on the USB port and network traffic, the machine has issues.

 

This means if you happen to update the flash by writing to it while heavy network traffic is occuring, the same problem will manifest.

 

Go with a tested, tried and true motherboard. Time is money and you've already spent allot of time on this.

Yep, that's why I am trying the Intel ITX MB, although reports are it has trouble running headless and complains if no keyboard and monitor are attached.  I figure Intel ought to be able to figure out IRQ management.  It's their chipset, CPU and NIC on this board.

I wish the Supermicro ITX offerings did not use the mobile G2 socket and used the standard desktop 1155 socket.  Although I cannot find any of the mobile i3 CPUs in retail channels, I did find one on eBay that, supposedly is "new" pulled from an HP laptop (why would anyone pull a "new" CPU out of a "new" laptop?).  If it stays at .99 and $5 shipping, I'll pick it up and try the Supermicro if the Intel also has issues.  I don't know why Supermicro thinks I want a mobile CPU on a "server" board.  It also uses SODIMM laptop RAM.  The advantage to the Supermicro is that it has 6 onboard SATA ports instead of the 4 of the Biostar and Intel MBs.

Go with a tested, tried and true motherboard. Time is money and you've already spent allot of time on this.

Fortunately (or not, depending on how you look at it), I am in a lull right now between contract jobs and do have some time while no one is paying me any money!  In 14 days I'll be off on a two-month job in the Asia/Pacific region, so, I am trying to get my unRAID server all settled down before then since I will be neglecting it for two months. 

There's a problem with the IRQ assignments and a driver not sharing the IRQ's nicely.

Whenever there is input on the USB port and network traffic, the machine has issues.

 

This means if you happen to update the flash by writing to it while heavy network traffic is occuring, the same problem will manifest.

 

Go with a tested, tried and true motherboard. Time is money and you've already spent allot of time on this.

 

Yep, that's why I am trying the Intel ITX MB, although reports are it has trouble running headless and complains if no keyboard and monitor are attached.  I figure Intel ought to be able to figure out IRQ management.  It's their chipset, CPU and NIC on this board.

 

I wish the Supermicro ITX offerings did not use the mobile G2 socket and used the standard desktop 1155 socket.  Although I cannot find any of the mobile i3 CPUs in retail channels, I did find one on eBay that, supposedly is "new" pulled from an HP laptop (why would anyone pull a "new" CPU out of a "new" laptop?).  If it stays at .99 and $5 shipping, I'll pick it up and try the Supermicro if the Intel also has issues.  I don't know why Supermicro thinks I want a mobile CPU on a "server" board.  It also uses SODIMM laptop RAM.  The advantage to the Supermicro is that it has 6 onboard SATA ports instead of the 4 of the Biostar and Intel MBs.

You could have gone with the Supermicro Atom ITX motherboard. That has the CPU already. IPMI, just add ram.

For a file server, you don't need all that much CPU power.  I love IPMI.

AS far as mobile CPU pulls., people upgrade them or destroy the laptop accidentally. so they part them out.

I have used core 2 duo mobile CPU's for my servers for years. Less power , less heat and good processing power.

My 2(two) HTPC pc's windows 7/XBMC both use core 2 duo 2.33Ghz mobile processors. 

my 2(two) Administrative servers (torrent and login firewall) both use mobile 2.26 single core & 2.33 dual core ghz mobile processors.

The only reason I went with the i3 instead of an Atom CPU was that I may want to do video encoding for streaming.  I may end up with the Supermicro anyway even though the i3 mobile CPU runs at "only" 2.2 GHz compared to 3.1 for the desktop version.  Supposedly, encoding requires at least a 2.8 GHz CPU but, I think the mobile versions are capable of stepping up to 3.0 GHz on demand.

Fair nuff...

Is the Asrock H67M-ITX an option?  http://www.asrock.com/mb/overview.asp?Model=H67M-ITX

I have one in my Hackintosh, but I have not tried it with unRAID thus far.

That MB also has the Realtek 8111E NIC as does my Biostar MB.  At this point, I think this NIC/driver may be part of the issue, so, I am going to try the Intel and, possibly, the Supermicro boards.  They both have Intel NICs.

My Intel MB arrives tomorrow, hopefully, it's BIOS is better at handling IRQs.

I had a preclear running on the disk attached to the PCIe SATA controller.  It had been running without interruption for 15 hours and was 32% through the post-read process.  This is faster than other preclears using the MB SATA ports.  I started the preclear from the unRAID server console, switched video input to my desktop and started a telnet session to the server.  I was careful to not switch back to the unRAID console input on the monitor and to watch preclear progress and performance through unMenu.  I started the array, read from it on and off throughout the day; wrote files to it, etc.  It was slightly sluggish sometimes and just as quick as always sometimes, but, I attributed this to the preclear running at the same time.  No problems.

My big mistake came this morning when I started up my desktop.  The monitor, of course, detected an active input on the unRAID server and switched there instead of my desktop PC - immediately - IRQ #16 was disabled and post-read speed dropped to 2.5 MB/sec from 112 MB/sec.  Had I manually switched the monitor to the desktop input before booting up to the desktop I think I could have avoided that problem.

Switching the monitor input is the only action that now seems to trigger disabling IRQ #16.  My current setup uses the standard syslinux.cfg.  I am not using irqpoll, irqfixup or noapic switches.  I don't like unresolved mysteries so, I have another day to play before swapping MBs and seeing if that resolves the issue.

As someone pointed out earlier in this thread, I could just run headless which is what I will be doing anyway, but, I want to know why switching a monitor input messes with IRQs.  Network activity no longer seems to affect it like it did before.

I have tweaked BIOS settings dealing with graphics.  Instead if initializing PEG/IGD (PCIe graphics card - I don't have a graphics card in the PCIe slot as that's where my SATA controller is), it is now set to initialize IGD which I think is the built-in graphics only).  Other than that, I don't think I have changed anything relating to graphics.  Fun stuff.