RAID disk fail, VMware, DELL R710

ziceman · 2018-06-15T01:26:43.000Z

Once of the DELL VMWare hosts running a RAID 10 4-disk array appears a problem drive.

They had not installed DELL System Mgmt VM app or configure the IDRaC interface, so I am only able to look at the VSphere client interface.

The info showed within is confusing to me. The majority of disk status lines show “Normal” / green check mark, there are alerts associated with multiple drives, and references to “Parity Check in Progress - Deassert” and “Rebuild Aborted”, “In Critical Array - Assert”. I have included screen shots below.

In looking at the front of the server, only 1 of the 4 disk trays has a amber light. The other 3 look fine.

I was gong to just hot swap the amber one, but the VShphere messages are giving me pause. Can anyone provide some insight for me?

Screen_Shot_2018-06-14_at_10.03.11_PM.png718×248 17.7 KB

Screen_Shot_2018-06-14_at_10.21.01_PM.png800×597 202 KB

wanneseulaers · 2018-06-15T03:06:01.000Z

From the screenshot, the only drive that shows to be failed is Drive 0 in Bay 1, the rest show to be in a critical array (because one drive is failed or failing), not failed themselves.

Going on this info and the lights on the server, I’d just swap out the drive with the amber light.

Also: R710, not R720.

ziceman · 2018-06-15T03:24:17.000Z

Thanks, Wannes. Yes, 710. My apologies.

chris.hone · 2018-06-15T04:43:31.000Z

Replace the failed drive, just be prepared for disk rebuild overhead. I just rebuilt my Compaq raid configured the same way and it took 4 days to completely rebuild and check.

harry1028 · 2018-06-15T12:00:09.000Z

Always hot swap a RAID drive.

ziceman · 2018-06-19T15:10:05.000Z

OK. Was out at the client site on Saturday, and I replaced the drive. All lights green and rebuild is under way, - but I have no way of tracking the progress.

See my related posts here:

DELL iDrAC storage missing, R710 Hardware

Cannot see the Storage Section from the iDrac interface. I have seen other posts where users suggest "this is because iDrac is initializing before the raid controller", but there does not appear to be any solutions referenced. Can the sequence be controlled? Or is a firmware upgrade needed? If the latter, why would DELL ever create the system in a manner in which the iDrac could not see their own RAID controller?

DELL R710, Open Manage install, VMWare 6.0 Virtualization

Trying to install Open Manage on a DELL R710 running VMWare 6.0. Downloaded the appropriate file and followed instructions from the following page: http://www.dell.com/support/article/us/en/19/how10613/how-to-install-openmanage-server-administrator-omsa-on-vmware-to-collect-logs?lang=en When running the esxcli installer with the uploaded VIB, I get the this: [DELL_VMWare_ESM_fail.png] I figure it is something simple, but this is production server in and high usage environment. I know a…

The VShpere health area for Storage now looks like this:

Screen_Shot_2018-06-18_at_9.35.27_AM.png556×863 22.2 KB

dota7allstars · 2018-07-23T07:53:02.000Z

Replacing the fault drive should be able to resolve the matter

ziceman · 2018-07-23T12:58:31.000Z

Sorry it took me so long to come back to this. Yes, swapping the drive brought everything back to normal (after a long rebuild).

I am still disappointed in in the lack of storage info in the iDRAC interface application. While I realize this is included in some of the newer DELL hardware platforms, it seems silly to me that an enterprise-grade remote access system would ever have been created in its first versions that actually excluded storage status. I mean, what is the point?

Anyway, problem solved for now…

davelewis2 · 2019-03-12T16:22:48.000Z

Hi,

I have a Dell T710 as well, and I’m having the same issue. I’m hoping someone sees this and could tell me how WANNES new that there was only 1 drive that failed, when his picture showed Drive 0,1,2 and 3 with a red alert on it.

Ziceman, you only only replaced the one drive right?

My picture is almost identical, except mine is Disk Drive Bay 1 Drive 5 0: In Critical Array - Assert, that the only alerts.

Thanks

davelewis2 · 2019-03-12T16:22:49.000Z

Hi,

I have a Dell T710 as well, and I’m having the same issue. I’m hoping someone sees this and could tell me how WANNES new that there was only 1 drive that failed, when his picture showed Drive 0,1,2 and 3 with a red alert on it.

Ziceman, you only only replaced the one drive right?

I My picture is almost identical, except mine is

Disk Drive Bay 1 Drive 5 0: In Critical Array - Assert

Disk Drive Bay 1 Drive 4 0: In Critical Array - Assert

Disk Drive Bay 1 Drive 3 0: In Critical Array - Assert

I Do Not have any that shows Driver Fault - Assert

Hopefully someone sees this !!!

Thanks

wanneseulaers · 2019-03-12T16:44:49.000Z

I could tell it was that drive, because that was the only one showing as failed. The rest was showing as “in a critical array”.

The reasoning is simple here:

“Drive x in critical array - Assert” : This drive belongs to an array that is not functioning properly (missing disk, bad disk, battery cache dead, issue reading out RAID status from lifecycle controller, stuff like that).
“Drive x: fault - Assert” : this drive has failed, or at least the lifecycle controller is telling esxi that this drive has failed.
“Drive x: predictive failure - Assert” : this drive is going to fail, or the lifecycle controller is telling esxi that this drive is going to fail.

If none of the alerts tell you a drive has failed or is going to fail, you can click open the battery alerts and check for any warnings or errors there.

Hope that helps

davelewis2 · 2019-03-12T17:37:30.000Z

WOW!!!,

THANK YOU so much for responding and so fast!!! I was about to purchase a a few hard drives and replace hot swap, but not sure if i should.

If i should, should i replace all the ones with alert on them. Also, would i put all 3 drives in at the same time.

Alert.jpg800×450 83 KB

Also, i’m not sure what “open the battery alerts” is

Thank You

wanneseulaers · 2019-03-13T05:25:34.000Z

From that screenshot, it doesn’t look like any drives are failed, so you shouldn’t replace any.

It looks like you have a 3 disk RAID 5 there, if more than 1 drive would have failed or you remove more than 1 at the same time, all data on that array would be gone.

You could also have a RAID 0 array, which would stop working the minute you pull a single drive.

In ziceman’s case he also had a single drive on the server giving an amber light, do you have any drives showing amber?
You should also have a small informational screen on the front of the server that shows error codes, if any.
I’d continue troubleshooting before swapping any disks at this point.

davelewis2 · 2019-03-19T15:19:27.000Z

Thanks again for responding!

i’m pretty sure it is a RAID 5, 6 harddirves, no amber lights no error messege on the display. I started all of this because the server locked up, and i was forced to go to my backups. I was called at home by the CEO that they couldn’t log on. I couldn’t get into the server, wouldn’t ping, so i was forced to go to my backups. When i got to work the server was on, looked normal, but it was froze.

That’s when i assumed it was the errors. I’ll try to do some more reasearch, my Idrac is garbage, doesn’t show anything, I’ve installed and removed Dell Open Manage essentials 4 times, but it won’t accept the credentials on set up. So i’m going to have to figure out another way to troubleshoot. Not sure where to go from there?

da-schmoo · 2019-03-19T15:36:01.000Z

Your graphic is showing those drives being a member of a Raid Array that’s in a critical state.

You need to scroll up or down to find the actual drive that failed or is having issues.

davelewis2 · 2019-03-25T12:49:52.000Z

Thanks for responding,

the picture shows the only drives that have an alert.