1

Not sure what happened over the weekend but I came to work Monday am with no email, internet and only some working network drives. Im in the process of upgrading our virtual environment, just getting funding now. I was limping this VMware 5.5 along and now Im out of options I think for any kind of support.

So I can’t tell if I had a network issue or a VMware issue but I lost a DHPC server a DC and a test machine. I can see the exchange .vmdk in one of the datastores but can’t figure out how to mount it. If there are any 3ed party vendors who do emergency support

Id be open to that also another set of eyes would be great.

7 Spice ups
2

First question , what changed? What were you doing before the weekend? Do the desktop computers on your network get an IP address and can they access the internet?

3

Nothing changed thats what puzzling. Im trying not to make any changes until Im able to get VMware current.

4

Have you checked any of the physical devices? Have you lost any storage (ie Hard Drives or has the RAID volume if there is one gone bad)? Do you have a SAN, VSAN, etc.? Sorry about all of the questions here but I usually try to KISS on something like this. We don’t know enough about your environment yet to be able to help.

Also; I see that your US based - have you reached out to a nearby MSP/Consulting company in your area that you trust yet to assist you?

Sorry for the questions but were genuinely trying to help you.

CK

1 Spice up
5

You have been backing up, right?

6

Tell us a bit more about your environment such as …

  1. Are you just running ESXi 5.5 as the hypervisor on bare metal?
  2. Any vCenter involved. If so, is that on its own bare metal running under Windows or is it the VCSA (virtual appliance).
    VMware 5.5 went EOL in September 2018, so I don’t think you have any kind of support from VMWare. Typical vendor response is to get up to a supported version - and very likely in this case – get on to supported hardware. All this being much more than you want to tackle right now.
    As for the .vmdk, have you tried to create a new virtual machine with no disk and simply attach that one?
    Do check your datastore for free space and corruption.
    Browse the files in /var/log (via SSH login).
    At this point, I’d also be looking at things using an SSL certificate, whether it is self-signed or not.
    This kind of unexpected poof likely means an SSL has expired. Or a vmware service that uses one won’t start because of it.
    If an SSL is not self-signed, it has a chain-of-trust. What I mean is there are intermediate and certificate authority (CA) certs that could have also expired. Intermediate and CA SSLs usually have a validity period on the order of 10 years, so at 3.5 years past EOL its more likely that came to pass recently.
    Back in 2018, purchased SSLs could be bought for over 2 years. Further back in 2015 or so, you could buy them for up to 4 or 5 years. Today we’re down to 397 days (1 year, 32 days) and are on the tail end of those SSLs having a 2 or 3 year term.
7

Check through the datastores to see if:

  • All the datastores are still there. You may have used an iSCSI or NFS datastore that was disconnected due to some reasons making the VMs that resided on that datastore become inaccessible.
  • The folders and VMDK files of your DHCP, DC, and Test machine are still present.

If the VMDK files are still there, you can recreate the virtual machines from scratch (just make sure to configure them as before) and attach those existing VMDK files to the new virtual machines to boot them. If that does not work, you can attach VMDK files to any existing running virtual machine to save the data. If virtual machines do not boot at all, there are some other options to extract their content http://www.vmwareblog.org/4-ways-extract-content-vmdk-vm-totally-dead/ .

Unfortunately, if files or folders are gone, there is no way to recover them, so you have to restore your missing virtual machines from backup.

1 Spice up
8

Turns out we lost a vdisk, there were six disks and two failed so gone. Its hard to tell why, the HP P2000 only had logs back as far as Monday. Everything went down on Saturday. I rebuilt the DC and DHCP last night network recourses came back. I’m going to have to recover Exchange from back up (probably take forever).

9

Consider implementing some additional disk state monitoring to avoid those kinds of situations in the future. Also, two failed disks should not lead to data loss usually. Are you running RAID5?

I’m going to have to recover Exchange from back up (probably take forever).

You have mentioned that exchange.vmdk is still around on one of the datastores. Probably you may try to reuse it if it is fine and accessible?

1 Spice up
10

The .vmdk is in the CK folder. Yeah RAID 5

eca6e61a-f2fa-402b-b522-15f719f7ab5b-Screenshot_2022-02-17_075831.png
eca6e61a-f2fa-402b-b522-15f719f7ab5b-Screenshot_2022-02-17_075831.png800×517 76.7 KB
11

My restore has been running for a 13 hours and Im at 38%. Then I still need to restore the database from last week.

12

The long recovery time is not a high price to pay for getting your data back compared to a complete data loss. Good luck with restoring your virtual machines, and make sure to avoid using HDD RAID5 in production for the future https://www.starwindsoftware.com/blog/raid-5-was-great-until-high-capacity-hdds-came-into-play .

1 Spice up