1

OK maybe I am over selling this a bit, but your typical planned down time gone awry!

Please resist sending advice as A) I am past the issue and B) I had (and have) plenty of paid for resources at my disposal so I am cared for.

The Plan:

Replace (2) Existing VM Ware host servers with brand new shiny ones on an ESXi Essentials bundle (only licensed for 3 host servers)

Add 10 GB modules to Switch chassis

Upgrade SAN to use 10 GB iSCSI instead of 1 GB

Planned 4 hours of down time at an almost 24/7 company.

When and Where is went WRONG

At hour 3 all we needed to do was integrate the newly racked servers into the vCenter and start moving VM’s onto them. You know the step where you simply “Add Host” and put in the Name or IP and Credentials, 2 seconds later it is added… WRONG!

After banging our heads against it for some time I insisted on getting VMWare support on the line. They did a lot of the same things. Hours passed as we tried again and again looking for what was holding us back.

At first the versions didn’t match but we got that straightened out.

Then we had certificate issues. Got that straightened out.

Still no love.

Had weird stuff like you could login to the new Hosts through other computers on the network but for some reason we could ping the hosts from each other but the vCenter server could not load the vSphere on the new hosts!?!?

Well there is not shining light as to what fixed our little problem but after hours of trying increasingly desperate tactics (Turning off services totally unrelated, uninstalling software that we really need but trying something/anything), Grasping at straws.

Somewhere around 3 hours after the downtime was scheduled to end the new host machine finally agreed to join the vCenter. We had done nothing from the past time we tried. I think we were trying to check sharkwire for network traffic at the time, but no need because magically the new host added!?!?

Also one of the two new hosts refused to scan the iSCSI and recognize the Datastores. We decided to attack that today since we at least had two working hosts and felt comfortable going home so late and worn out.

Don’t you know it the next day a little bump on the iSCSI nic and then whammo it was now seeing the Datastores, then add to the vCenter like a champ on the first try!?!?

Still no clue as to what the magic was, but makes a wonderful story of pain and misery…

This is why I hate IT! Ugh but I still love some of it.

@support-team-vmware

27 Spice ups
2

I’ve had days like that…Definitely not fun. Especially when you have no clue what you did that worked, so the root of the problem may still be lurking.

1 Spice up
3

Sounds like a barely-functioning connection on the NIC.If when you jogged it, everything suddenly worked, you probably shifted the contacts and it made a better connection.

3 Spice ups
4

What this shows with good planning it is the little things that can throw a big wrench into the project. You stuck with and by some good luck got it all working which is all that matters at the end of the day. Good Job. At first I thought you had looked into a mirror and scared yourself.

3 Spice ups
5

Those are the days I sing to myself “Everything I touch, I break”.

2 Spice ups
6

I hate it when that kind of stuff happens. But I think I know what caused the issues though:

aliens.jpg
aliens.jpg800×700 121 KB
2 Spice ups
7

Persistence…

dont_give_up3_square_sticker_3_x_3.jpg
dont_give_up3_square_sticker_3_x_3.jpg750×750 82.9 KB
3 Spice ups
8

Now I’m going to go through my whole network and jiggle all the cables to see what I can break, LOL

3 Spice ups
9

Makes for a really long day.

1 Spice up
10

Yea… I keep finding flaky network connections on our hosts after disconnecting and sliding them out to do maintenence, new fans/ram/whatever. It is hard to detect until later when you really try using every NIC. It’s simpler with a client pc, 1 NIC, 1 point of failure. With a server that has 9 NICs… the points of failure go up and some philosopher’s law backhands you.

Months ago I had a server (Vmhost2) with a flaky connection. If I bounced the port, it would go 1 gig for a short time then downgrade to 100 meg. This affected the servers vlan which was bonded on multiple NICs. Somehow every so often a server VM would try talking on that NIC and it would be lost to the rest of the world. I’d bounce the port on the switch and it would come back for a while. Being it was redundant, I ended up disabling it from Vcenter until I could visit the color and replace the cable and test the cable. I finally got to the COLO after a series of unfortunate events,I was sicker than a dog from a wisdom tooth infection for 2 months, I finally got in and replaced the cable. I re-enabled the NIC from Vcenter and tested the problem cable with 2 basic cable testers. Both pass. I chopped the ends off and chucked the cable anyways, noting the bad ju ju has been purged. Am I a little vindictive? Possibly.

After replacing a fan on a server (Vmhost1) last month, last week I discovered a NFS/main host NIC was running 100 mbit full duplex. I bounced the switchport it from the switch, Vcenter squeaked it lost connectivity to the vm host (ooops) while it settled on 100 meg, I watched from the switch as it went 1 gig, then blinked offline several times before downgrading to 100 mbit then Vcenter was happy again. I stopped fussing with it and logged in my notes, “Need to go to the COLO sometime and reseat Vmnic3’s cable (4 port NICs) or replace the cable and burn it as an effigy for all network problems.” Must burn the unclean!

4 Spice ups
11

VMware…iSCSI…4 hours…i wish you luck, good thing you had VMware support, at least you can see you did nothing wrong if they basically replicated everything you did

1 Spice up
12

This is why I love it when users or observers question your and your vendors’ skills/products. They scream, “It’s a Logical device! There has to be a fix! It has to be explainable…” That’s when you get out your magic Witch Doctor totem, quickly sacrifice an unfortunate nearby rodent to Cthulu over the Dark Sigil Odegra, wave the totem at whatever’s been failing and…

…it all ‘just starts working’.

There is no scientific or official explanation, and the incident is never spoken of again. By ANYONE.

1 Spice up
13

for future reference…

If you have two hosts and are licensed for 3 hosts.
you could have taken as long as you like adding 1 new host to the current set up
Migrated VMS off one of the old hosts and removed that host making space for the 2nd new host and so on.

if you do not apply any licenses to the new hosts and leave them as trials then you will also be able to use v motion and storage v motion to drag vms off the old hosts with no downtime.

i’m aware that you ran into issue adding the new host.
but rather than a do or die situation, you could have taken as long as was needed to diagnose the issue to your satisfaction be it a week or a month.

1 Spice up
14

Colin,
I wish we had. In fact I thought that was the plan but space in the rack as it was and overzealous to get done one server was evacuated and the two servers racked at the same time to save time. We also changed the vlan on the switch so there wasn’t space for the iSCSI either. Believe me I kept eying up the old server wondering if we could just fall back, though I didn’t want to do this again later…
Oh well. The things you learn days after a panicky night of banging your head against the same problem repeatedly.

1 Spice up