Vmware hosts network ports

cameronlemaster3558 · 2012-06-14T06:32:13.000Z

I was hoping I get some opinions on what would be the most realistic setup in the following situation.

We recently ordered some quad port network cards (Intel I350-T4) for our virtual hosts (HP DL360 G7’s running esxi 5.0 U1). Our hosts connect to our SAN (HP P2000) via iSCSI.

Now my question is how many ports do you think I should use for IP traffic, and how many for iSCSI? There are 8 total ports on each host now, 4 onboard, 4 from the new NICs.

Any opinions would greatly appreciated.

keith22 · 2012-06-14T06:45:14.000Z

We have 10 Ports and this is how we were configured by some VMware Support Agents according to best practices.

Capture.JPG417×673 132 KB

alaricwhitney · 2012-06-14T07:28:02.000Z

MHCKeith wrote:

We have 10 Ports and this is how we were configured by some VMware Support Agents according to best practices.

The funny thing is that your configuration is no longer supported by VMWare as of ESXi5.

Here’s what we use as per VMWare’s current best practices:

2 separate vSwitches (1 VMKernel each with 1 iscsi NIC each, you can no longer have more than one NIC per iSCSI vmkernel, or else you run the risk or crashing your SAN with too many open connections).

1 vswitch with at least 2 (or more) NIC’s for normal data

1 vswitch with 1 vmkernel for vmotion with at least 2 NICs

1 vswitch with 2 NIC’s for your DMZ network.

keith22 · 2012-06-14T07:35:26.000Z

We are setup the same. While the iSCSI are on the same vSwitch, they are 1-to-1 bound to an adapter and then the iSCSI Initiator is bound to each of those VMkernel Adapters.

Capture.JPG752×549 88.1 KB

keith22 · 2012-06-14T07:36:33.000Z

Man I wish we could attach more than one item to each post.

Capture.JPG560×595 102 KB

keith22 · 2012-06-14T07:38:19.000Z

I guess I should also mention that we are running ESXi5 and this was setup by a VMware Engineer knowing we are running ESXi5.

cameronlemaster3558 · 2012-06-14T08:03:52.000Z

What I was advised to do by a friend that also uses vmware is to just create two vSwitches.

vSwitch1: vmkernel for iSCSI and vMotion

vSwitch2: management and data

With each switch having 4 ports for load balancing.

alaricwhitney · 2012-06-14T08:05:32.000Z

Yes, I assumed you had a single NIC bound to each VMKernel. We used to be setup that way (As that used to be the best practice), but both Dell and VMWare storage pointed us away from that, and to the newly updated best practices documentation.

With ESXi5, you need to separate out each of your iscsi VMKernels onto it’s own vswitch. ESXi5 has a bunch of issues/bugs when you have 2 iscsi vmkernels in the same vswitch (vmware still tries to use the unbound NIC, causing all sorts of trouble, which we unfortunately ran into as well).

VMWare has released several patches trying to resolve these issues, however after a little bit of googling, you’ll find that people are still running into the issues with the old configuration. After we separated out the VMKernels onto individual vSwitches, all of our problems went away.

keith22 · 2012-06-14T08:15:11.000Z

That is good to know. I will have to look at changing that in our next scheduled down time. Thanks!

redovenbird · 2012-06-14T08:15:28.000Z

I would use port 3,4 from NIC1 and 1,2 from NIC2 for iSCSI.

This way you have redundancy in case one of the card fails you wont lose storage communication.

And use the other NICs as per your application needs.

alaricwhitney · 2012-06-14T08:30:29.000Z

The awesome part is that as long as your path redundancy is working, you can make that change during business hours. Simply leave one of your vmkernels and the bound NIC on the vswitch. I did a test case at one of our sites, and then converted the rest of our sites within an hour.

Of course, I was also under pressure at the time as well, as our SAN’s were timing out from too many open connections due to the related esxi5 bugs.

keith22 · 2012-06-14T08:48:36.000Z

That is good to know but we just got done with some long downtime as the last admin had all 10 ports in an single vSwitch with vMotion, iSCSI, Management all configured on 1 VMkernal and our Production network sharing the same vSwitch.

We crashed hard and required basicly re-doing everything from scratch. We were down 9am-3pm while VMware rebuilt our network in ESX. I learned a lot that day and am continuing to learn as we go as there is no budget for any training.

StorageNinja · 2012-06-14T14:34:52.000Z

AlaricW wrote:

2 separate vSwitches (1 VMKernel each with 1 iscsi NIC each, you can no longer have more than one NIC per iSCSI vmkernel, or else you run the risk or crashing your SAN with too many open connections).

My symmetric SAN laughs at puny path limits! (see attached picture, of a ridiculous 80 paths per VM host configuration with only 10 LUNs).

You still separate out vKernel’s into separate switches, but outside of it causing issues with EqualLogics I wasn’t aware of any other SANs blowing up from too many connections. The real reason in my mind is it fully isolates you from faults. If one Kernel gets a bad SCSI command and goes crazy, or one switch fabric blows up from a bad Spanning tree setting your entire environment will not come crashing down.

johnwhite · 2012-06-15T16:56:48.000Z

MHCKeith wrote:

We have 10 Ports and this is how we were configured by some VMware Support Agents according to best practices.

I think the reason you don’t want to do this is full redundancy in the paths, to the point of keeping a vSwitch failure from bringing down your storage.

jorgedelgados8632 · 2012-06-27T13:47:52.000Z

Hi AlaricW, do you have a link to a document from Dell/VMware on the change to separate vSwicthes for iSCSI??

StorageNinja · 2012-06-27T15:54:50.000Z

Jorge.Delgado@CR wrote:

Hi AlaricW, do you have a link to a document from Dell/VMware on the change to separate vSwicthes for iSCSI??

This has been best practices for quite some time. Separate vKerenel, and vSwitch for each iSCSI connection has been standard as long as I’ve been doing iSCSI (4.0).