VMWare ESX network setup-best practice

thompsma73 · 2012-12-17T21:40:36.000Z

Greetings.

We are running two hosts at ESXi 4.1.0. Each host is a Dell PowerEdge R610 with ~64GB RAM and six NIC’s in each host. Everything is working, but my gut tells me that our networking setup is not according to “best practice”.

I reviewed the ESX Configuration Guide (www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf) and the iSCSI SAN Configuration Guide (www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf). After reading these, I more convinced that I need to take a deeper look at our network configuration.

In a nutshell, vSwtich0 handles our iSCIS traffic and has 3 NIC’s (vmnic0,1,4). Should I seperate each physical NIC on a seperate vSwitch?

We have two requirements for iSCSI traffic. Of course, we need to provide a solid, redundent connection to our SAN for our host and we also need to provide a port group for VM iSCSI traffic to establish iSCSI connections from our virtual servers.

I have attached a screenshot of our existing network configuration.

@VMware @Dell_Technologies

joe9013 · 2012-12-17T21:49:29.000Z

I’ve read conflicting things on the ‘best practice’. I have my hosts set up 2-2-2 for NICs, so 2 do VM traffic, 2 handle vMotion traffic, and 2 handle iSCSI. And my hosts are set in Round Robin for storage access, which made for a slight (but noticeable) performance bump. In this setup, everything is separated but still allows for redundancy. Works nicely. YMMV.

Edit: Pretty picture.

Capture.PNG398×559 15.9 KB

dabeast · 2012-12-17T22:39:49.000Z

fusmichaelt:

In a nutshell, vSwtich0 handles our iSCIS traffic and has 3 NIC’s (vmnic0,1,4). Should I seperate each physical NIC on a seperate vSwitch?

If you did this, you would have no redundancy on the iSCSI traffic. Each vSwitch would have it’s own uplink - if that uplink failed, whatever ports that are on that vSwitch will be network isolated.

thompsma73 · 2012-12-17T22:57:52.000Z

Joseph0486: Interesting. Silly question, but from your picture, it looks like you cannot assign a iSCSI port to a VM for an software iSCSI connection. Many of our virtual servers have the need to connect to a SAN volume. Is this a correct assumption on my part? I like your vSwitch1 logic. Did you have to “bind” the iSCSI ports as directed on page 72 of the ESX Configuration Guide (www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf)?

da Beast: In my setup, I have 3 physical NIC’s plugged into that virtual switch (teamed together). Correct me if I am wrong, but wouldn’t that provide redundancy?

dabeast · 2012-12-17T23:02:52.000Z

fusmichaelt:

da Beast: In my setup, I have 3 physical NIC’s plugged into that virtual switch (teamed together). Correct me if I am wrong, but wouldn’t that provide redundancy?

da Beast:

fusmichaelt:

In a nutshell, vSwtich0 handles our iSCIS traffic and has 3 NIC’s (vmnic0,1,4). Should I seperate each physical NIC on a seperate vSwitch?

If you did this, you would have no redundancy on the iSCSI traffic. Each vSwitch would have it’s own uplink - if that uplink failed, whatever ports that are on that vSwitch will be network isolated.

I just answered your question - you now have redundancy - if you moved your physical nics each to their own vSwitch - you will no longer have redundancy.

Did I answer your “In a nutshell” question or did I read it wrong?

dabeast · 2012-12-17T23:07:19.000Z

fusmichaelt:

Joseph0486: Interesting. Silly question, but from your picture, it looks like you cannot assign a iSCSI port to a VM for an software iSCSI connection. Many of our virtual servers have the need to connect to a SAN volume. Is this a correct assumption on my part? I like your vSwitch1 logic. Did you have to “bind” the iSCSI ports as directed on page 72 of the ESX Configuration Guide ( www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf )?

Why wouldn’t you just connect an RDM lun?

joe9013 · 2012-12-17T23:13:21.000Z

fusmichaelt:

Joseph0486: Interesting. Silly question, but from your picture, it looks like you cannot assign a iSCSI port to a VM for an software iSCSI connection. Many of our virtual servers have the need to connect to a SAN volume. Is this a correct assumption on my part? I like your vSwitch1 logic. Did you have to “bind” the iSCSI ports as directed on page 72 of the ESX Configuration Guide (www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf)?

da Beast: In my setup, I have 3 physical NIC’s plugged into that virtual switch (teamed together). Correct me if I am wrong, but wouldn’t that provide redundancy?

We only have one VM that uses iSCSI, and it isn’t connecting to a SAN volume, (or more correctly, not a volume on our Dell SAN). vDisks all the way for us.

As for that document, I did not follow that, I followed documentation from Dell. Basically the same thing though, except Dells was all command line.

thompsma73 · 2012-12-17T23:17:08.000Z

da Beast: That’s a good question… Why wouldn’t I…? I will, from this point forward. Thanks for that!

StorageNinja · 2012-12-18T03:33:00.000Z

da Beast:

fusmichaelt:

In a nutshell, vSwtich0 handles our iSCIS traffic and has 3 NIC’s (vmnic0,1,4). Should I seperate each physical NIC on a seperate vSwitch?

If you did this, you would have no redundancy on the iSCSI traffic. Each vSwitch would have it’s own uplink - if that uplink failed, whatever ports that are on that vSwitch will be network isolated.

?!?!? You can do MPIO with separate switches (In fact, I’m a big fan of doing this so I can easily see which kernels are attached to which NIC’s without having to edit the config). You bind each iSCSI NIC to its own vKernel under best practices, so even if they are on the same vSwitch its not like they share links anyways, and are still dedicated paths. You NEVER use link aggregation for iSCSI.

StorageNinja · 2012-12-18T03:36:37.000Z

da Beast:

fusmichaelt:

Joseph0486: Interesting. Silly question, but from your picture, it looks like you cannot assign a iSCSI port to a VM for an software iSCSI connection. Many of our virtual servers have the need to connect to a SAN volume. Is this a correct assumption on my part? I like your vSwitch1 logic. Did you have to “bind” the iSCSI ports as directed on page 72 of the ESX Configuration Guide ( www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf )?

Why wouldn’t you just connect an RDM lun?

Because RDM’s are a massive pain in the ass. You have massively reduced visibility into performance. You have to set perennial reservations if you do them at scale or else the hosts take an epoch and then some to reboot. Physical RDM’s don’t support snapshots, and you loose the ability to pick up VMDK’s and move them. Your backup and recovery options are limited (CBT etc). RDMs are a last resort, a necessary evil, generally driven by MSClustering. Most vendors do NOT support RR pathing with RDMs (despite it being the default PSP). I’ve seen more accidental path thrashing because of RDM’s poorly configured. There’s just too much you can do wrong with them. You can’t consolidate multiple disks into a single LUN so your SAN configurations require a ton more work. You loose access to powerful Queue Depth management features (SIOC).

In short RDMs suck, use VMFS.

StorageNinja · 2012-12-18T03:37:18.000Z

fusmichaelt:

Joseph0486: Interesting. Silly question, but from your picture, it looks like you cannot assign a iSCSI port to a VM for an software iSCSI connection. Many of our virtual servers have the need to connect to a SAN volume. Is this a correct assumption on my part? I like your vSwitch1 logic. Did you have to “bind” the iSCSI ports as directed on page 72 of the ESX Configuration Guide (www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf)?

da Beast: In my setup, I have 3 physical NIC’s plugged into that virtual switch (teamed together). Correct me if I am wrong, but wouldn’t that provide redundancy?

Why are you not reading the best practices of your storage vendor? What SAN are you using?

Pathing, configuration, and SCSI options can vary a bit depending on your vendor. What are you using?

dabeast · 2012-12-18T05:12:50.000Z

john773:

da Beast:

fusmichaelt:

Joseph0486: Interesting. Silly question, but from your picture, it looks like you cannot assign a iSCSI port to a VM for an software iSCSI connection. Many of our virtual servers have the need to connect to a SAN volume. Is this a correct assumption on my part? I like your vSwitch1 logic. Did you have to “bind” the iSCSI ports as directed on page 72 of the ESX Configuration Guide ( www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf )?

Why wouldn’t you just connect an RDM lun?

Because RDM’s are a massive pain in the ass. You have massively reduced visibility into performance. You have to set perennial reservations if you do them at scale or else the hosts take an epoch and then some to reboot. Physical RDM’s don’t support snapshots, and you loose the ability to pick up VMDK’s and move them. Your backup and recovery options are limited (CBT etc). RDMs are a last resort, a necessary evil, generally driven by MSClustering. Most vendors do NOT support RR pathing with RDMs (despite it being the default PSP). I’ve seen more accidental path thrashing because of RDM’s poorly configured. There’s just too much you can do wrong with them. You can’t consolidate multiple disks into a single LUN so your SAN configurations require a ton more work. You loose access to powerful Queue Depth management features (SIOC).

In short RDMs suck, use VMFS.

agreed on the RDMs suck - but if they are already using software iSCSI at the guest level it is the same as an RDM at the host level. I wouldn’t set it up this way on a new install.

StorageNinja · 2012-12-18T05:29:07.000Z

da Beast:

john773:

da Beast:

fusmichaelt:

Joseph0486: Interesting. Silly question, but from your picture, it looks like you cannot assign a iSCSI port to a VM for an software iSCSI connection. Many of our virtual servers have the need to connect to a SAN volume. Is this a correct assumption on my part? I like your vSwitch1 logic. Did you have to “bind” the iSCSI ports as directed on page 72 of the ESX Configuration Guide ( www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_server_config.pdf )?

Why wouldn’t you just connect an RDM lun?

Because RDM’s are a massive pain in the ass. You have massively reduced visibility into performance. You have to set perennial reservations if you do them at scale or else the hosts take an epoch and then some to reboot. Physical RDM’s don’t support snapshots, and you loose the ability to pick up VMDK’s and move them. Your backup and recovery options are limited (CBT etc). RDMs are a last resort, a necessary evil, generally driven by MSClustering. Most vendors do NOT support RR pathing with RDMs (despite it being the default PSP). I’ve seen more accidental path thrashing because of RDM’s poorly configured. There’s just too much you can do wrong with them. You can’t consolidate multiple disks into a single LUN so your SAN configurations require a ton more work. You loose access to powerful Queue Depth management features (SIOC).

In short RDMs suck, use VMFS.

agreed on the RDMs suck - but if they are already using software iSCSI at the guest level it is the same as an RDM at the host level. I wouldn’t set it up this way on a new install.

Remount it as a vRDM, then perform a storage vMotion to another datastore, and you can turn a vRDM into a VMDK and get away from it almost non-disruptively. Get quick at swapping from direct iSCSI to RDM and you can blip a server over to RDM in under 10 seconds

This is the Nicholson LUN double swap. Don’t try this on SQL without stopping services

dabeast · 2012-12-18T06:39:22.000Z

Interesting…

joe9013 · 2012-12-18T15:06:48.000Z

You can monitor some performance metrics in ESXi by using Putty to connect to the server, then running the command ‘esxtop’. If you look at the network connections, it can also tell you which vNic the VMs are using. This is nice if you are monitoring ports and looking for patterns.

thompsma73 · 2012-12-18T23:11:55.000Z

John773: Why do you do this to me I will digest this information and then follow-up with this discussion.

thompsma73 · 2012-12-18T23:12:36.000Z

I am using a Dell Equallogic PS6000 series SAN.

joe9013 · 2012-12-18T23:29:28.000Z

fusmichaelt:

I am using a Dell Equallogic PS6000 series SAN.

This may be helpful in your config.

http://www.dellstorage.com/WorkArea/DownloadAsset.aspx?id=1090

StorageNinja · 2012-12-19T00:28:33.000Z

fusmichaelt:

John773: Why do you do this to me I will digest this information and then follow-up with this discussion.

And the storage Ninja having confused everyone slinks back into the darkness… My work here is done…

thompsma73 · 2012-12-19T22:37:18.000Z

Ok… still reading up on this, but I am learning the errors of my ways. Question, though. How would vMotion happen? Could this be done over the “Management” network (vSwitch)? We have 6 nics, I would like to use 3 for iSCSI, but that leaves only 3 nics for VM’s, MGMT, and vMotion. We have a segregated iSCSI network (10.100.0.0 /16). Our production network is 10.1.0.0 /16. Suggestions?