Blog: iSCSI

I learned the reason that VMware suggests having service consoles for ESX hosts on at least two distinct networks last week. I was troubleshooting intermittent backup issues with Veeam on a customer network and couldn’t really find any pattern to the failures. Two or three backups in a row would run successfully, then 5 in row might fail. The behavior was very random. However, the failures were always on Virtual Machines associated with a specific ESX host. At first I thought the host was healthy, but after watching the VI client for an extended period of time, I noticed that the ESX host would drop offline (showing disconnected in the VI client) and then come back online again.  This indicated the problem wasn’t just affecting the management/backup server. [more]

In order to level set my troubleshooting efforts, I decided to reboot this ESX host. However, after the reboot, I could not connect to it with the VI client. I could ping the IP assigned to the service console, but couldn’t SSH or connect via the VI client. I logged in via iLO and found that an ifconfig at the command line returned IP = 0.0.0.0…..interesting. So what is responding to my pings. I checked the arp cache on one of the switches and found that a thin client had been plugged in that had the same IP as my LAN service console. What is really odd is the MAC address for the thin client was all zeros AND the IP I was using for the LAN service console is not even available to be distributed by DHCP. I was not able to connect to the thin client to see how it was configured, but I was able to connect to ESX host via a second service console port that I placed on the iSCSI network. The management/backup server has a connection to the iSCSI network to do backups to disk so I was able to change the LAN-facing service console IP to another IP and everything started working fine. The backup issue was obviously being caused by changes in the arp entries on the backup server between the thin client and the ESX host. So, be aware that at boot-time, if ESX determines that the IP it is using for a service console is already in use, it just rips it out of the configuration and continues to boot with NO WARNINGS or ERRORS on the console.

I have been working on VMware vSphere v4 for the last few days and have found that enabling jumbo frames support is not as intuitive as the admin guide makes it sound. The process goes something like this.

1. Enable jumbo frame support on iSCSI SAN (MSA 2012i)
2. Enable jumbo frame support on switch (a Linksys business class in this case)

Both of these are very straight forward and simple to do, but here is where it gets a little tricky. vSphere v4 jumbo frame support for the iSCSI network must be enabled with the CLI. Two things must be done. You must enable jumbo frames on the vSwitch and then able it on the VMkernel port/ports. The command to enable JF enable on the vSwitch looks like this: [more]

esxcfg-vswitch -m <MTU> <vSwitch>

In my case, I wanted a 9000 MTU (9KB) for vSwitch1, so it was ‘esxcfg-vswitch -m 9000 vSwitch1’ . All good so far. Now, I needed to JF enable the VMkernel port. Admin guide says this is the command

esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name>

So my command looked something like this esxcfg-vmknic -a -I 192.168.155.21 -n 255.255.255.0 -m 9000 “iSCSI VMkernel”

When I ran this command, it said that this port group didn’t exist. Exactly…that is why I am running this command…to create the VMkernel port and associated port group. After MUCH troubleshooting, I finally found that the following sequence WILL work. It must have something to do with the way the port groups and their associated identifiers are created.

1. Create the VMkernel port in the vi client (GUI)
2. Remove the VMkernel port from the CLI with this command esxcfg-vmknic –d “iSCSI VMkernl”
3. Recreate the same VMkernel you just deleted with JF enabled esxcfg-vmknic -a -I 192.168.155.21 -n 255.255.255.0 -m 9000 “iSCSI VMkernel”