In working with XenServer over the past couple of months, I have found that information is harder to come by than it is with VMware. We are only using XenServer for one customer and they are using the free version so support is not an option. Up until last week, I had no need to get into the CLI of Xen much. It’s pretty easy to configure via XenCenter and our setup is pretty simple. However, the other day, our monitoring software detected an issue where the network interfaces on one of the monitored VMs was logging a high number of discards. One of the peculiar things was that the discards were the exactly the same for Tx and Rx. After some research, I decided that it would be a good idea to run off all the offloading features in XenServer. XenServer sees network interfaces in two forms: physical interfaces (pifs) and virtual interfaces (vifs). Pifs are the actual connections to the server. Vifs are the NIC interfaces of the VMs. Naturally, turning off all of this can only be done via the XenServer CLI. So, part one of the gotcha…here is a set of scripts that can help in manipulating network interfaces in Xenserver
Script to turn off all offloading techniques off on all vifs and pifs: [more]
====================================================
#!/bin/bash
if_modes="rx tx sg tso ufo gso"
if [[ "$1" == "--local" || "$1" == "-l" ]]; then
echo -n "disabling checksum offloading for local devices... "
for iface in $(ifconfig | awk '$0 ~ /Ethernet/ { print $1 }'); do
for if_mode in ${if_modes}; do
ethtool -K $iface $if_mode off 2>/dev/null
done
done
echo "done."
else
echo -n "disabling checksum offloading in xapi settings... "
for VIF in $(xe vif-list --minimal | sed -e 's/,/ /g')
do
###xe vif-param-clear uuid=$VIF param-name=other-config
for if_mode in ${if_modes}; do
xe vif-param-set uuid=$VIF other-config:ethtool-${if_mode}="off"
done
done
for PIF in $(xe pif-list --minimal | sed -e 's/,/ /g')
do
###xe pif-param-clear uuid=$PIF param-name=other-config
for if_mode in ${if_modes}; do
xe pif-param-set uuid=$PIF other-config:ethtool-${if_mode}="off"
done
done
echo "done."
fi
====================================================
- create text script file (turnOffloadingOff) using VI
- Change perms to make it a script
chmod 777 turnOffloadingOff
- Run script
./turnOffloadingOff
Other Useful XenServer Commands
- determine uuids of physical interfaces on XenServer
xe pif-list host-name-label=<hostname>
- determine parameters of the specific pif given the uuid
xe pif-param-list uuid=<uuid of pif>
- determine uuids of virtual interfaces of VMs on Xenserver
xe vif-list
- determine parameters of the specific vif given the uuid
xe vif-param-list uuid=<uuid>
- new VMs created since the script was executed will NOT have the same vif other-config settings disabled
xe vif-param-set uuid=<uuid of vif> other-config: ethtool-gso=”off”
xe vif-param-set uuid=<uuid of vif> other-config: ethtool-ufo=”off”
xe vif-param-set uuid=<uuid of vif> other-config: ethtool-tso=”off”
xe vif-param-set uuid=<uuid of vif> other-config: ethtool-sg=”off”
xe vif-param-set uuid=<uuid of vif> other-config: ethtool-tx=”off”
xe vif-param-set uuid=<uuid of vif> other-config: ethtool-rx=”off”
So on to part two of this post…this didn’t fix the problem. After a ton of additional troubleshooting, I determined that this behavior is due to the Citrix Paravirtual NIC driver. The issue goes away if you uninstall XenTools and the PV driver isn’t used. On Windows 2008 and Windows Vista/7 VMs, the PV Ethernet driver reports discards to the OS. In Windows 2003 and XP, it does not. Keep in mind the discards could be broadcast packets not intended for the VM or misc DOM0 traffic. In any case, it doesn’t make much sense, but there isn’t actually anything wrong with the VM. I ended up removing the monitoring of the VMs NIC interfaces because I did want to use the PV Ethernet driver.