Background setup:
This site has VMware vSphere 5.0 hosts which are connecting to NFS datastores on a NetApp SAN/NAS. There is a dedicated switch stack of Dell PowerConnect 5524 switches between the NetApp and the VMWare hosts.
Issue description:
Over the last couple weeks I have been seeing where VMWare virtual machines would pause or in some cases disconnect sessions. The Windows event log would consistently record an Event ID 129 with a Source of LSI_SAS: "Reset to device, \Device\RaidPort0, was issued." I did some further research and found that this event is usually generated when there is high I/O on the SAN. However, the SAN at this location wasn’t experiencing high I/O.
I started to notice the following NFS disconnect error while I was logged into the SAN:
nfsd.tcp.close.idle.notify:warning]: Shutting down idle connection to client (192.168.1.10) where receive side flow control has been enabled. There are 0 bytes in the receive buffer.
[more]
Resolution:
Per NetApp’s best practice document, flow-control should be disabled on the storage network when using modern hardware. I had flow-control enabled on the switch and the SAN and this apparently was causing the disconnect issues.
http://media.netapp.com/documents/tr-3749.pdf
Do you remember playing those games where you would look at two seemingly “identical” pictures and try to spot the 5 differences between the two? I’d like to play that, except there’s only one difference between these two:
Here’s the situation. We’re hosting files off of a CIFS share on our NetApp device. We needed to back up this CIFS data using Backup Exec. I went through and set up NDMP on both the NetApp and Backup Exec so that the communication could be established for Backup Exec to connect to the CIFS share and grab a copy of the data. In order to get the resource credentials to show successful, I had to set the Logon Account for the NetApp to be our domain Backup Exec Service Account (BESA) and the actual CIFS share to be the NDMP user account I had created for backup.
Secondly, there is another gotcha associated with how to set up the NDMP account for backup. If you use the root account, you can simply use the regular password as if you were logging into the NetApp device from the backup server. However, if you decide to create a new account, you must encrypt the account on the NetApp device. To create the account, do the following:
useradmin user add <login_name> -g “Backup Operators”
(Type in the password)
ndmp password <login_name>
The output after the password is what you will copy and paste into the backup server so that the NDMP user can authenticate.
We were receiving the following error on one of our customers Netapp SANS:
Autosupport (WEEKLY_LOG) cannot connect to url support.netapp.com/asupprod/post/1.0/postAsup through proxy (Could not find hostname '""', hostname lookup resolution error: Unknown host)
So I tried to check the auto support proxy configuration setting:
netapp> options autosupport.support.proxy
autosupport.support.proxy ""
After many attempts to remove “” from the proxy server configuration, I finally broke down and contacted support. The technician also tried a number of things to change the configuration and ultimately found the solution after he asked one of his colleagues. The way to get rid of the "" is a little counter-intuitive. You simply apply the following command: [more]
netapp> options autosupport.support.proxy ""
which yields you the following result:
autosupport.support.proxy
Fight fire with fire, I suppose.
Also, NetApp has a pretty cool tool that shows you the progress of you SnapMirror syncs. (SAN to SAN replication)
NetApp SnapMirror Progress monitor:
http://now.netapp.com/NOW/download/tools/smpm/