CoNetrix

Blog: EMC

Last week, one of our customers experienced a system wide outage due to a new MAC on the network. That’s right… it is typed correctly. During the setup of a MAC OS Lion 10.7 installation, one of the users had a home folder set in their Active Directory properties. When the MAC user logged in and the laptop attempted to map the user’s home folder, the EMC Celerra NS40 NAS that serves as the back-bone of the network serving CIFS shares and about a dozen iSCSI LUNs to Microsoft Exchange, SQL Server, and VMware hosts, experienced a kernel panic and crashed….very ungracefully I might add. The outage only lasted about 2-3 minutes while the datamover failed over to the standby. However, it took nearly two hours to clean up the carnage post-failure. Not to mention, two hours outside of business hours to fail the unit back to the primary datamover.

Specifics about the exact cause are not 100% clear, but two things are known. First of all, MAC OS 10.7 uses a different SAMBA client than previous versions. Formerly, MAC OS used a bundled open source SAMBA software for Windows file share and network directory services. With 10.7, they have rolled their own and replaced the open source code with their own flavor of SAMBA. According to EMC, the root cause of the issue is that MAC OS 10.7 passes a NULL value in one of the SAMBA message headers. Obviously, this is a problem for the Celerra. No word as to how gracefully Windows file server handle this issue.

Networking, Apple, SAMBA, Mac, EMC,
 

I came across a problem in one of our automated Robocopy command scripts when copying files from NTFS to EMC Celerra file system.  The problem was that every time Robocopy was told to copy files to the EMC Celerra over the VPN, the files were always detected as “newer” on from the originating source.  This caused Robocopy to copy every single file from our network to the remote network each and every time. 

I came across a command line switch for Robocopy: /FFT : Assume FAT File Times (2-second date/time granularity).  So what this does is force Robocopy to use FAT style time stamps which are 2-second granularity.  It allows enough flexibility to account for the way the time is recorded when doing a file copy from NTFS to another file system. This is needed when going between and NTFS and Linux/Unix/FAT or emulated file system.

Networking, VPN, Linux, Robocopy, NTFS, EMC,
 

We had an issue last week where backups of an Exchange 2007 server began to fail after we removed the EMC Replication Manager & EMC Solutions Enabler apps. The errors that we began to see in the Application log like this:

Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered.  This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider.  The error returned from CoCreateInstance on class with CLSID {bd902507-4491-4001-acdd-a540a2cad34b} and Name HWPRV is [0x80040154].

I went through the process described here http://support.microsoft.com/kb/940032  to reregister all the VSS stuff, but it didn’t work. After digging into the VSS CLI, I was seeing the following returned from issuing a “vssadmin list providers” [more]

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5} 
   Version: 1.0.0.7

Provider name: 'ERM VSS Provider'
   Provider type: Hardware
   Provider Id: {e929a027-cf8c-47bf-90a3-cd4241c7cace}
   Version: 1.0

It appeared as if the EMC VSS provider was not removed when I uninstalled the software. The online knowledgebase for EMC, said to fix it, re-install the apps, then start the VSS service, then uninstall the apps again suggesting that the provider would not have been removed if the service wasn’t running at the time the apps were uninstalled. I had a really hard time getting that stuff installed to start with so I didn’t want to start that again. I did some testing on a VM and found that I could remove the provider by just removing the registry key which matched the Provider Id listed by the vssadmin list providers command.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{e929a027-cf8c-47bf-90a3-cd4241c7cace}

After restarting the VSS service one time, the vssadmin list providers command provided this output

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

Success!! This could possibly be a problem and the fix could work with any applications that insert 3rd party VSS providers.

Networking, Backup, Volume Shadow Copy, Windows 2003 Server, EMC, backup,