Blog: Networking

I learned the reason that VMware suggests having service consoles for ESX hosts on at least two distinct networks last week. I was troubleshooting intermittent backup issues with Veeam on a customer network and couldn’t really find any pattern to the failures. Two or three backups in a row would run successfully, then 5 in row might fail. The behavior was very random. However, the failures were always on Virtual Machines associated with a specific ESX host. At first I thought the host was healthy, but after watching the VI client for an extended period of time, I noticed that the ESX host would drop offline (showing disconnected in the VI client) and then come back online again.  This indicated the problem wasn’t just affecting the management/backup server. [more]

In order to level set my troubleshooting efforts, I decided to reboot this ESX host. However, after the reboot, I could not connect to it with the VI client. I could ping the IP assigned to the service console, but couldn’t SSH or connect via the VI client. I logged in via iLO and found that an ifconfig at the command line returned IP = 0.0.0.0…..interesting. So what is responding to my pings. I checked the arp cache on one of the switches and found that a thin client had been plugged in that had the same IP as my LAN service console. What is really odd is the MAC address for the thin client was all zeros AND the IP I was using for the LAN service console is not even available to be distributed by DHCP. I was not able to connect to the thin client to see how it was configured, but I was able to connect to ESX host via a second service console port that I placed on the iSCSI network. The management/backup server has a connection to the iSCSI network to do backups to disk so I was able to change the LAN-facing service console IP to another IP and everything started working fine. The backup issue was obviously being caused by changes in the arp entries on the backup server between the thin client and the ESX host. So, be aware that at boot-time, if ESX determines that the IP it is using for a service console is already in use, it just rips it out of the configuration and continues to boot with NO WARNINGS or ERRORS on the console.


 

Back several months ago I tried to update my laptop to Snow Leopard (OSX 10.6).  Most things worked great, but at the end of the week when I started doing some of my reports, I noticed lots of file system problems.  The Word documents I was editing would become read-only after I saved them once.  New documents I created would be read-only.  As I got to digging, I found that any files I created on the file server were being created with empty permissions (as viewed from my laptop), and read-only permissions (via the checkbox) as viewed from the Windows side.  I found lots of people having the same problem with no real workaround.  I noticed the permissions would fix when I viewed them from the Finder, or when I did an ‘ls –l’ from the command line.  I restored my system back to Leopard from my backup (which was nice to have available) and waited for a fix.  Well, the fix came in the recent release of Snow Leopard 10.6.3.  I’ve updated again and everything works great.


 

We had a problem with a new Xerox ColorQube printer that was not allowing users to use the hole punching or stapling features (something that they had been promised by Xerox would work). When printing a job they would click on the printing preferences and then try to choose either hole punch or staple and they were both grayed out. When I was logged onto my account the option was not grayed out. This at first lead me to believe that the problem was a rights issue but it turns out it was not. I noticed that when I logged on with my account onto another terminal server that the options were grayed out for me as well. This lead me to compare settings within the printing preferences on my two profiles. [more]

I finally found through this that the problem was within the paper size choice. If the paper size setting was set to mixed output you were not allowed to choose to use the hole punching or stapling features (they were grayed out). When the size was specified such as legal or letter size the options would then become available.  It makes sense why the stapling and hole punching options were disabled with that paper size, but the user interface was less than intuitive.


 

I upgraded from Vista to Windows 7 about three weeks ago.  I decrypted my PGP encrypted drive before the upgrade and, after the upgrade, PGP recognized my disk wasn't encrypted and prompted me to encrypt my drive.  I started the encryption process but wound up pausing the process because of slow performance, intending to resume it after hours.  I installed some Windows and Lenovo (ThinkDamage…probably my 2nd mistake) updates which required a reboot.  After the reboot, PGP started trying to install itself and produced this error message…

"You cannot upgrade or remove PGP while a whole disk is processing. Installation terminated." [more]

I was unable to access the PGP console in order to resume the encryption, decrypt, etc.  An attempt to uninstall PGP produced the same error.  This was not good since I was scheduled to leave town on an audit within 24 hours and thought I might have to abandon the upgrade to Windows 7, restore a backup and re-encrypt the old Vista image before I left town.

A coworker suggested I log a ticket with PGP.  After doing so, I was poking around their site, searching for various terms from the error message and stumbled across a reference to a command line command.  About that same time, I received an auto-response from PGP which included several links, the last of which led me to information about the same command line command, pgpwde.

Here is the relevant section from the page above:

SECTION 2 - PGPWDE Command Line

The following commands will help diagnose and decrypt the disk. Other commands can be listed by typing pgpwde --help.

  1. To begin working with the PGPWDE interface open a command prompt and change to the PGP installation directory (default directory shown) C:\Program Files\PGP Corporation\PGP desktop.
  2. To list all installed hard disks in the system type: pgpwde --enum. Entering this command will give us a list of disks with numbers we will use in the next few steps.
  3. Now type pgpwde --status --disk 1. Substitute the PGP WDE disk number listed in the previous step for the number 1 in the command if different. The output of this command will tell us whether the disk is still encrypted.
    • If the disk is not encrypted, "Disk 1 is not instrumented by bootguard" will be the output.
    • If the disk is encrypted, the output will display:
      • "Disk 1 is instrumented by Bootguard."
      • The total number of sectors.
      • A Highwater value (number of sectors encrypted).
      • Whether the current key is valid.
  4. Type pgpwde --list-user --disk 1. This will tell us the user information contained on the disk. This will help in multi-user environments to determine which user passphrase was used to implement WDE.
  5. Type pgpwde --decrypt --disk 1 --passphrase {mypasswordhere}. This will start the decryption process. To view progress, type the status command listed in step 3 and note the Highwater number, this number will get smaller and smaller as the number of sectors encrypted decreases.

This command line command allowed me to decrypt the partially encrypted disk.  I then uninstalled PGP to be safe, reinstalled PGP and encrypted my disk without further incident.


 

I recently upgraded a Windows server to the latest version of Symantec Endpoint Protection and the server was no longer accessible on the network after the upgrade.  The server would not respond to network requests even though the console was working.  It turned out the full SEP feature set was installed, including the SEP firewall.  Additionally, now the firewall policy was applied to the server.  This caused the SEP firewall to isolate the server from the network. 

 To fix the problem I uninstalled Enpoint Protection and reinstalled without the firewall feature set.  I also applied a firewall policy just in case the firewall feature was installed on that server again.  My suggestion is to modify the SEP installation document, so that a firewall policy is not applied.


 

One way to exclude directories (thus not single files or filettypes) of roaming profiles to be placed on the servers is by using the Group Policy Object:

  • User Configuration
  • Administrative Templates
  • System
  • User Profiles
  • "Exclude directories in roaming profile" [more]

You can enable this and type in the folders you want to exclude.  You only type the name of the folder from the root directory of the profile.  So if you want to exclude "D:\Documents and Settings\tuser\Application Data\Microsoft\Internet Explorer\UserData" then you type in “Application Data\Microsoft\Internet Explorer\UserData”.  For extended folder entries you separate each by a semi-colon:  "UserData;Cookies;My Documents;Temp;Start Menu;Application Data\Microsoft\Internet Explorer\UserData;"

Be sure to include a semicolon at the end.

To verify delivery to the targeted user accounts, go to a device where a targeted user account has logged on and check the following registry key manually: HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows\System


 

I’ve been running the Office 2010 beta for a while, although I’ve seen this problem occur on Office 2007 as well. Periodically, I’ll lose my ability to select text with the mouse inside Outlook. It just simply won’t work. Closing and restarting Outlook always fixes the problem, but it’s an annoying problem to have to deal with. After some searching on the internet, a solution from Microsoft popped up. From http://support.microsoft.com/kb/940791 [more]

Problem Description:

You install an automatic update for Microsoft Office Word 2007 on a Windows Vista-based computer and then restart the computer. If Word 2007 was running when the computer was restarted, you experience one or more of the following symptoms:

  • The mouse does not work when you use Word.
  • You cannot open a Word document from the Search window in Windows Vista.
  • You cannot open a Word document from Windows Desktop Search.
  • Word crashes when you try to start or exit Word.
  • Word crashes when you open the Open dialog box.
  • Word crashes when you save a document.
  • Word crashes when you close a document.

The fix is simply to open the registry, browse to and delete the following registry subkey: HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Word\Data

Then close and restart your Word applications (Word, Outlook, etc.). So far, this seems to have fixed my problem, although I’m going to give it another week or two before I call it comfirmed.


 

At a client site, I have been testing some automated ways to move users from v1 to v2 profiles. All their users are on Windows XP and we are moving them to Windows 7. I was looking specifically for a graceful way to allow interoperability between the profile versions and keep us from having to touch every user profile to copy over data. What I found was a little annoying. There is really only two ways to migrate data from v1 to v2 profiles.

  1. Use Folder redirection to share data between the profile versions by redirecting relevant data to a network share that can be used by both profiles.
  2. Use the user state migration tool [more]

If you are NOT using roaming profiles, the USMT is the best way. If you are using roaming profiles, the folder redirection is the best way. The gotcha here is to make sure you create the folder redirection policy FIRST on a Windows Vista, 7, or 2008 system BEFORE editing it on a Windows XP or 2003 system. There is something about the way the GPO is created/built that will not allow it to apply to vista, 7 , and 2008 systems if it is created with XP or 2003 first.


 

I was trying to use Cisco’s Adaptive Security Device Manager (ASDM) to connect to our ASA in the office.  I was getting an authentication error but I knew my credentials were correct and it was working for another engineer.  The Java console contained the error “java.io.IOException: Authentication failure”.  I found several references to proxy issues related to this error, so I went to the Network Settings section of the Java app in the control panel and manually specified our proxy server (including the local bypass addresses) and it started working.  The proxy setting was set to “use browser settings” but obviously this wasn’t working.


 

Conditions:

  1. Machines that used to run ISA Firewall client
  2. Uninstallation of ISA Firewall client
  3. New PROXY settings configured
  4. SEP 11.5 installed.

Many machines began getting errors in the application logs from Event Source: crypt32, Event ID: 8.  The description of the error says “Failed auto update retrieval of third-party root list sequence number from: [more]http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootseq.txt”.

I eventually stumbled across a few forums that eventually led me towards this issue happening after installing SEP 11.5.  What seemed to be happening is that the machines attempted to update its root certificates from Microsoft Update at two hour intervals.  The machine will attempt to connect using the SYSTEM account, so it is important that this account also has the correct PROXY settings.  It is likely that after removal of ISA Firewall client, the settings for the SYSTEM account were left in the registry pointing to the old PROXY server. 

The SYSTEM account can always be found in the registry at HKEY_USERS\S-1-5-18. I found that on machines that were not working, the registry keys under HKEY_USERS\S-1-5-18\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\DefaultConnectionSettings were pointing to the old PROXY script whereas the working ones were pointing to the correct PROXY wpad.dat configuration file.  I had to pull the settings from a newer system because this registry key is a binary key, so you cannot simply type the value.

Be sure that the machine also has unauthenticated user access allowed through any web filtering appliance to www.download.windowsupdate.com also.  
 
More information can be found at http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/1f626f1854285036802574e4002de4c7?OpenDocument