Blog: Networking

My iPhone connects in my office to wi-fi which also is able to connect through my VPN router.  For my laptop, I had set the DHCP settings on my wireless router to include the internal CoNetrix DNS server.  When I connected my phone which uses Exchange active-sync to connect, it would get an error about the certificate authority being untrusted and hit OK to continue. 

Later on I noticed that my phone kept getting synchronization errors and would get the pop up about the certificate authority being untrusted.  What I later noticed was that the server name would change from our internal to external back and forth.  [more]

I later realized that our DNS server had a host record that was the same as our external mail server address.  Each time the phone went on and off my wireless network, it would keep switching server names because the internal DNS would resolve to the actual internal server name. 

I removed the DNS server for the CoNetrix internal network from my wireless router and the phone has only connected to webmail externally.  It no longer tries internal access.

Changing Product Key on Windows 2003 (and SBS2003):  I needed to change the product key on a SBS machine that was virtualized from an IT consulting customer’s machine. I did not want to re-activate the SBS machine using the customer’s key since the machine was still active and did not want to interfere with the proper operation of the production machine. I needed to install our MSDN key since this is the proper usage for that type situation.

In order to do this, when the product ask you to activate, chose the option to telephone a customer service representative to activate Windows.  Then when the Activate Windows by phone comes up, chose the “Change Product Key” at the bottom to enter the new product key. Then cancel out of this operation and activate the windows over the internet as usual. [more]

If you need to connect to a VPN that uses RSA’s SecurID authentication and if you are using the RSA SecurID App on the iPhone it can be tricky entering the SecurID passcode in the VPN connection dialog.  Fortunately you can copy and paste the passcode on the iPhone.  Open the RSA SecurID app and enter your PIN.  Press and hold the passcode field until the [Copy] appears.  Copy the passcode.  Initiate the VPN connection and paste the passcode in the appropriate field.

I have done a couple SBS 2003 to SBS 2008 migrations in the past and because the customers were relatively small with less than 20 users, we opted to just build the new SBS 2008 server with a completely new domain and then migrate the data. We would recreate all the user accounts, mailboxes, and shares and choose a maintenance window to do the switchover. It works fine for companies with low complexity and downtime flexibility. However, I just started an SBS 2003 to SBS 2008 migration for a company that operates pretty much 24-7 and they have a fairly high complexity level on the application side.

I have tried some of the SBS 2000 -> SBS 2003 migration tools and they were pretty pathetic, but I had to have an easier way for this migration. After some research, I came across a set of processes that you can use to migrate SBS 2003 -> SBS 2008. They are documented pretty well by Microsoft here: http://www.microsoft.com/downloads/details.aspx?FamilyID=52b7ea63-78af-4a96-811e-284f5c1de13b&displaylang=en.  The process is pretty tedious...I won’t get into it here. You can read the doc if you are interested.

The gotcha is what I encountered during the migration install. [more]When you provide the unattended install file for the SBS 2008 install, first it goes through and installs the OS. After a reboot, it starts the actual migration process. I was watching closely and noticed that it seemed to hang at about 20%. I looked at the event logs on both ends and couldn’t find anything then after some a search on Google I found a user blog that mentioned accessing the sbssetup.log file via file share during the install. I checked it out and it indicated that the new SBS server could not find a server in its AD site to replicate with. I reviewed AD sites and services and noticed that there was no subnet associated with the current default AD site. I added the subnet and waited a few minutes and the progress began to move forward…yeah success!

No so fast. It move about 10% and stalled again. Again after looking in the log file, it was an AD replication problem. This time the SBS 2003 server had a sysvol replication journal wrap problem. I found the MS article on how to fix it and applied the registry fix and restarted NTFRS. Again, progress.

The migration ran for another 3 hours and stopped about 95%. Another log reviewed showed the problem was DHCP related. I connected remotely to the DHCP service and I could see that the scope was getting created and the server options were set, but the reservations were not there. I started reviewing those thinking there might be something there holding it up. By accident, I found that one of the MAC addresses in one of the reservations had a trailing space following the MAC. I removed that reservation and after a short wait, the migration continued and finally finished….6 ½ hours later.

So two things. 1. Watch the log file 2. Get lucky

I received a machine from a customer that would not boot. The machine had been operating flawlessly for several months… then suddenly it would not boot. The typical error was: “The file is possibly corrupt. The file header checksum does not match the computed checksum. Also, once in a while I got an error from BOOTMGR:.  I ran diagnostics on both of the hard drives (mirrored) in the system from a BIOS diagnostic option and they reported NO errors. [more]

I could not boot from the Windows 7 CD. I could not boot from the recovery CD made from Windows 7. I finally made a bootable USB drive with DOS and it worked. I then saw an option in the BIOS to run a memory test… did it and it failed.

The entire problem (after hours of troubleshooting) was a bad memory stick!

A customer who has two terminal servers (TS1 & TS2) that can be accessed using a shared name (TS) was unable to access them from their remote sites. I was able to access TS1 and TS2 from a remote server but not TS. I could also connect using the IP of each server but not the shared IP. What I found was that there was a static ARP entry on the main and backup router for TS. The MAC address on the ARP entry did not match the one on the servers. Both of the servers are virtual machines and this was caused by the ESX update and installation of the updated VMTools on the terminal servers the night before. The MAC addresses on the virtual NICs had changed. The ARP entry was removed and they could connect using the shared name.

One of our customers uses VMware VCB backups integrated with CommVault Simpana. The CommVault job simply calls a pre-backup script to snapshot the VM and copy all the VM files to the VCB proxy, backs up the files from the proxy to the CommVault media server, then a post-backup script commits the snapshot and purges the VM files from the VCB proxy.

Recently, we upgraded this customer from VMware VI3.5 to VMware vSphere v4 Update 2. For most of the VMs that are backed up with VCB, we had no issues at all. The backups ran the weekend following the upgrade with no issues. However, all of the VMs that had been secured with the Windows Security Configuration Wizard would not back up. These VMs are in the DMZ and are locked down very tight because they host externally available web applications. The issue is that each time a backup was initiated from CommVault, the VCB script would return a non-zero error due to a snapshot failure in VMware. VMware’s error was “Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.” This would happen when using VCB scripts, but I could create a snapshot without error from the VI client. [more]

After much research and testing, I determined that the problem was hold-over from the VMTools upgrade. In the new version of VMTools, a new service is installed called VMware Snapshot Provider is installed. This service gets installed when VMTools is upgraded. Its purpose is to help facilitate application consistent snapshots through the VMTools. On the servers that were getting the “quiesced snapshot error”, this service was not present at all, but VMTools had already been updated…very strange. Here is where the Security Configuration Wizard comes in. Part of our lockdown policy is to disable a service called COM+ System Application. This service manages the configuration and tracking of COM+ based components. Apparently, without this service enabled, VMTools upgrade will NOT install the VMware Snapshot Provider service. Without the service, no quiesced snapshots and you get errors when creating snapshots via the VCB integration modules.

So why could I create a snapshot from the Vi client? Well, VMware knows that you are using VCB to create snapshots for the purpose of backup. What good would the backup be if it wasn’t app consistent? The VI client, on the other hand, will first try to create an app consistent snapshot, but if it fails or times out, it will go ahead and create the snapshot “crash consistent” without error. VCB is not as forgiving. If the guest quiesce fails, the snapshot fails…end of story. The solution was to uninstall the VMTools, reboot, temporarily enable and start the COM+ System Application service, install VMTools, then disable the COM+ System Application service. After I did that, backups have been running fine since.

During recent bank audits, our LANguard scans have been flagging some systems by saying “Administrator account with blank password”.  We would typically look at the systems it flagged, determine they were printers, and not worry about it too much.  After some unsuccessful poking around in LANguard, one of our network engineers and I could not figure out what tests it uses to determine that the admin password is blank.  My coworker recommended attempting to connect a shared drive the next time I see that scan result at a bank.  As usual, my coworker's intuition was right.  The next time LANguard came up with that finding, I was able to connect to share drives (\\printer name\ipc$) on multiple printers using the username “Administrator” and a blank password for authentication. [more]

So far, the only reason I have found that printers are using SMB file sharing is to allow access to any flash memory cards that might be in the printer.  At this point, it doesn’t seem like a big security risk, but there may be a time when printers will need to be setup with a telnet management password, an HTTP management password, and a Windows administrator password.

I had a problem with VMware Workstation 7.0.1 this weekend. It is a known problem which causes the vmdk to corrupt. This has happened to me a couple times before, but in those cases I just reverted to a snapshot to fix it. This time it was too much work, so I did some research.

Turns out this has been fixed in 7.1.1 build-282343 and fusion 3.1.1  Everyone who is using Workstation 7 or Fusion 3, you should install the latest copy to avoid this issue. In case you have the problem, the fix can be found at: [more]http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1023856