Blog

We recently deployed VRF groups on our core switches and there are a few key changes to troubleshooting/configuration.
Ping:
Standard way: ping 192.168.1.1
w/ VRFs: ping vrf NAME 192.168.1.1 (where NAME is the name of the vrf group)

Traceroute
Standard way: traceroute 192.168.1.1
w/ VRFs: traceroute 192.168.1.1 /vrf NAME

If you are trying to make a connection to a destination that resides in a VRF from the core switches you will most likely have to include some sort of VRF tagging.  This even applies to services like RADIUS, NTP, DHCP Relay, and TACACS.


 

Upon receiving my new Lenovo ThinkPad laptop, I set up fingerprint authorization through the Lenovo software.  After ensuring all my fingerprints were scanned properly, I rebooted the machine.   I tried to the use my fingerprint to login and the light flashed green.  Unfortunately, the machine wouldn’t proceed any farther in the process.

It appears you have to go into Windows 7 itself and enable ‘Domain Login’ under the Windows Biometric section in order to actually allow domain authorization.  Otherwise, the software will just let you access local accounts.


 

I was building a new Server 2008 machine for a customer a few weeks ago. After installing the OS, I decided to activate the OS. I was told  I would have to activate over the phone because Internet activations do not work from the office. Before I could activate though, I needed to change the product key to the customer’s key. When I typed in the new key, I received an error stating “Invalid product key”. I decided to call Microsoft. They verified the key was correct several times in different departments. I was told it might be a key/media mismatch. So I reinstalled and immediately tried to change the product key again. It again gave me the same error. I was finally told that keys generated after SP2 was released would not be recognized as valid until the system was running SP2. I installed SP2 and the key activated without a problem.


 

We ran into a problem recently where users on a Windows 2008 R2 terminal server would lose their connection to SMB shares.  Fully-qualified domain names do not get disconnected.

There is a Hotfix from available from Microsoft that fixes this problem:  http://support.microsoft.com/kb/2194664.  The Hotfix is integrated into into Windows 2008 R2 SP1 and the next Windows 7 service pack.


 

I received some scanned PDF documents from a customer who had scanned them at high resolution. When I tried to use Acrobat to OCR the files, I got an error message stating the pages were larger than the 45" maximum. I was able to OCR the files once I printed them to individual PDFs (which put them into 8.5X11 pages).


 

I had to troubleshoot a point to point T1 circuit that was down.  The circuit is joined between two different carriers.  One side of the circuit was Verizon, and the other side was AT&T.  We weren’t sure who to call originally, so both of the carriers were called to troubleshoot the circuit.

While tests were being done, I was able to go onsite and tell right off the smartjack had no lights on it at all.  AT&T local technician was eventually dispatched to move the connection over to a spare smartjack onsite.  After everything was moved, we rebooted the routers at both ends, but the circuit was still dead. [more]

The technician finally decided to try sending a loopback clear signal down the entire line stating that “It appeared that there was still a software loopback somewhere on the line that wasn’t removed after testing.”  After he sent the signal down the line, we rebooted the routers and the circuit came back up.

This is something handy we can ask the onsite technicians to look for in the case where everything looks like it should be working but isn’t.


 

I had come across a user that was unable to print to their local printer.  The printer was situated underneath the desk in a dark corner so it wasn’t very visible.  I decided to check the USB cable first to make sure it was connected.  Unplugging it and plugging it back in showed the USB connected message in the system tray.

As I pulled the printer out a little further, I saw that it had a LCD panel that said “Select your language”.  I hit OK to select English, then selected the Country.  After the selections were confirmed, all of the jobs in the print queue started printing.

The user had replaced the toner cartridge recently, so that could have been when the prompt started.  If you're working on a printer problem in the future it's probably worth asking the user if it has an LCD screen and to check that it is not waiting on some kind of response.


 

Block level vmdk backups have limitations that will GET YOU.  Backup Exec and Veeam both have the ability to backup the vmdk files in a VMware environment and still retain enough information in the backup set to do individual file level restores.  However, both products will ONLY work if you have vmdk disks partitioned using the MBR (Master Boot Record) type tables and NOT the more modern GPT (Guid Partition Table) structure.


 

This is sort of a follow up to a post about the Firefox addon Certificate Patrol.  The addon Perspectives also helps watch out for certificate related problems.  When you go to a secure web site, Perspectives can (with a click or automatically) check with several “notaries” scattered around the world and tell you whether they are getting the same certificate from that site as you are.  Read http://perspectives-project.org/ for more details.  Here is a link to the Firefox addon: https://addons.mozilla.org/en-US/firefox/addon/perspectives.  There is also an Alpha, very experimental Chrome addon https://chrome.google.com/webstore/detail/lnppfgdnjafeikakadfopejdpglpiahn.

This project is out of Carnegie Mellon University  The notary server is open source, so anyone can run their own servers.  By default, the plugin uses several servers that seem to be run by the Massachusetts Institute of Technology.


 

A week or so ago, I had a task to do some work on the VMware cluster for one of our customers. It was late (~11:00 PM) when I finished the task, but I decided to go ahead and try to do a P2V conversion on one of the physical servers that we had targeted for virtualization a long time ago. It’s that server that sits in the corner and just keeps running…and nobody knows a thing about it. Everybody is just happy it keeps running because it’s on 10 year old hardware. It’s the poster child for virtualization. I was 90% sure that the conversion wouldn’t even kick off because the server has such an odd-ball configuration, but to my surprise it worked fine. P2V conversion was done in about 90 minutes (~12:30 AM). When I do these conversions, the first thing I do is start the VM up without the NICs connected just to see if the new VM will boot.

I knew it was too good to be true...BSOD with immediate reboot. Error message “Inaccessible boot device”. If it were any other error message, I probably would have just hung it up for the night, but I have fixed conversions that give this error a number of times. Most of the time it’s due to the existence of a recovery partition on the physical system and the boot.ini has the wrong partition numbers in the boot parameters…easy fix. I attached the vmdk to a helper VM to access the files. The boot.ini file looked fine and all the files necessary for boot made it over. So, I took a closer look at the conversion logs. I didn’t notice it while the conversion was running, but right at the end of the job, a warning was logged…”virtual machine reconfiguration failed”. So what is reconfiguration… never noticed that before. [more]

The “reconfiguration” keyword as well as the stop code led me to the following VMware article: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1005208. Basically, the article provides instructions on how to inject the VMware SCSI controller driver into the physical machine pre-conversion. The reconfiguration task that failed on the conversion indicated that VMware was unable to inject the driver and registry keys into the VM post conversion…ah yes, the odd-ball configuration. The server was built with only a 100 MB c:\ partition. The c:\ partition only contains the windows boot files. The boot.ini references the \windows folder on partition(2), which is the d:\ drive. Why was the server built that way??...nobody knows. So, VMware tried to inject the drivers to fix the boot problem, but couldn’t because it doesn’t read the boot.ini to find out where the \windows folder it. It just assumes it will be at c:\windows and that c:\ will be partition(1). So, there is the cause…so how do I fix it. By this time, its 1:30 AM and I really didn’t want to wait 90 minutes for the conversion to run again. The article explains how to fix this problem. I just had to figure out how to make those changes post-conversion. Specifically, we want to follow the Windows 2003 (lsilogic controller) instructions. So here we go.

  1.  Copying the symmpi.sys file to the %systemroot%\system32\drivers\ folder is pretty easy. Just mount the vmdk to a helper VM and copy the file from another windows 2003 virtual machine (on vmware of course) to the target vmdk.
  2. The registry edits is where it gets a little tricky. The goal in this step is to manually load the registry hives for the VM that won’t boot and inject the .reg file changes into the registry.
    • Follow the article to export the registry sub-hives to .reg files from another windows 2003 virtual machine.
    • From your helper VM where your unbootable vmdk is mounted, using regedit.exe, highlight HKEY_LOCAL_MACHINE and choose File -> Load Hive.
    • Browse to %systemroot%\system32\config inside your unbootable vmdk. For this instance, it was on partition(2) or d:\windows\system32\config. This directory contains all the physical files that provide the registry tree. Choose the file named “system” to load.
    • You will be prompted for a name to load it as…just type the hostname of the unbootable VM (server1). This will load the system hive for the unbootable VM under HKEY_LOCAL_MACHINE in regedit.
    • If you browse down into that mounted registry hive, you will see there is no CurrentControlSet. Your registry keys that were exported need to be put back in CurrentControlSet. Your safest bet is to restore the registry hives into ALL control sets so that you can be sure you get it in the right spot.
    • You have to make sure that the registry changes get imported into the mounted registry hive and not your local machine registry hive. Open the .reg files and do a search and replace for the string “HKEY_LOCAL_MACHINE\system\currentcontrolset” and replace it with HKEY_LOCAL_MACHINE\server1\system\controlset001”. As mentioned in the step above, you may have multiple controlsets (controlset001, controlset002, etc.) which means you will have to edit the .reg file multiple times and re-merge it to get the registry settings into all control sets.
    • Right-click on each of the .reg files and merge them. Double-check to make sure all the entries got added.
    • Highlight HKEY_LOCAL_MACHINE\server1, and choose File à Unload Hive to unmount system registry file.
  3. Remove the vmdk from the helper virtual machine. Make sure to just disconnect it and not delete it from disk.
  4. Start your previously unbootable virtual machine. Should work find now. (3:00 AM)

So, this process is way easier if you just re-do the conversion. I ended up having to do this conversion again due to other circumstances and did the injection pre-conversion the second time and it works as well. However, I have had conversions that took 4-5 hours due to the amount of data involved. This process is obviously justified in those situations.