Blog: Cisco

During troubleshooting it is often necessary to see what traffic is being passed between two networks or two hosts. The ASA software now features a built-in packet capture tool.

Below are the steps you need to take:

For the sake of this tutorial, let’s assume that we are troubleshooting traffic between a host with the address of 192.168.1.1 and a host with the address of 10.10.10.1.

Step 1. Define the traffic that you are interested in seeing via an ACL named “cap”: [more]

ASA(config)#access-list cap extended permit ip host 192.168.1.1 host 10.10.10.1
ASA(config)#access-list cap extended permit ip host 10.10.10.1 host 192.168.1.1
ASA(config)#access-list cap extended permit icmp host 192.168.1.1 host 10.10.10.1
ASA(config)#access-list cap extended permit icmp host 10.10.10.1 host 192.168.1.1

Step 2. Create and start the packet capture process named “capin”:

ASA(config)#capture capin access-list cap

Step 3. Generate some traffic between the two hosts.

Since our ACL in this case is set to detect all IP and ICMP traffic between the host we can just start a simple ping betweent the hosts.

From the host 192.168.1.1:
ping 10.10.10.1
From the host 10.10.10.1
ping 192.168.1.1

Step 4. Analyze the packet capture.

ASA#show capture capin
*This will output all of the traffic that it captured.

Step 5. Turn off the packet capture and remove the ACL:

ASA(config)#no capture capin
ASA(config)#clear configure access-list cap

Miscellaneous notes/commands:

You can clear the capture log by using this command:
ASA#clear capture capin

You can also use the pipe functionality when viewing the capture output:
ASA#show capture capin | inc 192.168.1.1

This can also be done via the ASDM, but what fun is that?


 

After converting a site to a new MPLS provider I began to experience about 20% packet loss to that site.  There were a lot of things that changed during the migration:

  • Added GRE Tunnels
  • Implemented EIGRP to handle routing all of the LAN subnets
  • Restricted BGP to only handle the WAN, or MPLS, interfaces

These are the troubleshooting steps I took to narrow down the problem:

  1. Ping from the tunnel interface at the main site to the tunnel interface at the branch site.  0% Packet Loss
  2. Ping from the LAN port on router at the main site to the tunnel interface at the branch site. 0% Packet Loss
  3. Ping from the LAN port on router at the main site to the LAN port at the branch site. 0% Packet Loss
  4. Ping from a client at the main site to the tunnel interface at the branch site. 0% Packet Loss
  5. Ping from a client at the main site to a client at the branch site. ~20% Packet Loss
  6. Ping from the LAN port on router at the main site to a client at the branch site. ~20% Packet Loss
  7. Ping from the tunnel interface at the main site to a client at the branch site. ~20% Packet Loss

This process seemingly narrowed it down to the problem originating at the branch site.  I checked for negotiation errors in the logs of the switch and the routers.  BGP appeared to be working fine because the peer was up and I was receiving all the routes that I expected.  The ping loss seemed to be very random.  I then decided to enable debugging on the router and start a continuous ping from a client at the main site to a client at the branch site.  I quickly noticed that every time I saw packet loss, I also so a BGP error message being logged.  There were a few different error messages that were being populated and each caused different amounts of ping loss. 

Apparently, the ping loss wasn’t as random as I thought!  After speaking with a coworker about the BGP turn up he was currently doing with another customer, he suggested that I needed to add a static route to the branch router for the BGP peer.  Everything began working!  So, to make a long story short, it is best to have a specific static route added for a BGP peer if that peer isn’t directly connected. Even if that static route has the same next-hop as the default route.


 

When changing some settings in Internet Explorer recently, I stumbled across the “Use SSL 3.0” and “Use TLS 1.2” settings under the Advanced tab of Internet Options.  For a long time, I have been running without SSL 2.0, TLS 1.0, and TLS 1.1 enabled, but I wondered if SSL 3.0 is even necessary anymore (TLS superseded SSL 3.0 in 1999).  So I unchecked the “Use SSL 3.0” check box.  I did the same in my Firefox settings.  I ran that way for at least a couple of weeks without any noticeable issues.  Then last week I was onsite at a bank and tried to use the Cisco AnyConnect SSL VPN.  It did not connect, so I tried it that night from the hotel.  It still did not connect.  The Cisco IPSEC VPN client worked perfectly.  After a couple of days of the AnyConnect client not working, I was about to send an email to one of our network engineers asking if anything had changed when I remembered the “Use SSL 3.0” setting.  After re-enabling SSL 3.0 in both IE and Firefox, the AnyConnect client worked.  Aside from the SSL 3.0 setting breaking AnyConnect, the more general GOTCHA is that the AnyConnect client uses at least some of the web browser settings when establishing its connection, so I now know to include them when troubleshooting the VPN.


 

Cisco has a built in tool that allows you to test AAA server connectivity.  It is included in both the CLI and the ASDM.  This tool is useful when you are setting up a AAA server because you don't have to login and out of the device in order to test the connectivity.

The CLI command is:
test aaa-server authentication SERVERNAME [more]

After entering this command you will be prompted to enter the server IP address, and a username and password.

The packet-tracer command is another tool that comes in handy when testing access-lists and VPN configurations.  It simulates a packet that transverses through the ASA and prints out the each step it takes and points out where it is failing.

To trace a HTTP packet from the inside address of 10.1.1.1 to outside address of 4.2.2.2 the command would be:
packet-tracer input inside tcp 10.1.1.1 www 4.2.2.2 www detailed


 

The switch ports on the Cisco routers don’t appear to be as robust as a standard switch.  All clients at a clients motor bank went down the other morning.  After travelling on site, it turned out the problem was a bad cable end on one client.  All seven devices connected to the same 8-port switch on the branch router, and this one bad cable took down all seven devices.  I could move the faulty device from the router’s switch port to an external switch, and everything would work.  I went ahead and replaced the bad cable end, and reconnected all devices to the router’s switchports.


 

When working with Cisco 800 model routers (and probably any Cisco Integrated Services Router) you might run into an issue that the VLAN which is assigned to the LAN ethernet ports is in an “up/down” state.  This is because an SVI must meet all of the following conditions to transition to the full "up/up" state:

  • The VLAN must exist and be active in the VLAN database.
  • At least one switched port in the VLAN (access or trunk) must be up.
  • That port must be in the STP forwarding state.

Sometimes it is necessary to have that VLAN interface up even if there are no devices or ports using that VLAN.  [more]The most recent case that I experienced this need was when I was trying to transfer IOS images remotely across a VPN connection.  Because the transfer was traversing across the VPN I had to source the file transfer from the internal VLAN interface.   There weren’t any PC’s connected to the router so the VLAN interface was in an “up/down” state.  To resolve this issue, I could have either connected a PC or a loopback into the router or simply forced the VLAN into an “up/up” state.  Issuing the “no autostate” command on the VLAN interface will bring the interface up.  Basically, the command just tells the VLAN interface to ignore the above mentioned prerequisites.  Note: This command is only available in certain IOS images.


 

During an IT security audit this week, I had the chance to test the Cisco AnyConnect VPN client from a VPN-hostile network.  The bank has implemented multiple layers of Internet filtering including web content filtering, outbound port filtering, and inbound IP address/port filtering.  As expected, the old Cisco IPSEC VPN client could not connect.  The AnyConnect client, however, connected on the first try without having to ask the bank to modify any of their access controls.

FYI, the AnyConnect client dynamically determines if it can also use a Datagram Transport Layer Security (DTLS) tunnel via UDP in addition to the SSL tunnel.  If the DTLS tunnel connects and is “healthy” the client will use both the SSL and DTLS tunnels to transmit data.  If the DTLS tunnel cannot connect or is unreliable, the client will dynamically switch to using only the SSL tunnel.  You can read more about it on Cisco’s AnyConnect FAQ page.


 

If you use the setup wizard for Cisco ASA appliances to allow SSH access it doesn’t auto-generate a key.  It will create the access-rules, but you still won’t be able to SSH to the firewall until the key is generated.  The quickest way to generate the key is via the command:

generate crypto key rsa modulus-size” [more]

Note: The modulus-size can be 512, 768, 1024, or 2048.  The value of 1024 is recommended.


 

We use MRTG on our internal network to help monitor bandwidth for several network connections.  After setting up monitoring for several VPN’s I noticed a couple of the busier VPN’s would “flat line” during the day but look normal after hours.  When MRTG can’t access the device to get data, it continues to plot the previous value which creates this flat line effect.  Therefore I initially thought there was a problem with those VPN’s, but the VPN’s were working fine.  After digging further I noticed the MRTG cfgmaker utility put  “MaxBytes[xxxx]: 1125” for the graph of the VPN interface.  This tells MRTG how to scale the graph, but it also discards data above that threshold.  So during the day the bandwidth was above the threshold and displaying the flat line graph because it was displaying the last known good data value.  I increased the value on the MaxBytes statement to fix the problem.


 

I was trying to use Cisco’s Adaptive Security Device Manager (ASDM) to connect to our ASA in the office.  I was getting an authentication error but I knew my credentials were correct and it was working for another engineer.  The Java console contained the error “java.io.IOException: Authentication failure”.  I found several references to proxy issues related to this error, so I went to the Network Settings section of the Java app in the control panel and manually specified our proxy server (including the local bypass addresses) and it started working.  The proxy setting was set to “use browser settings” but obviously this wasn’t working.