Wireshark Reviews

I really get excited when I am able to reproduce problems in the lab.

With this specific case, the customer was experiencing errors within their web browsers that looked like either a network or server issue. The specific symptom was that certain images would not display. If you waited a while, and ‘refreshed’ the page, more of it loaded or the entire page loaded properly.

I’m sure you can imagine the chaos this type of intermittent problem causes. The sequence of events unfolds in the following manner; the client reports the webpage issue to the help desk and the help desk tests the webpage with mixed results. In either event, the problem goes to the server group who tests and finds nothing wrong, and then the problem goes to the network group which, in most cases, does not see the problem. Then the political fist fights, finger pointing and witch hunt commence…..

In this case, they even managed to capture some packets during the problem and saw a HTTP “Service Unavailable” message and were having issues interpreting exactly what that would mean. I was there doing some other work when they dumped, uh, I mean asked me if I could help.

They explained that when the problem was occurring, the network management system was not reporting that the server or application was down. I asked how they knew that and they said that they pinged the server, tested for tcp port 80 and lastly retrieved the html page. Wow, I was impressed. I don’t see too many people monitoring from the IP layer up to the Application layer.

I then told them that even though this was an excellent way of monitoring, I wasn’t too surprised that no outages were recorded. If it was an application issue, the pings will still work as well the TCP port check. If all you did was retrieve a single html file, it would not use the same number of connections as actually loading a page and rendering images, etc…

That’s when the lab work came in. I went to my lab and configured IIS to only accept 1 connection, created a simple html file which had a few images on it. After the first try I saw the exact same issue the client experienced as well as the same HTTP message in the analyzer. AWESOME!!!

In the video below you will see how I did it and the results.

Enjoy
http://www.youtube.com/watch?v=-xVqKe53t5s

It always gives me sense of satisfaction when I have a challenge and can leverage some knowledge to figure out.

Today I was in the lab and was powering on two Cisco switches when I noticed that they weren’t labeled with their IP addresses. I’m not sure why I did not label them, but now I have to pay for it.

For those of you who have not been in this situation before I will explain. My switches have a DB9 serial connection and of course good luck finding a computer with a serial port. So now I have to rummage through the box of wires to find the serial to USB adapter. I have had to buy a second one in 2 years since my original does not have a Windows 7 driver, but I digress. After I find the cable, I have to find the installation disk because last week I migrated to a new laptop…. I’m sure you get the picture.

On to plan B. I know the switches have IP addresses since I hard code IP addresses on all of my switches.

Now here’s where a bit of knowledge comes in. I know that when a device powers up and either obtains an IP addresses via DHCP/BOOTP or statically has an IP assigned it will send out a specific ARP called a gratuitous ARP.

Perfect, now all I have to do is make sure the switch port is connected to my subnet, start any protocol analyzer (I chose Wireshark) and power up the switches.

In this video I show you how to find the Gratuitous ARP quickly, create a display filter and lastly, locate the 2 switches’ IP addresses.

Enjoy
http://www.youtube.com/watch?v=EUmHdVeBBNc

NAT Packet Analysis Using Wireshark

One of the most popular questions I get when people get the hang of protocol analysis is the daunting exercise of multitrace analysis. As with anything else the best advice is to start with the basics before tackling anything complicated.

Multitrace analysis is only effective if you truly understand your vendors products, networking and how it relates to the OSI model or packet analysis. I always suggest that you start at layer 1 and work yourself up. The key is to know what fields in the frame or packet changes, or remains the same. Ideally when you figure this out you can use a better capture or display filter

A multitrace capture of a hub, switched, or bridged network is most straight forward since a hub or switch is transparent at layer 1 or 2 and doesn’t change anything in the packet.

When you move up to layer 3 or routing, several things change in the packet such as MAC address, IP TTL and TOS. Of course your mileage will vary, and any device could be configured to muck with more bits in the packet, but I figure I would give you a point of reference.

At layer 4 we get into application gateways, proxy, firewalls and NAT type devices where the following packet fields gets modified; MAC address, IP address, IP TOS, TCP/UDP port numbers, TCP ACK/SEQ values, etc.

Lastly at layer 7, we are dealing with multi-tiered applications and basically everything changes in the packet.

In this video example I do a multitrace analysis of a simple netgear router/NAT/firewall device where I take a trace from the WAN and LAN side to compare. Not to sound like a broken record, but please remember that your devices might behave totally differently and these notes and techniques should only be used as a reference in your environment.

http://www.youtube.com/watch?v=J9FzaFryQIw

Multitrace analysis can be the most interesting, rewarding and unfortunately, most frustrating exercise an analyst will face.

Before we get to the packet analysis, setting up your tools for simultaneous capturing can be a feat in itself.

The time issue is the most critical when using 2 devices since the time is used to calculate the delay, jitter or latency. Some people are fine with syncing both devices to a common ntp server.

Then there’s the “how the #!!$!@#!!” do I physically capture . This is where you have to be familiar with the problem, the network you are working on and what equipment is available to you. If you are lucky enough to be able to change the speed and duplex to 100 half duplex a good old hub fits the bill. Other than the mirror/span command, a tap is also very helpful. Trust me every one of these suggestions comes with their own caveats. You may have to try different tools for different scenarios.

For example, if I am doing a simple pc bootup/login baseline, I am interested in things like total data transferred, which IP’s I am talking to, protocols used, errors, etc. In this case speed and duplex is not important and I can go with a hub. But if I was troubleshooting why something is taking too long, like a backup or replication, changing the speed and duplex would not be a good idea.

If you are lucky enough and can capture from one device, the time accuracy issue goes away and life does get a bit easier. But now you have 2 different captures in the same trace, Yikes!!!! Not to mention that different network interfaces have different latency or behaviors. I remember trying a usb to 10/100 ethernet adapter to capture packets and quickly realized that this adapter added 30 ms to every packet. Again, if I was troubleshooting latency, this won’t do.

Lastly, if you’re fortunate enough, you might even have an application that takes multiple trace files and calculates all sorts of stuff out for you (hmm.. next article?).

In this example I use Wireshark, my laptops WiFi and Ethernet ports to capture my packet traversing a residential home router. I show some tips and tricks along the way and hope this will help you out.

http://www.youtube.com/watch?v=CAS_Kb4VYjo

Documenting a Problem With Wireshark

I remember talking to a group about the ‘superman syndrome’ where the analyst wants to swoop in and save the day. I explained that like most forensic tasks, protocol analysis can be tedius, confusing and downright boring at times. Alright who wants to capture some packets now!?

If you can’t see it, you can’t fix it. That is why I like to use protocol analysis to minimally document the problem that I’m experiencing. Even if the packets don’t show any anomalies, that worth knowing as well, isn’t it? If you do see an anomaly, you might not have the solution but at least you know what it looks like when its broken.

Ideally protocol analysis is most helpful when you have two traces to compare; the good and bad trace. In most realistic scenarios, the client will not have a good trace and just the current bad trace. I’m our classes I review how to make use of what you have.

In this example the customer had a DSL line with an issue and another DSL line what worked fine. The customer mentioned that whenever the DSL circuit ‘acted up’, they simply rebooted the modem. Both DSL circuits went to the same carrier, ordered at the same time, provisioned the same way and even use the same hardware. Perfect, example of something I can compare. I also noticed that these are not just modems, but they route, dhcp, firewall and NAT.

What I found, is that the problem circuit was having issues passing larger frames, while the other had no issues. After the reboot the problem circuit now behaves like the good one. Upon further investigsation I noticed the problem modem had older firmware and suggested they get that firmware updated.

So, even though I couldn’t ‘fix’ the problem, we know exactly what the problem is and what to look for if the problem returns.

http://www.youtube.com/watch?v=OBT5XGOA3EU