Friday, September 2, 2011

The art of network debugging with tcpdump

The art of network debugging with tcpdump

Introduction
Tcpdump is powerful tool for network monitoring and data acquisition it’s working as Swiss knife for network troubleshooting. tcpdump simply powerful application dump traffic on a network this is what tcpdum created for it’s not creates to be analyzer like Wireshark however every application have it’s usage.
Tcpdump give you data without any prejudgment and let burden of analysis is placed directly on the user rather than the application this will sharp your skills and give you deep understand how the protocol work. i notes this happen in almost everything around us and this one of thing make Linux most powerful OS it’s not make user depend in application in the same time it give him tools this Linux beauty

Promiscuous mode
Promiscuous mode is a configuration of a network card that makes the card pass all traffic it receives to the central processing unit rather than just packets addressed to, In Linux root privileges is require to enable promiscuous mode.

How Tcpdump work and howto use ?
Tcpdump must be able to put the interface (typically an Ethernet) into promiscuous mode to read all the network traffic and decode it. tcpdump is use like any other Linux command and it’s installed by default in many Linux distribution

Basic and essential options
The most essential options in tcpdump is -i option which tell tcpdump which interface to put in promiscuous mode and listen to the traffic will coming. another favorite option is -n, which requests that don’t convert addresses (i.e., host addresses, port numbers, etc.) to names this save time and run faster plus in most case we troubleshooting ip not the names. here is the full list for common used options:

-D

Print the list of the network interfaces available on the system and on which tcpdump can capture packets. This can be useful on systems that don’t have a command to list them (e.g. Windows systems)

-i

Listen on interface. If unspecified, tcpdump searches the system interface list for the lowest numbered, configured up interface (excluding loopback).

-vvv

Increase the amount of packet information you get

-c

Exit after receiving count packets.

-n

Don’t convert addresses (i.e., host addresses, port numbers, etc.) to names.

-C

Specifies the size the dump file must reach before a new one with a numeric extension is created. The units of file_size are millions of bytes (1,000,000 bytes, not 1,048,576 bytes).

-F

Use file as input for the filter expression. An additional expression given on the command line is ignored.

-S

Print absolute sequence numbers.

-p

Don’t put the interface into promiscuous mode.

-r

Read packets from file (which was created with the -w option). Standard input is used if file is ‘‘-’’.

-t

Don’t print a timestamp on each dump line.

-X

When parsing and printing, in addition to printing the headers of each packet, print the data of each packet (minus its link level header)
in hex and ASCII. This is very handy for analysing new protocols.

-XX

When parsing and printing, in addition to printing the headers of each packet, print the data of each packet, including its link level
header, in hex and ASCII.

-E

Print the link-level header on each dump line.

-z

Drops privileges (if root) and changes user ID to user and the group ID to the primary group of user.

Basic Example

List of supported NIC that can be used by tcpdump

[root@Machine ~]# tcpdump -D
1.eth0
2.any (Pseudo-device that captures on all interfaces)
3.lo

Capture all traffic path in eth0 then write it to file name tcpdumpfile and set the each file size to be around 3M without trying to resolve IP/Port name

[root@Machine ~]# tcpdump -nnxX -i eth0 -w tcpdumpfile -C 3
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
82 packets captured
166 packets received by filter
0 packets dropped by kernel

Expressions:
Searching in the Network traffic like searching in the ocean, you need to know exactly what you searching for. tcpdump can really collect huge a amount of traffic so you need to guide tcpdump which kind of traffic you intersing in this can be done with expression.
The expression consists of one or more primitives. Primitives usually consist of an id (name or number) preceded by one or more qualifiers.There are three different kinds of qualifier
type: Possible types are host, net , port and portrange. E.g., ‘host foo’, ‘net 128.3’, ‘port 20’, ‘portrange 6000-6008’. If there is no type qualifier, host is assumed.
dir: Possible directions are src, dst, src or dst and src and dst. E.g., ‘src foo’, ‘dst net 128.3’, ‘src or dst port ftp-data’. If there is no dir qualifier, src or dst is assumed.
proto: Possible protos are: ether, fddi, tr, wlan, ip, ip6, arp, rarp, decnet, tcp and udp. E.g., ‘ether src foo’, ‘arp net 128.3’, ‘tcp port 21’, ‘udp portrange 7000-7009’. If there is no proto qualifier, all protocols are assumed.

host ip-address/hostname

True if either the IPv4/v6 source or destination of the packet is ip-address/hostname also can be used with dst host and src host

port number/port-name

True if either the source or destination port of the packet is number/port-name also can be used with dst port and src port

portrange numbe1-number2

True if either the source or destination port of the packet is between number1 and number2 also can be used with dst portrange and src portrange

ether host MAC

True if either the Ethernet source or destination address is MAC also can be used ether src and ether dst

ether broadcast

True if the packet is an Ethernet broadcast packet. and broadcast can be use directly

gateway ip-address

True if the packet used ip-address as a gateway

net network-address

True if either the IPv4/v6 source or destination address of the packet has a network number of network-address. also can be used with src net and dst net

ip broadcast

True if the packet is an IPv4 broadcast packet.

vlan vlan_id

True if the packet is an IEEE 802.1Q VLAN packet. If [vlan_id] is specified, only true if the packet has the specified vlan_id

mpls label_num

True if the packet is an MPLS packet. If [label_num] is specified, only true is the packet has the specified label_num

vpi /vci number

True if the packet is an ATM packet,with a virtual path/channel identifier of number

less/greater length

True if the packet has a length less/greater than or equal to length

More Example

To print all packets arriving from or departing to 10.0.2.2

[root@server ~]# tcpdump -nnvvv -i eth0 host 10.0.2.2
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
07:14:13.748986 IP (tos 0×0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.0.2.15 > 10.0.2.2: ICMP echo request, id 56073, seq 1, length 64
07:14:13.749484 IP (tos 0×0, ttl 255, id 6544, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.0.2.2 > 10.0.2.15: ICMP echo reply, id 56073, seq 1, length 64

capture all traffic dst to http port and coming form my loop back interface

[root@server ~]# tcpdump -vvv -i lo dst port http
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes
07:18:39.937891 IP (tos 0×0, ttl 55, id 52591, offset 0, flags [none], proto: TCP (6), length: 44) server.36901 > 10.0.2.15.http: S, cksum 0xc515 (correct), 1306286511:1306286511(0) win 4096
07:18:40.937973 IP (tos 0×0, ttl 39, id 62265, offset 0, flags [none], proto: TCP (6), length: 44) server.36902 > 10.0.2.15.http: S, cksum 0xc514 (correct), 1306352046:1306352046(0) win 4096

capture all Address Resolution Protocol (ARP) packets

[root@server ~]# tcpdump -nnevvv -c 3 arp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
07:23:47.914195 08:00:27:ed:89:bd > ff:ff:ff:ff:ff:ff, ethertype ARP (0×0806), length 42: arp who-has 10.0.2.3 tell 10.0.2.15
07:23:47.914514 52:54:00:12:35:03 > 08:00:27:ed:89:bd, ethertype ARP (0×0806), length 60: arp reply 10.0.2.3 is-at 52:54:00:12:35:03
07:23:54.698897 08:00:27:ed:89:bd > ff:ff:ff:ff:ff:ff, ethertype ARP (0×0806), length 42: arp who-has 10.0.2.2 tell 10.0.2.15
3 packets captured
7 packets received by filter
0 packets dropped by kernel

Try to capture icmp traffic AND to or from the host 10.0.2.2

[root@server ~]# tcpdump -nn icmp host 10.0.2.2
tcpdump: ‘icmp’ modifier applied to host
[root@server ~]#

In last example we was trying to use tow valid expressions with tcpdump but How come tcpdump know and differentiate this tow expressions ?? answer: with combine them

Expressions combination
Tcpdump understanding boolean operators (AND, NOT, OR). grouping this boolean operators between the expressions can create any rule limited only with your imagination.
Boolean operators define the relationships between expressions or groups of expressions. The relationship can be True or False and depend in this tcpdump will take the resolution to capture the packet or not

  • AND (and == && ) Give True ONLY AND ONLY IF both expression True else give False
  • NOT (not == !) Reverse the resolution if it was Ture it will be False and if it was False it will be True
  • OR (or == ||) Give False ONLY AND ONLY IF both expression False else give True

Advanced Example

All IP packets between 10.0.2.15 and any host except 10.0.2.2

[root@server ~]# tcpdump -c 10 ip host 10.0.2.15 and not 10.0.2.2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
08:08:14.352939 IP 10.0.2.15.ssh > 192.168.99.166.38395: P 2998871938:2998872054(116) ack 155036726 win 35420
08:08:14.365762 IP 192.168.99.166.38395 > 10.0.2.15.ssh: . ack 116 win 8760
08:08:14.367732 IP 10.0.2.15.ssh > 192.168.99.166.38395: P 116:232(116) ack 1 win 35420
08:08:14.368449 IP 192.168.99.166.38395 > 10.0.2.15.ssh: . ack 232 win 8760
08:08:14.356135 IP 10.0.2.15.35379 > 10.0.2.3.domain: 19282+ PTR? 166.99.168.192.in-addr.arpa. (45)
08:08:14.358604 IP 10.0.2.3.domain > 10.0.2.15.35379: 19282 NXDomain* 0/1/0 (130)
08:08:14.360064 IP 10.0.2.15.56712 > 10.0.2.3.domain: 44586+ PTR? 15.2.0.10.in-addr.arpa. (40)
08:08:14.362235 IP 10.0.2.3.domain > 10.0.2.15.56712: 44586 NXDomain 0/1/0 (117)
08:08:14.365138 IP 10.0.2.15.ssh > 192.168.99.166.38395: P 232:396(164) ack 1 win 35420
08:08:14.365603 IP 192.168.99.166.38395 > 10.0.2.15.ssh: . ack 396 win 8760
10 packets captured
39 packets received by filter
0 packets dropped by kernel

Capture packets related with Mail , Web and FTP service

[root@server ~]# tcpdump -c 3 -i eth0 port smtp or http or ftp-data or ftp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
08:13:45.441400 IP 10.0.2.15.47362 > 10.0.2.2.smtp: S 2545888906:2545888906(0) win 2048
08:13:45.452871 IP 10.0.2.15.47362 > 10.0.2.2.smtp: R 2545888907:2545888907(0) win 0
08:13:50.663261 IP 10.0.2.15.41938 > 10.0.2.2.ftp: S 846604284:846604284(0) win 1024
3 packets captured
6 packets received by filter
0 packets dropped by kernel

Capture only traffic from Internet

[root@server ~]# tcpdump -c 5 -nn -i eth0 ’src net not 192.168.0.0/16 and not 10.0.0.0/8′
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
08:24:51.413583 IP 87.248.113.14.80 > 10.0.2.15.50484: S 1979840001:1979840001(0) ack 18214165 win 8192
08:24:51.418167 IP 87.248.113.14.80 > 10.0.2.15.50484: . ack 121 win 8760
08:24:51.453019 IP 87.248.113.14.80 > 10.0.2.15.50484: P 1:1025(1024) ack 121 win 8760
08:24:51.463017 IP 87.248.113.14.80 > 10.0.2.15.50484: P 1025:2049(1024) ack 121 win 8760
08:24:51.468393 IP 87.248.113.14.80 > 10.0.2.15.50484: P 2049:3073(1024) ack 121 win 8760
5 packets captured
10 packets received by filter
0 packets dropped by kernel

Capture ftp and http

[root@server ~]# tcpdump -nn -i eth0 src host 10.0.2.15 and ‘dst port 21 or 80′
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
08:28:14.867601 IP 10.0.2.15.36888 > 72.232.194.162.80: . ack 3094288542 win 3072
08:28:15.048748 IP 10.0.2.15.36867 > 72.232.194.162.21: S 3753909241:3753909241(0) win 3072
08:28:15.149659 IP 10.0.2.15.36868 > 72.232.194.162.21: S 3753974776:3753974776(0) win 3072
08:28:17.362931 IP 10.0.2.15.60855 > 72.232.194.162.80: . ack 1168658718 win 1024
08:28:17.478546 IP 10.0.2.15.60831 > 72.232.194.162.80: S 2859656818:2859656818(0) win 2048
08:28:17.554347 IP 10.0.2.15.60831 > 72.232.194.162.80: R 2859656819:2859656819(0) win 0

6 packets captured
12 packets received by filter
0 packets dropped by kernel

Problems and hints
No output

Check to make sure you’re specifying the correct network interface with the -i option, which I suggest you always use explicitly. If you’re having DNS problems, TCPdump might hang trying to lookup DNS names for IP addresses, try the -f or -n options to disable this feature. If you still see nothing, check the kernel interface – TCPdump might be mis-configured for your system.

Dropped packets
At the end of its run, TCPdump will inform you if any packets were dropped in the kernel. If this becomes a problem, it’s likely that your host can’t keep up with the network traffic and decode it at the same time. Try using TCPdump’s -w option to bypass the decoding and write the raw packets to a file, then come back later and decode the file with the -r switch. You can also try using -s to reduce the capture snapshot size.


Messages that end like [|rip] and [|domain]

Messages ending with [|proto] indicate that the packet couldn’t be completely decoded because the capture snapshot size (the so-called “snarf
length”) was too small. Increase it with the -s switch.