I tend to use tcpdump when working on remote servers with multiple services running.
Thus it’s important to specify exactly which hosts and ports I want to see, or end up buried in output.
Here’s a canonical example of looking at output from a REST service on a remote host that’s listening on port 8080:
# CentOS 6/7 sudo yum -y install tcpdump sudo tcpdump -nn -vv -i eth0 -s0 -A host X.X.X.X and port 8080
Options decoder: -nn is no name or port lookups, -i is network interface, -s0 is unlimited snapshot length, -A is ASCII output.
# Mac OS X brew install tcpdump sudo tcpdump -nn -vv -i en0 -s0 -A host X.X.X.X and port 8080
A more advanced example is to only capture on HTTP data packets on port 80. Avoid capturing the TCP session setup (SYN / FIN / ACK):
sudo tcpdump 'tcp port 8080 and (((ip[2:2] - ((ip&0xf)<<2)) - ((tcp&0xf0)>>2)) != 0)'
- use tcpdump, wireshark, telnet, netcat and nmap periodically so that you’re familiar with your environment, and proficient when you need them. Ensure you have permission for those hosts!
- if you’re using ssh, ensure you’re not dumping port 22 to avoid excessive display output
- seeing zero-length responses is worth paying attention to, but some listeners just work that way
- seeing duplicate responses is also worth paying even closer attention to as that usually indicates an unexpected problem (unless you’re dumping on localhost)
- seeing unexpected responses from port 3128 usually means there is a proxy
- if you nuke your terminal with binary data, try the clear or reset commands, or redirect to a file
- on a physical server you can see actual packets with MTU size, but in virtualized/cloud environments they will likely be merged before you can see them.
- when things just don’t seem to make sense, the ip route, tcproute and telnet commands take you down to bedrock.