root@server:~# tcpdump -c100 -A -l -tttt -s0 -qpnni any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
2013-04-28 13:57:16.725851 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`....(.@*..Hx...........*..P@ ...........}.P/.Hp......8@. .........
..b.........
2013-04-28 13:57:16.728057 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`....(.7*..P@ ..........*..Hx............P.}}.[./.Hq..7..{.........
^.....b.....
2013-04-28 13:57:16.728093 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@ ...........}.P/.Hq}.[............
..b.^...
2013-04-28 13:57:16.728445 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 166
`......@*..Hx...........*..P@ ...........}.P/.Hq}.[............
..b.^...GET / HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: www.google.com
Accept: */*
2013-04-28 13:57:16.729989 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`.... .7*..P@ ..........*..Hx............P.}}.[./.I.....^E.....
^.....b.
2013-04-28 13:57:16.742677 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 988
`......7*..P@ ..........*..Hx............P.}}.[./.I......h.....
^.....b.HTTP/1.1 302 Found
Location: http://www.google.co.uk/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=0241f520b15aae6e:FF=0:TM=1367157332:LM=1367157332:S=5lVTK-ZqTfrki7HN; expires=Tue, 28-Apr-2015 13:55:32 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=OyloOHElfW8AKOcsRJ4DeQnfMhqfmnqJgcpYXlsSrN2ouREV9KHjS9boJZTfBoFZvYtVg0ugcBa2lJQKX-WrQ_uMxoIvPg-4JehPfFEdyGl_oh0RS37x_V6a_ozMElzJ; expires=Mon, 28-Oct-2013 13:55:32 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Sun, 28 Apr 2013 13:55:32 GMT
Server: gws
Content-Length: 221
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/">here</A>.
</BODY></HTML>
2013-04-28 13:57:16.742700 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@ ...........}.P/.I.}._w...........
..b.^...
2013-04-28 13:57:16.743753 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@ ...........}.P/.I.}._w...........
..b.^...
2013-04-28 13:57:16.745725 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`.... .7*..P@ ..........*..Hx............P.}}._w/.I.....ZS.....
^.....b.
2013-04-28 13:57:16.745743 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@ ...........}.P/.I.}._x...........
..b.^...
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Showing posts with label netadmin. Show all posts
Showing posts with label netadmin. Show all posts
Sunday, April 28, 2013
How to capture HTTP traffic using tcpdump
Everyone knows tcpdump but not everyone knows how to use it in efficient way. Below is a nice trick how to capture the HTTP GET request and the server response in plain text.
Sunday, April 10, 2011
How to filter and dump in clear text http requests and responses from the network dump using the cli program tshark
There are many programs that allow us to save network traffic into a file for later analyze. Once you have the file(s) we could like to quickly investigate the data inside and verify what we sent and what we revived.
If we are interested, let's say only in the HTTP traffic than the command line 'tshark' has only limited capabilities in a way to present us the data for review and investigation.
We can still play with the various options: '-Tfields' and multiple '-e' but still don't get the complete headers output.
The complete list that can be used with '-e' can be found here .
Alternatively we can experiment with the '-Tpdml' that will create alike XML file. But even with all this flexibility we still can't print a custom name header like in this curl request bellow:
To solve this little problem we have created a small program written in python. It takes the output from 'tshark -S -V' and parses it to present the data in a way we want.
Example how to use it:
An example output from the 'parse.py' when running on the command line:
aaa
If we are interested, let's say only in the HTTP traffic than the command line 'tshark' has only limited capabilities in a way to present us the data for review and investigation.
We can still play with the various options: '-Tfields' and multiple '-e' but still don't get the complete headers output.
The complete list that can be used with '-e' can be found here .
Alternatively we can experiment with the '-Tpdml' that will create alike XML file. But even with all this flexibility we still can't print a custom name header like in this curl request bellow:
curl -H "Rado: my_value" -v -o /tmp/page.html http://rtomaszewski.blogspot.com/2011/04/tshark-in-network-troubleshooting.html
To solve this little problem we have created a small program written in python. It takes the output from 'tshark -S -V' and parses it to present the data in a way we want.
Example how to use it:
# tshark -r /tmp/net.pcap -R http -V | parse.py -d # tshark -r /tmp/net.pcap -R http -V | parse.py # tshark -nn -s0 -i any -w /tmp/net.pcap -f tcp -R http -l -S -V | parse.py # tshark -nn -s0 -i any -w /tmp/net.pcap -f tcp -R http -l -S -V | parse.py -d
An example output from the 'parse.py' when running on the command line:
$ curl -H "Rado: my_value" -v -o /tmp/page.html http://rtomaszewski.blogspot.com/2011/04/tshark-in-network-troubleshooting.html
* About to connect() to rtomaszewski.blogspot.com port 80 (#0)
* Trying 209.85.229.132... connected
* Connected to rtomaszewski.blogspot.com (209.85.229.132) port 80 (#0)
> GET /2011/04/tshark-in-network-troubleshooting.html HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: rtomaszewski.blogspot.com
> Accept: */*
> Rado: my_value
>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
< ETag: "b6d5837e-31bb-4473-95af-da3c1d466295"
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Server: GSE
< Age: 1202
< Date: Sun, 10 Apr 2011 19:14:00 GMT
< Expires: Sun, 10 Apr 2011 19:14:00 GMT
< Last-Modified: Sun, 10 Apr 2011 19:03:05 GMT
< Cache-Control: public, must-revalidate, proxy-revalidate, max-age=0
< Transfer-Encoding: chunked
<
{ [data not shown]
100 49993 0 49993 0 0 28925 0 --:--:-- 0:00:01 --:--:-- 57529* Connection #0 to host rtomaszewski.blogspot.com left intact
* Closing connection #0
# tshark -nn -s0 -i wlan0 -w /tmp/net.pcap -R http -f tcp -S -V | ./parse.py
Running as user "root" and group "root". This could be dangerous.
Capturing on wlan0
Internet Protocol, Src: 192.168.43.111 (192.168.43.111), Dst: 209.85.229.132 (209.85.229.132)
Transmission Control Protocol, Src Port: 38391 (38391), Dst Port: 80 (80), Seq: 1, Ack: 1, Len: 226
[Stream index: 1]
GET /2011/04/tshark-in-network-troubleshooting.html HTTP/1.1\r\n
User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15\r\n
Host: rtomaszewski.blogspot.com\r\n
Accept: */*\r\n
Rado: my_value\r\n
Internet Protocol, Src: 209.85.229.132 (209.85.229.132), Dst: 192.168.43.111 (192.168.43.111)
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 38391 (38391), Seq: 49070, Ack: 227, Len: 1375
[Stream index: 1]
HTTP/1.1 200 OK\r\n
Content-Type: text/html; charset=UTF-8\r\n
ETag: "b6d5837e-31bb-4473-95af-da3c1d466295"\r\n
X-Content-Type-Options: nosniff\r\n
X-XSS-Protection: 1; mode=block\r\n
Server: GSE\r\n
Age: 1202\r\n
Date: Sun, 10 Apr 2011 19:14:00 GMT\r\n
Expires: Sun, 10 Apr 2011 19:14:00 GMT\r\n
Last-Modified: Sun, 10 Apr 2011 19:03:05 GMT\r\n
Cache-Control: public, must-revalidate, proxy-revalidate, max-age=0\r\n
Transfer-Encoding: chunked\r\n
2 packets captured
The program can be downloaded form here: parse.py Additionally the source code of this short tool can be seen here: #!/usr/bin/python
#
## tested on python 2.6.5
#
# author : radoslaw tomaszewski
import sys
import re
import inspect
class ParseTsharkOut:
no=0
debugYes=0
ipRe=None
tcpRe=None
protRe=None
ipInfo=[]
tcpInfo=[]
protInfo=[]
auxStart=0
tcpAuxStart=0
protAux=1
def __init__(self):
self.ipRe="Internet Protocol,(.*)$"
self.tcpRe=["Transmission Control Protocol,(.*)$", " (\[Stream index:.*)$"]
self.protRe=["Hypertext Transfer Protocol", " ([^\[ ].*)$", "^$| *(\\\\r|\\\\n)"]
def debug(self, s):
if self.debugYes :
parent=inspect.stack()[1][3]
#parent=inspect.stack()
print("debug:[" + str(parent) + "] " + s.rstrip())
def usage(self):
print("todo")
def ipParse(self,s):
self.debug(s)
tmp=re.match(self.ipRe, s)
# tmp=re.match("..", s)
if tmp is None :
return 0
else:
self.ipInfo.append(tmp.group(0))
return 1
def tcpParse(self,s):
self.debug(s)
ret=0
tmp=re.match(self.tcpRe[self.tcpAuxStart], s)
if tmp is None :
return 0
else:
self.tcpInfo.append(tmp.group(0))
ret=self.tcpAuxStart
self.tcpAuxStart=(self.tcpAuxStart + 1 ) % 2
return ret
def protParse(self,s):
self.debug(s)
if ( self.protAux ) :
if re.match(self.protRe[0], s):
self.protAux=0
return 0
else :
if re.match(self.protRe[2], s):
self.protAux=1
self.show()
return 1
tmp=re.match(self.protRe[1], s)
if tmp is None :
return 0
else:
self.protInfo.append(tmp.group(0))
return 0
def parse(self, s):
funcs=[self.ipParse, self.tcpParse, self.protParse]
if funcs[self.auxStart](s):
self.auxStart= ( self.auxStart + 1 ) % 3
def show(self):
self.debug("start")
for i in self.ipInfo:
print(i)
for i in self.tcpInfo:
print(i)
for i in self.protInfo:
print(i)
print("")
self.ipInfo=[]
self.tcpInfo=[]
self.protInfo=[]
def main(self):
try:
if sys.argv[1] == '-d':
self.debugYes=1
self.debug("debuging is turn on")
except (IndexError):
None
self.debug("main start")
for l in sys.stdin:
self.parse(l)
return 0
if __name__ == "__main__":
sys.exit(ParseTsharkOut().main())
aaa
Saturday, March 26, 2011
tcpdump wrapper for all network and systems troubleshooter
UPDATE:
This other post describe a more elegant way how to do the job tshark in network troubleshooting
Origin post:
One of the tool that a network engineers relay every day is a network sniffer. One of the most famous I believe is the 'tcpdump'.
Very often when you troubleshoot a problem you run it many time to verify the traffic on the wire. Let say that at some point you see where the problem may be and you need to sent an email with your analyze to another person.
Documenting your results can be very time consuming. To minimize our time and increase the quality of the results we would like to attach the dump files we review ourself of course. Unfortunately it can be a little annoying if we need to repeat our troubleshooting again only to save the dumps on the disk this time. Often sending the analyzed text output form tcpdumps is not enough as well.
This small tcpdump wrapper bellow can save you a lot of time by saving the tcpdump data to file and still letting you to follow the data on the screen in a live troubleshooting.
For couple of examples how to run in please scroll down.
The file with source code can be found here mytcpdump.sh
Usage help
Examples:
These 2 examples bellow show how to use this small wrapper. Each time we can monitor live traffic on the console output from tcpdump and in the same time be sure that a copy of the raw tcpdump data is written to the disk.
The file you may want to copy then later is shown at the beginning after the header '[timestmap]'. In our examples the file names are:
This other post describe a more elegant way how to do the job tshark in network troubleshooting
Origin post:
One of the tool that a network engineers relay every day is a network sniffer. One of the most famous I believe is the 'tcpdump'.
Very often when you troubleshoot a problem you run it many time to verify the traffic on the wire. Let say that at some point you see where the problem may be and you need to sent an email with your analyze to another person.
Documenting your results can be very time consuming. To minimize our time and increase the quality of the results we would like to attach the dump files we review ourself of course. Unfortunately it can be a little annoying if we need to repeat our troubleshooting again only to save the dumps on the disk this time. Often sending the analyzed text output form tcpdumps is not enough as well.
This small tcpdump wrapper bellow can save you a lot of time by saving the tcpdump data to file and still letting you to follow the data on the screen in a live troubleshooting.
For couple of examples how to run in please scroll down.
The file with source code can be found here mytcpdump.sh
# you can define the filter and options in your bash variables
# example:
# T_FILTER='arp or icmp or not ip ( net 10.0.0.0/8 )
# T_OPTIONS='-s0 -nn'
# ------------------------------------------
# arg1 - filter to the wireshark
# arg2 - options to wireshark
mytcpdump () {
# parse args
DEFAULT_OPT='-s0 -l -nn -w - -i any'
if [ 'x-h' = x"$1" ] ; then
echo
echo "usage: mytcpdump [arg1] [arg2]"
echo " arg1 - wireshark network filter, by example: 'arp and (net 10/8)'"
echo " arg2 - wireshark options, default: '$DEFAULT_OPT'"
echo ""
echo " example:"
echo " mytcpdump"
echo " mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22'"
echo " mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22' '-s0 -l -nn -i eth0 -w -' "
echo
return
fi
# filters
if [ '1' != 1"$1" ] ; then
filter="$1"
elif [ '2' != 2"$T_FILTER" ]; then
filter=$T_FILTER
else
filter=""
fi
# options
if [ '1' != 1"$2" ] ; then
opts="$2"
elif [ '2' != 2"$T_OPTIONS" ]; then
opts=$T_OPTIONS
else
opts="$DEFAULT_OPT"
fi
t=`date +%s`;
echo "[$t]: timestamp is $t"
echo "[$t]: wireshark optoins are <$opts>"
echo "[$t]: wireshark filter is <$filter>"
cmd="tcpdump $opts $filter"
echo "[$t]: tcpdump cmd is <$cmd>"
f="/var/tmp/tcpdump.$t.pcap"
echo "[$t]: tcpdump pcap file <$f>"
chain="$cmd | tee $f | tcpdump -r- -nn"
echo "[$t]: running the bash command chains <$chain>"
$cmd | tee $f | tcpdump -r- -nn
}
alias myt='mytcpdump'
Usage help
# myt -h usage: mytcpdump [arg1] [arg2] arg1 - wireshark network filter, by example: 'arp and (net 10/8)' arg2 - wireshark options, default: '-s0 -l -nn -w - -i any' example: mytcpdump mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22' mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22' '-s0 -l -nn -i eth0 -w -'
Examples:
These 2 examples bellow show how to use this small wrapper. Each time we can monitor live traffic on the console output from tcpdump and in the same time be sure that a copy of the raw tcpdump data is written to the disk.
The file you may want to copy then later is shown at the beginning after the header '[timestmap]'. In our examples the file names are:
/var/tmp/tcpdump.1301166859.pcap /var/tmp/tcpdump.1301166869.pcap
# myt [1301166859]: timestamp is 1301166859 [1301166859]: wireshark optoins are <-s0 -l -nn -w - -i any> [1301166859]: wireshark filter is <> [1301166859]: tcpdump cmd is <tcpdump -s0 -l -nn -w - -i any > [1301166859]: tcpdump pcap file </var/tmp/tcpdump.1301166859.pcap> [1301166859]: running the bash command chains <tcpdump -s0 -l -nn -w - -i any | tee /var/tmp/tcpdump.1301166859.pcap | tcpdump -r- -nn> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes reading from file -, link-type LINUX_SLL (Linux cooked) 19:14:19.837101 IP 192.168.43.111 > 212.77.100.101: ICMP echo request, id 42106, seq 5396, length 64 ^Ctcpdump: pcap_loop: error reading dump file: Interrupted system call 4 packets captured 6 packets received by filter 0 packets dropped by kernel # myt 'icmp or arp' [1301166869]: timestamp is 1301166869 [1301166869]: wireshark optoins are <-s0 -l -nn -w - -i any> [1301166869]: wireshark filter is <icmp or arp> [1301166869]: tcpdump cmd is <tcpdump -s0 -l -nn -w - -i any icmp or arp> [1301166869]: tcpdump pcap file </var/tmp/tcpdump.1301166869.pcap> [1301166869]: running the bash command chains <tcpdump -s0 -l -nn -w - -i any icmp or arp | tee /var/tmp/tcpdump.1301166869.pcap | tcpdump -r- -nn> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes reading from file -, link-type LINUX_SLL (Linux cooked) 19:14:29.847649 IP 192.168.43.111 > 212.77.100.101: ICMP echo request, id 42106, seq 5406, length 64 ^C2 packets captured 2 packets received by filter 0 packets dropped by kernel tcpdump: pcap_loop: error reading dump file: Interrupted system call
Subscribe to:
Posts (Atom)
Posts
