How to calculate a number of new SSL/TCP connections per every 10ms

Hardware load balancers like F5 are a graet product that offers a lot of featreus still combined with a simple and intuitive management GUI. The only problem is the price you have to pay to buy it and than further to pay the support and the license fees.

When working with F5 I have run once into an interesting SSL/TLS problem. It is documented and described SOL6475: Overview of SSL TPS licensing limits.

The most important part from the solutions is:

The BIG-IP system measures SSL TPS based on client-side connection attempts to any
virtual server configured with a Client SSL profile. SSL TPS is enforced across a
sliding time window. The BIG-IP system utilizes a 10ms window (1/100 of a second)
to calculate the current TPS. If the number of TPS requests within any 10ms window
exceeds 1/100 of the licensed TPS, an error message regarding the TPS limit being
reached is sent to the /var/log/ltm file.

Problem

How to know what clients IPs cause the error to be logged. How to measure and calculate the number of SSL connection per seconds for even 10ms.

Solution

As there are no tools on F5 that helps you to find this out I thought that a simple way to get some visibility of it would be to capture all TCP SYN packets hitting the LB and then later do some analysis of it. An implementation of this ideas in a form of a python script can be found here [1].

Demonstration

To test sslAnalyze.py script we need first a tcpdump file. For this purpose we can use the nmap command and run a SYN flood. For the desciption of the nmap options you can take a look here [2].

$ nmap -P0 -TNormal -D 1.2.3.4,1.2.3.5,1.2.3.6,1.2.3.7,1.2.3.8,1.2.3.9,1.2.3.10 -iR 10

All what we have to do now is to run on one session a tcpump and on the other the nmap command. As we are only interested in the TCP SYN packets we should tailor the tcpdump filtering syntax properly. A tcpdump that will capture only the SYN packets:

$ tcpdump -vvv -nn -i eth0 -w /var/tmp/syn-flood-example.pcap 'tcp[13]&2!=0 and tcp[13]&16==0' 

All what we have to do is not run our script to see the statistics.

I have to quickly explain the script itself. Once run it will prints on stdout a listing of found connections and additionally will create a log file with a name sslConnHigh.txt for only these connections that are over the threshold.

The parameters that you have to specify are:

• param1 - tcpdump file (it has to have only SYN packets) 
• param2 - time fractions in microseconds ( 1000000 microseconds -> 1 second ) 
• param3 - connection threshold per time to log this result to a sslConnHigh.txt file

Examples

# Example 1: to see a  number of connection per 1 second 

$ python sslAnalyze.py  syn-flood-example.pcap 1000000 1

# Example 2: to see a number of connection per every 500ms (half a second)

$ python sslAnalyze.py  syn-flood-example.pcap 500000 1

# Example3: to see a number of connection per every 500ms (half a second) and log only
# these timestamps that have more than 100 connection in a single half a second
# some example output has been attached as well below

$ python sslAnalyze.py  syn-flood-example.pcap 500000 100

keeping the line: reading from file syn-flood-example.pcap, link-type EN10MB (Ethernet)
                     date     timestamp     sumOfConn [... 500000 microsecond periods ... ]
 Tue Aug 14 23:33:30 2012    1344983610       sum:183     0  183 
 Tue Aug 14 23:33:31 2012    1344983611        sum:95     6   89 
 Tue Aug 14 23:33:32 2012    1344983612       sum:614   430  184 
 Tue Aug 14 23:33:33 2012    1344983613       sum:520   216  304 

To better understand why F5 logs the error message and what trigger the TPS log error messages we have to run this command:

# 10 milliseconds = 10000 microseconds
$ python sslAnalyze.py  syn-flood-example.pcap 10000 [F5_SSL_total_TPS]
$ cat sslConnHigh.txt

In the output you are going to see the timestamps (rounded to 1 second) where the number of connections in a single 10ms window are above the licensing limit you device has. For further analize you can extract these data from the tcpdump with a help of tcpslice tool.

# 1268649656 is an example timestamp from above
$ tcpslice 1268649656  +1 syn-flood-example.pcap -w 1268649656.pcap

$ tcpdump -tt -nr 1268649656.pcap

reading from file 1268649656.pcap, link-type EN10MB (Ethernet)
1268649656.042723 vlan 4093, p 0, IP 19.26.168.192.4598 - 19.26.225.215.443: S 2973530156:2973530156(0) win 64512 mss 1460,nop,nop,sackOK
1268649656.056163 vlan 4093, p 0, IP 19.89.139.199.1622 - 19.26.225.23.443: S 1522394445:1522394445(0) win 64512 mss 1460,nop,wscale 0,nop,nop,sackOK

References

1. https://github.com/rtomaszewski/experiments/blob/master/sslAnalyze.py
2. http://www.hcsw.org/reading/nmapguide.txt
3. http://danielmiessler.com/study/tcpdump/