How to use F5 Wireshark Plugin for LTM troubleshooting

In this post we are going to look how to use F5 Wireshark Plugin to troubleshoot networking issues on BigIP LTM.

• Download the and install the plugin in your Wireshark

The full instruction are here F5 Wireshark Plugin. In essence you needed to copy the f5ethtrailer.dll file into C:\Program Files (x86)\wireshark\wireshark16\WiresharkPortable\ and restart my Wireshark.

Once you restart wireshark go to menu Help - About Wireshark, Plugins tab. You should be able to see the plugin listed there if properly installed.

• The plugin is useful only if you take a capture on LTM with 'noise' information.

The noise is an internal information that TMM is attaching and managing for every packet when is being processed. To have a capture with noise these are the minimal options you need to specify:

tcpdump -w /var/tmp/capture.pcap -s0 -i _interface_:nnn

where the _interface_ can be:

•  1.1 - example of an physical interface
• dmz_vlan - a name you gave to your vlan when created
• 0.0 - is the equivalent of 'any' interface what means capture on all interfaces and all vlans

My favourite syntax is usually something like this:

tcpdump -s0 -nn -w /var/tmp/test1-$(date +%s).pcap -i 0.0:nnn '(host _ip_ and port _port_ ) or arp or not ip' 

• Open the capture in wireshark as normal

Once you open you will noticed that there is additional section in the packet details.

• The most useful part of using this plugin is that you can quickly and easily find the client and server site traffic in the capture (It can be a challenging when you have multiple tcp streams and OneConnect profile):
• Find a single packet of the flow you are interested in (search for VIP or client ip for example).
• Find the "Flow ID" from the F5 Ethernet trailer (see the picture above for example).
• Click with right mouse taste on the Flow ID field and select "Prepare as Filter".
• In the Filter box (on top ) it will pre-populate the syntax for you.
• Copy the hex value and delete the '.flowid == hex' part and start typing '.'  (dot).
• It will mediately give you a list of possible options, select anyflowid and copy the hex back as it was originally. Example:

The original filter         : f5ethtrailer.flowid == 0x0d2e6dc0

Filter after modifications  : f5ethtrailer.anyflowid == 0x0d2e6dc0

• Press Apply button

This filter is gong to find the client and server site flows for you. You can then analyse them packet by packet to find out and understand how and why LTM load balance it to one or another pool member.

References

https://devcentral.f5.com/wiki/advdesignconfig.F5WiresharkPlugin.ashx
https://devcentral.f5.com/questions/tcpdump-with-multiple-pool-members
SOL13637: Capturing internal TMM information with tcpdump

How to do URL based load balancing on F5

There are many load balancers out there. Some of them offer a great flexibility to control the traffic by allowing a user to upload a custom script that implement the load balancing algorithm to solve a particular problem.

Problem

How to do HTTP URL based load balancing on F5.

Solution and demonstration

This is an iRule script that inspects the HTTP GET URL string to decided where to load balance it: https://github.com/rtomaszewski/f5/blob/master/lb-based-on-url.tcl.

Create default pool

Create VIP

Create custom pools

Testing

To verify that our iRule is working properly we can enable debugging by changing the iRule variable DEBUG to 1.

Next we can simulate traffic

curl -v http://vip/
curl -v http://vip/url1
curl -v http://vip/url2
curl -v http://vip/url3

And watch the logs on the lb.

tail -f /var/log/ltm

Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80
Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url1
Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url1
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url2
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url2
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url3
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url3

Reference

https://devcentral.f5.com/wiki/iRules.HomePage.ashx

VLAN failsafe problem for F5 HA new builds

The F5 load balancers are powerful devices that support variety of high availability features. A full list  and description can be found at Configuring High Availability in TMOS Management Guide for BIG-IP Systems" document.

Supported HA features:

The BIG-IP system offers three types of fail-safe:

System fail-safe Monitors the switch board component and a set of key system services.

Gateway fail-safe Monitors traffic between the BIG-IP system and a gateway router.

VLAN fail-safe Monitors traffic on a VLAN.

Problem

Both F5s in HA cluster fail over and go into standby mode when fail-safe is enabled on the VLAN that doesn't see any traffic.

Analysis and workaround descriptions

When you are building a new HA cluster this is not going to cause any major issues. Usually for new builds the cluster will be build without any servers behind the LTM devices. If both devices go into standby mode it may be surprising but a simple ping from one F5 to another should bring both them into standby/active again.

Unfortunately the issue can be experienced as well as in production when you are adding a new VLAN to a running HA cluster. If the VLAN has fail-safe enabled and if there are not devices behind both F5 LTMs the new VLAN may trigger the fail over on both nodes. As expected it is caused by the VLAN fail-safe when there is not traffic on the VLAN.

There are couple workarounds that can be applied, two examples are listed below.

Workaround 1

As per the SOL13297: Overview of VLAN failsafe (10.x - 11.x) set true to LTM data base variable failover.vlanfailsafe.resettimeronanyframe.

 
modify /sys db failover.vlanfailsafe.resettimeronanyframe value [true|false]

Workaround 2

Create a pool with all self IPs of the affected VLAN to allow the LTM to detect traffic on the new VLAN and to prevent fail-safe to kick in.

Problem booting virtual F5 LTM on Virtual box

There is no better way to learn about LTM load balancer then playing and testing it even though it may be a virtual appliance and running LTM  10.1 version.

To get started download the LTM BIGIP-10.1.0.3341.1084.ova software and the base registration key (you will need to register first).

Problem

After you import the ova file and boot the BigIp you can be presented with the following error:

 
Memory for crash kernel (0x0 to 0x0) notwithin permissible range

Resolution

The VM should be bootable out of the box on a PC with an Intel CPU  64bit. If you are using AMD 64bit system instead you need to enable the "Enable IO APIC" option under VM properties, System, Motherboard.

After the change the system will still show the message but the booting process will no longer stop.

After the system boots login as root/default. Finds the management ip and using your OS browser navigate to https://ip to activate the trial license.

 
tmsh list /sys management-ip

References

https://www.f5.com/trial/
http://lost-and-found-narihiro.blogspot.co.uk/2011/04/how-to-fly-big-ip-ltm-ve-in-vmware.html
https://forums.virtualbox.org/viewtopic.php?f=5&t=24988

Ways to parse tmsh output and automate task on F5

Problem

How to parse the output from tmsh shell.
How to write your own scripts to automate manual or complex actions on the F5 load balancer.

Solution

By default the 'show' action generate human readable output. With the additional option 'field-fmt' you can generate more machine-readable format that is more suitable for parsing.

 
lbal1(Active)(tmos.ltm.virtual)# show vs_1.1.1.1

Ltm::Virtual Server: vs_1.1.1.1
---------------------------------------------------------
Status
  Availability     : available
  State            : enabled
  Reason           : The virtual server is available
  CMP              : enabled
  CMP Mode         : all-cpus


Traffic                    ClientSide  Ephemeral  General
  Bits In                       89.1G          0        -
  Bits Out                       1.1T          0        -
  Packets In                    89.5M          0        -
  Packets Out                  124.7M          0        -
  Current Connections              54          0        -
  Maximum Connections             334          0        -
  Total Connections              2.6M          0        -
  Min Conn Duration/msec            -          -      367
  Max Conn Duration/msec            -          -   186.4M
  Mean Conn Duration/msec           -          -    84.1K
  Total Requests                    -          -     6.0M

 
lbal1(Active)(tmos.ltm.virtual)# show vs_1.1.1.1 field-fmt
ltm virtual vs_1.1.1.1 {
    clientside.bits-in 89.1G
    clientside.bits-out 1.1T
    clientside.cur-conns 67
    clientside.max-conns 334
    clientside.pkts-in 89.5M
    clientside.pkts-out 124.7M
    clientside.tot-conns 2.6M
    cs-max-conn-dur 186.4M
    cs-mean-conn-dur 76.7K
    cs-min-conn-dur 367
    ephemeral.bits-in 0
    ephemeral.bits-out 0
    ephemeral.cur-conns 0
    ephemeral.max-conns 0
    ephemeral.pkts-in 0
    ephemeral.pkts-out 0
    ephemeral.tot-conns 0
    name vs_95.138.158.17
    tot-requests 6.0M
    virtual-server.cmp-enable-mode all-cpus
    virtual-server.cmp-enabled enabled
    virtual-server.status.availability-state available
    virtual-server.status.enabled-state enabled
    virtual-server.status.status-reason The virtual server is available
}

To create your own scripts you can use grep to parse the output above or take it to the next level and write a native script in TCL using the tmsh API.

References

1. https://devcentral.f5.com/wiki/TMSH.ConfigSearch.ashx
2. https://devcentral.f5.com/community/group/aft/1180665/asg/53
3. https://devcentral.f5.com/wiki/TMSH.tmsh__get_config.ashx
4. https://devcentral.f5.com/wiki/TMSH.tmsh__get_status.ashx

F5 Network BigIp cheat sheet

This post is a work in progress...

• How to generate a list with one self ip and vlans per line 

# tmsh list /net self  | egrep 'self|vlan' | xargs -n 6 echo
net self 10.2.2.2/30 { vlan FAILOVER
net self 10.176.30.100/19 { vlan hybridServiceNet-140
net self 10.176.30.102/19 { vlan hybridServiceNet-140
net self 10.176.94.132/19 { vlan hybridServiceNet-142

Or 


# tmsh list net self | egrep -v 'floating|unit|allow-service' | xargs -n 7
net self 10.178.191.49/18 { vlan rackconnect110 }
net self 10.179.63.181/18 { vlan rackconnect112 }

• How to simulate F5 health check requests with empty Host header

http://rtomaszewski.blogspot.co.uk/2013/04/how-to-simulate-f5-health-check.html

• How to parse tmsh output

http://rtomaszewski.blogspot.co.uk/2013/07/ways-to-parse-tmsh-output-and-automate.html

Cisco is leaving the ADC market and recommends NetScaler from Citrix

The networking giant Cisco is leaving the competitive Application Delivery Controller (ADC) market. The Cisco CSS and Cisco ACE products were not able to keep the pace with the competitors like F5 Networks, Citrix, Radware and Brocade.

The doc [3] provides some guidance how to choice the right hardware.

References

1. http://www.citrix.com/products/netscaler-application-delivery-controller/how-it-helps/cisco.html
2. http://www.networkworld.com/community/blog/ciscos-exit-adcs-should-come-no-surprise
3. http://www.citrix.com/content/dam/citrix/en_us/documents/products/ACE_Product_Migration_Guide_final.pdf

How to simulate F5 health check requests with empty Host header

The curl tool is one of the tool you use on a daily basis when verifying and checking health checks on the F5 load balancer. But it has its limitations. One of these is that it don't insert an empty Host header.

 
# curl -v -H 'Host:' http://192.168.1.138 -o tmp
* About to connect() to 192.168.1.138 port 80
*   Trying 192.168.1.138... connected
* Connected to 192.168.1.138 (192.168.1.138) port 80
> GET /id.aspx HTTP/1.1
> User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Accept: */*
>

This simple command will use nc tool to initiate a TCP session and sent the whole HTTP request as you define it in the string between the quotes. That way we can simulate an empty Host header if we need it. For more option how to troubleshoot monitoring issue check this one https://devcentral.f5.com/wiki/advdesignconfig.TroubleshootingLtmMonitors.ashx.

 
$ echo -e "GET / HTTP/1.1\r\nHost:\r\nConnection: Close\r\n\r\n" | tee req.txt | nc -v 192.168.1.186 80
$ cat req.txt
GET / HTTP/1.1
Host:
Connection: Close

How to host 2 ssl sites on a single public IP on F5 load balancer

As the pool of free IPv4 is getting lower it is important to efficiently mange our existing assigned public IPs. One thing that can be used is to use SSL offloading and hosting multiple HTTPS sites on single public IP using wild cart certificates.

The core of this solution is the configuration of the lb and the vhost setting on your server. The server reads the HTTP Host header and base on it decide what site the user try to access.

Problem

How to configure servers and load balancer to host 2 ssl sites on a single public IP.

Analisis and example configuration

• Example F5 configuration

For SSL to work we need a wild card certificate.
We have to import the certificate and key to F5 and create client site ssl profile.

Example wild cart certificate:

 
# openssl x509 -in /var/tmp/wildcard.rado.net.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:9c:4a:4b:11:11:bc
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certificates.godaddy.com/repository, CN=Go Daddy Secure Certification Authority/serialNumber=111111
        Validity
            Not Before: Nov 21 11:57:16 2012 GMT
            Not After : Nov 20 09:31:37 2013 GMT
        Subject: O=*.rado.net, OU=Domain Control Validated, CN=*.rado.net
        ....

SSL profile:

 
# tmsh list ltm profile client-ssl  wildcard-client-ssl-profile
ltm profile client-ssl wildcard-client-ssl-profile {
    cert wildcard-rado.net.crt
    chain CA.crt
    defaults-from clientssl
    key wildcard-rado.net.key
}

Virtual server to terminate and load balance the traffic:

 
# tmsh list ltm virtual VS-1.1.1.1-443
ltm virtual VS-1.1.1.1-443 {
    destination 192.168.99.68:https
    ip-protocol tcp
    mask 255.255.255.255a
    pool POOL-192.168.99.68-80
    profiles {
        http { }
        tcp { }
        wildcard-client-ssl-profile {
            context clientside
        }
    }
}

Servers where the traffic is going to be load balanced:

 
# tmsh list ltm pool POOL-192.168.99.68-80
ltm pool POOL-192.168.99.68-80 {
    load-balancing-mode least-connections-member
    members {
        10.177.1.1:http {
            session monitor-enabled
        }
        10.177.1.2:http {
            session monitor-enabled
        }
    }
    monitor http
}

• Apache configuration on one of the servers


This is a simple example configuration for the http server.


 
# cat vhost1.conf
<VirtualHost *:80>
        ServerName vhost1.rado.net
        DocumentRoot /var/www/vhost1
</VirtualHost>

# cat vhost2.conf
<VirtualHost *:80>
        ServerName vhost2.rado.net
        DocumentRoot /var/www/vhost2
</VirtualHost>

# cat /var/www/vhost1/index.html
<html><body><h1>It works!</h1>
<p>
This is VHOST 1
</p>
</body></html>

# a2ensite vhost1.conf 
# a2ensite vhost2.conf 

# service apache2 reload


A quick local test on the server will confirm if the config is correct.


 
# curl -v -H "Host: vhost1.rado.net" http://5.1.1.1
* About to connect() to 5.1.1.1 port 80 (#0)
*   Trying 5.1.1.1... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> Host: vhost1.rado.net
>
< HTTP/1.1 200 OK
< Date: Sat, 30 Mar 2013 23:42:20 GMT
< Server: Apache/2.2.22 (Ubuntu)
< Last-Modified: Sat, 30 Mar 2013 23:19:21 GMT
< ETag: "b46bb-47-4d92c9e17f040"
< Accept-Ranges: bytes
< Content-Length: 71
< Vary: Accept-Encoding
< Content-Type: text/html
<
<html><body><h1>It works!</h1>
<p>
This is VHOST 1
</p>
</body></html>
* Connection #0 to host 5.79.21.166 left intact
* Closing connection #0

Redhat application deployment automaton with Puppet module and F5

I have found this youtube video [1] on the the Puppet channel. It is an interesting presentation about the day to day issues the folks from Redhat run into when developing, managing and deploying new code for customer facing sites they host.

On of the things they mention that was difficult for them and why the started this project were:

• Lack of a cross functional team who could understand the whole architecture.
• People tend to have a limited understanding of other areas that they don't work in; Examples are SysAdmin about NetAdmin, NetAdmin about Develpers etc.
• On new depoyment a difficulties to find a team/person who can take an ownership of an issue as the problem may be above the area of their own expertise.
• Diverse and inconsistencies dev, staging and production systems.

To solve the problem they decided to refactor the architecture and automate as much as it was possible. The video [1] shows what issues they run into and how they solved it with a help of Puppet module they created and the F5 load balancer.

In short a message they convey in the presentation is: automate, standardize, and automate once again.

References

1. Managing F5 LTM with Puppet - Matthew Carpenter and Bret McMillan of Red Hat
2. http://www.youtube.com/user/PuppetLabsInc

Software defined networks (SDN) with F5 and Microsoft Hyper-V

There is a lot of going on in the network space. As the virtualization is changing the server landscape there is as well as more and more talk about virtualization in the network.

Microsoft and F5 has collaborated together and with the new BigIp software release and as well as generation Windows Hyper-V technology they both offer a Software defined networks (SDN) solution for Windows based cloud servers.

Software Defined Networking, Enabled in Windows Server 2012 and System Center 2012 SP1, Virtual Machine Manager

F5'S Network Virtualization solution optimizes app delivery for Windows Server 2012 Hyper-V;

MEC 2012--F5 Network Virtualization Solution

How to calculate a number of new SSL/TCP connections per every 10ms

Hardware load balancers like F5 are a graet product that offers a lot of featreus still combined with a simple and intuitive management GUI. The only problem is the price you have to pay to buy it and than further to pay the support and the license fees.

When working with F5 I have run once into an interesting SSL/TLS problem. It is documented and described SOL6475: Overview of SSL TPS licensing limits.

The most important part from the solutions is:

The BIG-IP system measures SSL TPS based on client-side connection attempts to any
virtual server configured with a Client SSL profile. SSL TPS is enforced across a
sliding time window. The BIG-IP system utilizes a 10ms window (1/100 of a second)
to calculate the current TPS. If the number of TPS requests within any 10ms window
exceeds 1/100 of the licensed TPS, an error message regarding the TPS limit being
reached is sent to the /var/log/ltm file.

Problem

How to know what clients IPs cause the error to be logged. How to measure and calculate the number of SSL connection per seconds for even 10ms.

Solution

As there are no tools on F5 that helps you to find this out I thought that a simple way to get some visibility of it would be to capture all TCP SYN packets hitting the LB and then later do some analysis of it. An implementation of this ideas in a form of a python script can be found here [1].

Demonstration

To test sslAnalyze.py script we need first a tcpdump file. For this purpose we can use the nmap command and run a SYN flood. For the desciption of the nmap options you can take a look here [2].

$ nmap -P0 -TNormal -D 1.2.3.4,1.2.3.5,1.2.3.6,1.2.3.7,1.2.3.8,1.2.3.9,1.2.3.10 -iR 10

All what we have to do now is to run on one session a tcpump and on the other the nmap command. As we are only interested in the TCP SYN packets we should tailor the tcpdump filtering syntax properly. A tcpdump that will capture only the SYN packets:

$ tcpdump -vvv -nn -i eth0 -w /var/tmp/syn-flood-example.pcap 'tcp[13]&2!=0 and tcp[13]&16==0' 

All what we have to do is not run our script to see the statistics.

I have to quickly explain the script itself. Once run it will prints on stdout a listing of found connections and additionally will create a log file with a name sslConnHigh.txt for only these connections that are over the threshold.

The parameters that you have to specify are:

• param1 - tcpdump file (it has to have only SYN packets) 
• param2 - time fractions in microseconds ( 1000000 microseconds -> 1 second ) 
• param3 - connection threshold per time to log this result to a sslConnHigh.txt file

Examples

# Example 1: to see a  number of connection per 1 second 

$ python sslAnalyze.py  syn-flood-example.pcap 1000000 1

# Example 2: to see a number of connection per every 500ms (half a second)

$ python sslAnalyze.py  syn-flood-example.pcap 500000 1

# Example3: to see a number of connection per every 500ms (half a second) and log only
# these timestamps that have more than 100 connection in a single half a second
# some example output has been attached as well below

$ python sslAnalyze.py  syn-flood-example.pcap 500000 100

keeping the line: reading from file syn-flood-example.pcap, link-type EN10MB (Ethernet)
                     date     timestamp     sumOfConn [... 500000 microsecond periods ... ]
 Tue Aug 14 23:33:30 2012    1344983610       sum:183     0  183 
 Tue Aug 14 23:33:31 2012    1344983611        sum:95     6   89 
 Tue Aug 14 23:33:32 2012    1344983612       sum:614   430  184 
 Tue Aug 14 23:33:33 2012    1344983613       sum:520   216  304 

To better understand why F5 logs the error message and what trigger the TPS log error messages we have to run this command:

# 10 milliseconds = 10000 microseconds
$ python sslAnalyze.py  syn-flood-example.pcap 10000 [F5_SSL_total_TPS]
$ cat sslConnHigh.txt

In the output you are going to see the timestamps (rounded to 1 second) where the number of connections in a single 10ms window are above the licensing limit you device has. For further analize you can extract these data from the tcpdump with a help of tcpslice tool.

# 1268649656 is an example timestamp from above
$ tcpslice 1268649656  +1 syn-flood-example.pcap -w 1268649656.pcap

$ tcpdump -tt -nr 1268649656.pcap

reading from file 1268649656.pcap, link-type EN10MB (Ethernet)
1268649656.042723 vlan 4093, p 0, IP 19.26.168.192.4598 - 19.26.225.215.443: S 2973530156:2973530156(0) win 64512 mss 1460,nop,nop,sackOK
1268649656.056163 vlan 4093, p 0, IP 19.89.139.199.1622 - 19.26.225.23.443: S 1522394445:1522394445(0) win 64512 mss 1460,nop,wscale 0,nop,nop,sackOK

References

1. https://github.com/rtomaszewski/experiments/blob/master/sslAnalyze.py
2. http://www.hcsw.org/reading/nmapguide.txt
3. http://danielmiessler.com/study/tcpdump/

Do you know what is powering Rackconnect at Rackspace

These a few videos are showing some inside into the Hybrid Cloud Product Rackspace has to offer to its customers:

If you are looking for more information these are some links to read more about Rackconnect:

http://www.rackspace.com/cloud/hybrid/dedicated_cloud/rackconnect/

http://www.rackspace.com/cloud/hybrid/dedicated_cloud/

Working for the best in the industry

Technology leaders

I have recently start doing more analysis and  research of the market trends. It is extraordinary good feeling to know that you can volunteer for a work at the best companies worldwide ;). First it was F5 Networks and now with Rackspace. The blend of Load balancing, Cloud and Linux technologies is going to be the future foundation for next generation data center solution architectures.

References

www.f5.com
www.rackspace.com
www.openstack.org
http://www.diigo.com/user/rtomaszewski/magic_quadrants%20f5