Default ADX Brocade health settings

On Broacde ADX load balancer we configure virtual server (as a virtual IP used to load balancer the incoming traffic) and real servers as resources where the reqeusts should be sent to (aka pool members).

A user intuition says that if you enable healt check (regardless of type  L2/L3/L4 or L7) on a real server the ADX should continue to observer and availability of the server and react to changes. Unfortunately the default setting on the ADX for L4/L7 health checks are not working that way.

# example config

 
server real myserver1 10.0.0.1
 port http
 port http keepalive
 port http url "HEAD /"
 port http l4-check-only
 port ssl
 port ssl l4-check-only
!
server real myserver2 10.0.0.2
 port http
 port http keepalive
 port http url "HEAD /"
 port http l4-check-only
 port ssl
 port ssl l4-check-only

server virtual VIP_1.1.1.1 172.16.0.100
 predictor least-conn
 port http
 port http csw-policy "redirect_ssl"
 port http csw
 port ssl sticky
 bind http myserver1 http myserver2  http
 bind ssl myserver1 ssl myserver2  ssl

When all applications run correctly we can see that the VIP status on the LB is tracked correctly:

 
#sh server bind

Virtual server: VIP_108.166.11.177       Status: enabled  IP: 172.16.0.100
        http -------> myserver1: 10.0.0.1,  http (Active)
                      myserver2: 10.0.0.2,  http (Active)
         ssl -------> myserver1: 10.0.0.1,  ssl (Active)
                      myserver2: 10.0.0.2,  ssl (Active)

Problem

The LB is not changing status for the port 443 once the application is stopped on the server (myserver1 or myserver2)

Solution

By default the heal checks are performe only at the begining

http://www.brocade.com/support/Product_Manuals/ServerIron_SLBGuide/health.4.2.html

By default, the ServerIron ADX does not repeat the Layer 4 health check after bringing up the port when you bind the real server to the virtual server. However, you can enable a periodic keepalive health check for the port. To configure the keepalive health check globally, configure a port profile for the port. You also can enable or disable the keepalive health check on individual real servers.

Once you bind a real server to the virtual serve the ADX is not tracking the availability of the real server any more. To force the ADX to sent heal check requests on a regular basis to the real server to monitor its availability we have to enable the availability options like below.

 
conf t
server real myserver1
port ssl keepalive

server real myserver2
port ssl keepalive

The final configuratin is going to look like:

 
server real myserver1 10.0.0.1
 port http
 port http keepalive
 port http url "HEAD /"
 port http l4-check-only
 port ssl
 port ssl l4-check-only
 port ssl keepalive
!
server real myserver2 10.0.0.2
 port http
 port http keepalive
 port http url "HEAD /"
 port http l4-check-only
 port ssl
 port ssl l4-check-only
 port ssl keepalive

server virtual VIP_1.1.1.1 172.16.0.100
 predictor least-conn
 port http
 port http csw-policy "redirect_ssl"
 port http csw
 port ssl sticky
 bind http myserver1 http myserver2  http
 bind ssl myserver1 ssl myserver2  ssl

The origin of the name Google

I've recently watched a movie National Geographic - Inside Google (High-Definition) (another link to the video is below as well) and was astonished by the Google word root :). The Google company name is based on a big number name that in math is called googol. You put 'e' instead of 'o' and there you we have a Google namesake ;).

Openstack progress and growing rate

Openstack has finished official 3 years. By looking back we can understand its future and where it is going. Couple of interesting facts taken from the slide OpenStack 3rd Birthday Presentation.

• Popularity among communities and companies 

• Number of source code line 

• Number of companies being involved in the project

References

1. http://www.openstack.org/birthday?awesm=awe.sm_r36Ss

ASA IPsec VPN filters explained

There is a standard ACL that we use to control the ingress and egress traffic of an interface on the ASA firewall. When it comes to IPsec VPN there are vpn-filter ACLs that can be used additionally (or instead) to control the traffic on a more granular basis.

Problem

How does a vpn-filter (VPN filter ACL/VPN filter access list ) works and how it is different from a standard ACL.

Solution description

The way the ASA is processing and applying the standard ACL is different from how vpn filter ACL (vpn-filter ACL) work.

Normally when defining the VPN filter ACL rules you will specify them in this format:

access-list <acl-no> <permit/deny> ip <remote network> <local network>

where,
 - local network are the FW local segments or segments we want a VPN client to have access to
 - remote network is the network the VPN traffic (or the VPN user traffic) is coming from

Below are some extracts from available documentation I found:

Description from various documentation links:

• PIX/ASA 7.x and Later: VPN Filter (Permit Specific Port or Protocol) Configuration Example for L2L and Remote Access

When you construct the ACLs for use with the vpn-filter feature the ACLs are constructed with the post-decrypted traffic (inbound VPN traffic) in mind. However, they are also applied to the traffic originated in the opposite direction.

• Configuring Tunnel Groups, Group Policies, and Users from Cisco ASA 5500 Series Configuration Guide using the CLI, 8.3

A vpn-filter command is applied to post-decrypted traffic after it exits a tunnel and pre-encrypted traffic before it enters a tunnel. An ACL that is used for a vpn-filter should NOT also be used for an interface access-group. When a vpn-filter command is applied to a group policy that governs Remote Access VPN client connections, the ACL should be configured with the client assigned IP addresses in the src_ip position of the ACL and the local network in the dest_ip position of the ACL.

When a vpn-filter command is applied to a group-policy that governs a LAN to LAN VPN connection, the ACL should be configured with the remote network in the src_ip position of the ACL and the local network in the dest_ip position of the ACL.

• vpn-filter from Cisco ASA 5500 Series Command Reference, 8.2

By design, the vpn-filter feature allows for traffic to be filtered in inbound direction only. The outbound rule is automatically compiled.

• VPN-Filter v Interface ACLs on ASA 5525

What you also have to consider especially with L2L VPN Filter ACL is that their format is different from interface ACL and Client VPN Filter ACL.

In the L2L VPN Filter ACLs you ALWAYS define the source address as the "remote network". This creates every now and then confusion on how the rules should actually look like. I also potentially allows more traffic than you want as the single ACL rule is bidirectional.

• ASA unidirectional VPN tunnel

The ASA has this strange little thing called a filter list.  It is basically an acl that works in both directions for vpn traffic.  The challenge is that it is one acl that works in both directions.  So the granularity is limited a bit, but you can control things based on the tcp/udp port.

• VPN-filter question in 8.2.3 or 8.3.2

Well firstly the group-policy and hence the vpn-filters will take effect only after tunnel comes up. So you would not be able to restrict what traffic brings up the tunnel using vpn-filters.

Conclusion and things to remember 

1. The vpn-filters work on top of crypto domain; you need to first define an interesting traffic to bring tunell up in crypto domain and later it can be more filtered by the VPN filters
2. Single VPN filter ACL is stateful (once a rule allows the traffic through the return traffic will be allowed as well)
3. But for every single VPN filter ACL the ASA engine will create another implicit ACL rule; the 2th rule will permit the other peer to initiate traffic and sent it over VPN tunnel
4. You define the vpn-filter rule from the remote FW perspective; on our FW the vpn filter inspect the ingress traffic; once the decrypted packet enters FW and is encrypted the src and destination and ports are checked by our filter
5. VPN filters are not checking egress VPN traffic (how to inspect an encrypted traffic any way)
6. The FW incoming traffic on its local networks is controlled by the implicit VPN filter rules
7. In another words the vpn filter control explicitly packets AFTER they are decrypted by the FW and the implicit FW rules control the traffic BEFORE it enters the tunnel; even if you want to filter only the traffic BEFORE it enters the VPN you need to specify the explicit AFTER VPN filter instead
8. The vpn filter control the in and out VPN traffic in the same time; you can't have 2 VPN filters configured 
9. Changes to the VPN filter using DENY statements take affect immediately
10. Changes to the VPN filter using the PERMIT statements requires the tunnel to be restarted

References

1. http://www.alfredtong.com/cisco/cisco-asa-vpn-filter-tips-and-misc/
2. http://netribe.blogspot.co.uk/2009/03/these-cisco-asa-vpn-filters-could-trip.html

Nicira NVP network controller

Networking is evolution and often going toward automation and virtual networking. Looking at the available resources we can say that there is a lot of stuff to read about Cisco Nexus platform and the VXLAN protocol. Comparing this with the SDN vendors like Nicira or Midokura the situation is not the same. There isn't available open documentation for the products. There is only a few videos on YouTube that demonstrate what the products do and how they look like.

Below are some of the videos I came across that gives you a glitch into the Nicira NVP cluster.

• VMware Nicira Network Virtualization and VPLEX

• OpenStack with VMware vSphere

Ways to parse tmsh output and automate task on F5

Problem

How to parse the output from tmsh shell.
How to write your own scripts to automate manual or complex actions on the F5 load balancer.

Solution

By default the 'show' action generate human readable output. With the additional option 'field-fmt' you can generate more machine-readable format that is more suitable for parsing.

 
lbal1(Active)(tmos.ltm.virtual)# show vs_1.1.1.1

Ltm::Virtual Server: vs_1.1.1.1
---------------------------------------------------------
Status
  Availability     : available
  State            : enabled
  Reason           : The virtual server is available
  CMP              : enabled
  CMP Mode         : all-cpus


Traffic                    ClientSide  Ephemeral  General
  Bits In                       89.1G          0        -
  Bits Out                       1.1T          0        -
  Packets In                    89.5M          0        -
  Packets Out                  124.7M          0        -
  Current Connections              54          0        -
  Maximum Connections             334          0        -
  Total Connections              2.6M          0        -
  Min Conn Duration/msec            -          -      367
  Max Conn Duration/msec            -          -   186.4M
  Mean Conn Duration/msec           -          -    84.1K
  Total Requests                    -          -     6.0M

 
lbal1(Active)(tmos.ltm.virtual)# show vs_1.1.1.1 field-fmt
ltm virtual vs_1.1.1.1 {
    clientside.bits-in 89.1G
    clientside.bits-out 1.1T
    clientside.cur-conns 67
    clientside.max-conns 334
    clientside.pkts-in 89.5M
    clientside.pkts-out 124.7M
    clientside.tot-conns 2.6M
    cs-max-conn-dur 186.4M
    cs-mean-conn-dur 76.7K
    cs-min-conn-dur 367
    ephemeral.bits-in 0
    ephemeral.bits-out 0
    ephemeral.cur-conns 0
    ephemeral.max-conns 0
    ephemeral.pkts-in 0
    ephemeral.pkts-out 0
    ephemeral.tot-conns 0
    name vs_95.138.158.17
    tot-requests 6.0M
    virtual-server.cmp-enable-mode all-cpus
    virtual-server.cmp-enabled enabled
    virtual-server.status.availability-state available
    virtual-server.status.enabled-state enabled
    virtual-server.status.status-reason The virtual server is available
}

To create your own scripts you can use grep to parse the output above or take it to the next level and write a native script in TCL using the tmsh API.

References

1. https://devcentral.f5.com/wiki/TMSH.ConfigSearch.ashx
2. https://devcentral.f5.com/community/group/aft/1180665/asg/53
3. https://devcentral.f5.com/wiki/TMSH.tmsh__get_config.ashx
4. https://devcentral.f5.com/wiki/TMSH.tmsh__get_status.ashx

What is AppFlow

In networking one of the words that has recently become popular is 'flow'. We have OpenFlow protocols that is a new emerging standard in network visualization that uses flows to describe what action should happen to a packet in OVS switch. We have sFlow and Netflow that are widely used by networking vendors for statistics information and performance data gathering from network devices. I've come recently across a new term: AppFlow.

Problem

What is AppFlow?

Analisis

In the video below we can learn that AppFlow is a protocols that can be used for application monitoring and visibility. With the industry acceptance of ADS/ADC systems (Application Delivery System and Application Delivery Controller) it is therefor logically rational to extend load balancers functionality to allow them to collect application specific data. The idea is similar how sFlow/Netflow work where additionally to IP and TCP based information we want to collect application specify information as well.

References

1. http://blogs.citrix.com/2011/05/09/introducing-appflow-netflow-for-apps/
2. http://www.youtube.com/watch?v=MNbLi-t7ZPE
3. http://www.appflow.org/
4. http://www.citrix.com/products/netscaler-application-delivery-controller/features/visibility.html
5. http://www.citrix.com/products/netscaler-application-delivery-controller/how-it-works/technologies.html

The purpose of OVSDB in Open vSwitch stack

Open vSwitch is a virtual switch that is most commonly used with all sort of types of hypervisor like XEN, KVM and others. As a software component in a hypervisor software stack it allows to implement physical and virtual network connectivity between virtual machines (VM) that run on it. Because of its capabilities (example: managing GRE tunnels between hypervisors) it is often used in cloud deployments to provide interconnectivity between VM on different hypervisor servers as well.

The Open vSwitch secret to its successfully adoption is a modular design. The different components provide:

• Support for management tasks with the help of OVSDB protocol
• Data plane support (OpenFlow flow control protocol) 
• Support for external control and management to implement a sophisticated control plane mechanism 

These capabilities provide foundation for any software trying to implement a network management, orchestration and automation systems.

In previous post (What is Openflow) we discussed the benefits of OpenFlow protocol and its data wire protocol frame headers (How is a single OpenFlow flow defined on a switch ).

For the OVSDB there is an internet draf here: http://tools.ietf.org/html/draft-pfaff-ovsdb-proto-02. The protocols is used additionally to Openflow to implement actions like:

• Virtual switch (bridge) creation
• Virtual port creation/deletion on bridges
• Configuration of managers (used to manage bridges) and controllers (used to managed flows on bridges) 

Additional examples can be found on the ovs-discuss mailing list like in this one post: http://openvswitch.org/pipermail/discuss/2013-July/010464.html

References

1. http://openvswitch.org/faq/ / 
2. http://tools.ietf.org/html/draft-pfaff-ovsdb-proto-02
3. http://networkheresy.com/tag/ovsdb/

Migration from Sublime 2 to Sublime 3

With the advent of the version 3 of my favourite sublime editor I decided to give it a try and see what need to be change to migrate all my settings from the sublime2 to sublime3. Below is a list of changes I did.

• Copy the Settings User file 
• Copy the Key Binding - User
• Install the Packet Control plugin manager from here using the git method (press Control+Shift+P, type install to see if this has been properly installed)
• Not all packages are yet compatible with ST3, here is a more verbose list https://github.com/wbond/sublime_package_control/wiki/Sublime-Text-3-Compatible-Packages

At first glance it was easy to copy the basic settings. There are some compatibility issues with the modules although.

References

1. http://www.sublimetext.com/blog/articles/sublime-text-3-beta

How to view and edit cookies in Chrome

When you configuring more complicated scenarios on load balancers it is only a matter of time when you start looking at the HTTP requests and responses because things don't work the way you want.

Problem

Q1: How to view browser cookies when viewing sites?
Q2: How to edit cookies in Chrome?

Analisis and results description

Add Q1:

• Start the Developer tools in Chrome. 
• Refresh your page with Control+F5.
• Navigate to the Resources tab and select Cookies

Add Q2:

The Chrome Developer tools doesn't allow you to edit cookies. You can only view or delete them.
To edit cookie values we can use one of the additional plugins: Edit This Cookie

• Install "Edit This Cookie" plugin
• Refresh page with Control+F5
• Once page loads click the cookie icon on the Chrome  
• Change the cookie 

NAT order of operation on Cisco ASA firewall

There are many types of NAT you can configure on the ASA FW. This is a short summary with examples for ASA 8.2/8.3 software.

• Dynamic NAT

! The pool-number parameter ( 2, in this case) binds the 'global' and 'nat' commands
nat (dmz) 2 10.10.10.128 255.255.255.128
global (out) 2 172.16.16.129-172.16.16.254 netmask 255.255.255.128

• Dynamic PAT

! Configuring PAT using the global address 172.16.16.126
ASA2(config)# nat (dmz) 1 10.10.10.0 255.255.255.0
ASA2(config)# global (out) 1 172.16.16.126

• Identity NAT

! Configuring Identity NAT
ASA2(config)# nat (dmz) 0 10.10.10.128 255.255.255.192

• Static NAT

! Static translation for an individual host (/32 is the default netmask)
ASA2(config)# static (dmz,out) 172.16.16.140 10.10.10.140

• Static Policy NAT

ASA2(config)# access-list STATIC-POLICY1 extended permit ip host 10.10.10.148 host 172.16.16.200
ASA2(config)# static (dmz,out) 172.18.18.148 access-list STATIC-POLICY1

! For other destinations source address 10.10.10.148 is translated to 172.16.16.148
ASA2(config)# static (dmz,out) 172.16.16.148 10.10.10.148 netmask 255.255.255.255

• Dynamic Policy NAT

! Packets from 10.10.10.128/27 travelling to host 172.16.16.200 undergo Policy NAT
access-list DYN-POLICY1 extended permit ip 10.10.10.128 255.255.255.224 host 172.16.16.200
!
nat (dmz) 4 access-list DYN-POLICY1
global (out) 4 172.17.17.129-172.17.17.158 netmask 255.255.255.224
!

• Dynamic Policy PAT

! Policy PAT for source subnet 10.10.10.128/26 going to host 172.16.16.200
access-list DYN-POLICY2 extended permit ip 10.10.10.128 255.255.255.192 host 172.16.16.200
!
ASA2(config)# nat (dmz) 3 access-list DYN-POLICY2
ASA2(config)# global (out) 3 172.16.16.125

• NAT Exemption

! Connections between 10.10.10.128/28 and 172.16.16.0/24 are exempted from NAT
access-list NONAT extended permit ip 10.10.10.128 255.255.255.240 172.16.16.0 255.255.255.0
!
nat (dmz) 0 access-list NONAT

Problem

In what order and precedence is ASA firewall processing various NAT configurations.

NAT precedence rules 

Step 1.
NAT Exemption: This is always the first to be checked and has precedence over any other type of NAT rule that eventually conflicts with it.

Step 2.
Static Policy NAT: The motivation for this type of rule is to allow the selection of distinct global addresses for a given laddr, depending on the destination address (faddr) being contacted. This can be thought of as an exception to a generic static statement and, as such, needs to be configured before regular statics.

Step 3.
Static NAT: This is checked before all the Dynamic, Dynamic Policy, and Identity NAT rules. If some hosts that fall within a NAT Exemption range require static translation, the pertinent exceptions in the nat 0 access-list command need to be created, as previously illustrated in Example 8-14.

Step 4.
Dynamic Policy NAT/PAT: These rules are checked before the Dynamic and Identity NAT rules. If two rules of this category exist, the precedence is given by the order in which they were configured (does not matter if is a Policy PAT or Policy NAT).

Step 5.
Identity NAT: This unidirectional translation bypass method is evaluated before any Dynamic NAT or Dynamic PAT rule.

Step 6.
Dynamic NAT: This category has precedence over Dynamic PAT only.

Step 7.
Dynamic PAT: If after running through all the previous NAT types, ASA does not find a match, it still searches for a Dynamic PAT. If no matching rule is located, the implicit deny rule that characterizes the NAT-control model is enforced.

References

• Cisco Firewalls By: Alexandre M.S.P Moraes, chapter "Outbound NAT Analysis" 
• http://blog.ciscoinferno.net/you-me-and-nat-8-3
• Network Address Translation (NAT) order of operation in the PIX Firewall
• Packet Flow through Cisco ASA Firewall
• Order of NAT Commands Used to Match Real Addresses