Search This Blog

Monday, September 1, 2014

How to automatically rotate the root password on cloud server

Description

I need to create a public cloud server and use it as a bastion in a secure way.
I hate the java/javascript console that you have to use when something doesn't work with your cloud.
I want to keep the root user enabled. As leaving the password authentication for root is a security risk we need to mitigate this.
By default the default loging method is going to be RSA public key.

The reason I want to keep the root user enabled is that you can easely reset its passwors using the https://mycloud.rackspace.com/ portal. Otherwise the root user should be practically not available.

We could leave it enabled but there is always a risk that somebody with enough time may want to try to hack us.

Problem

How to set up a root password rotation using Cron in Linux,

Solution
 
# crontab -l
# for debugging
# */10 *  *  *    *     echo root:$(/usr/bin/makepasswd --chars 15) | /usr/bin/tee /tmp/test.txt | /usr/sbin/chpasswd
  */10 *  *  *    *     echo root:$(/usr/bin/makepasswd --chars 15) | /usr/sbin/chpasswd

This mitigates the root password attacks and still gives us a possibility to reset the root password over the portal and login over a regular ssh session.

We don't care what the new root password is, if I need it I'll reset it on the myrack portal.

Tuesday, May 27, 2014

How to use F5 Wireshark Plugin for LTM troubleshooting

In this post we are going to look how to use F5 Wireshark Plugin to troubleshoot networking issues on BigIP LTM.
  • Download the and install the plugin in your Wireshark
The full instruction are here F5 Wireshark Plugin. In essence you needed to copy the f5ethtrailer.dll file into C:\Program Files (x86)\wireshark\wireshark16\WiresharkPortable\ and restart my Wireshark.

Once you restart wireshark go to menu Help - About Wireshark, Plugins tab. You should be able to see the plugin listed there if properly installed.

  • The plugin is useful only if you take a capture on LTM with 'noise' information.
The noise is an internal information that TMM is attaching and managing for every packet when is being processed. To have a capture with noise these are the minimal options you need to specify:

tcpdump -w /var/tmp/capture.pcap -s0 -i _interface_:nnn

where the _interface_ can be:
    •  1.1 - example of an physical interface
    • dmz_vlan - a name you gave to your vlan when created
    • 0.0 - is the equivalent of 'any' interface what means capture on all interfaces and all vlans
My favourite syntax is usually something like this:

tcpdump -s0 -nn -w /var/tmp/test1-$(date +%s).pcap -i 0.0:nnn '(host _ip_ and port _port_ ) or arp or not ip' 
  • Open the capture in wireshark as normal
Once you open you will noticed that there is additional section in the packet details.

  • The most useful part of using this plugin is that you can quickly and easily find the client and server site traffic in the capture (It can be a challenging when you have multiple tcp streams and OneConnect profile):
    • Find a single packet of the flow you are interested in (search for VIP or client ip for example).
    • Find the "Flow ID" from the F5 Ethernet trailer (see the picture above for example).
    • Click with right mouse taste on the Flow ID field and select "Prepare as Filter".
    • In the Filter box (on top ) it will pre-populate the syntax for you.
    • Copy the hex value and delete the '.flowid == hex' part and start typing '.'  (dot).
    • It will mediately give you a list of possible options, select anyflowid and copy the hex back as it was originally. Example:
The original filter         : f5ethtrailer.flowid == 0x0d2e6dc0
Filter after modifications  : f5ethtrailer.anyflowid == 0x0d2e6dc0
    • Press Apply button
This filter is gong to find the client and server site flows for you. You can then analyse them packet by packet to find out and understand how and why LTM load balance it to one or another pool member.

References

https://devcentral.f5.com/wiki/advdesignconfig.F5WiresharkPlugin.ashx
https://devcentral.f5.com/questions/tcpdump-with-multiple-pool-members
SOL13637: Capturing internal TMM information with tcpdump

Wednesday, May 21, 2014

Simple MySQL and SQL exercises

How to create a sample MySQL data base and user

You can download an example data base sql file from here: http://www.mysqltutorial.org/mysql-sample-database.aspx. After unziping you should find following file:
 
rado2@ubuntu12-04:~$ ls -la mysqlsampledatabase.sql
-rw-rw-r-- 1 rado2 rado2 190711 May 23  2013 mysqlsampledatabase.sql
 
rado2@ubuntu12-04:~$ more mysqlsampledatabase.sql
/*
http://www.mysqltutorial.org
*/

CREATE DATABASE /*!32312 IF NOT EXISTS*/`classicmodels` /*!40100 DEFAULT CHARACTER SET latin1 */;

USE `classicmodels`;

/*Table structure for table `customers` */

DROP TABLE IF EXISTS `customers`;

CREATE TABLE `customers` (
  `customerNumber` int(11) NOT NULL,
  `customerName` varchar(50) NOT NULL,
  `contactLastName` varchar(50) NOT NULL,
  `contactFirstName` varchar(50) NOT NULL,
....

We don't want to use a root user to manipulate our data base records. To create a separate user you can run these commands:
 
$ mysql -u root -p
mysql> use information_schema;
mysql> CREATE USER 'rado2'@'localhost';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'rado2'@'localhost';
mysql> select * from USER_PRIVILEGES ;

To import and inspect the database we can use this commands:
 
$ mysql -u rado2 < mysqlsampledatabase.sql

$ mysql -u rado2
mysql> show databases;
mysql> show tables;
+-------------------------+
| Tables_in_classicmodels |
+-------------------------+
| customers               |
| employees               |
| offices                 |
| orderdetails            |
| orders                  |
| payments                |
| productlines            |
| products                |
+-------------------------+
8 rows in set (0.00 sec)

mysql> select * from employees  LIMIT 5;
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+----------------------+
| employeeNumber | lastName  | firstName | extension | email                           | officeCode | reportsTo | jobTitle             |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+----------------------+
|           1002 | Murphy    | Diane     | x5800     | dmurphy@classicmodelcars.com    | 1          |      NULL | President            |
|           1056 | Patterson | Mary      | x4611     | mpatterso@classicmodelcars.com  | 1          |      1002 | VP Sales             |
|           1076 | Firrelli  | Jeff      | x9273     | jfirrelli@classicmodelcars.com  | 1          |      1002 | VP Marketing         |
|           1088 | Patterson | William   | x4871     | wpatterson@classicmodelcars.com | 6          |      1056 | Sales Manager (APAC) |
|           1102 | Bondur    | Gerard    | x5408     | gbondur@classicmodelcars.com    | 4          |      1056 | Sale Manager (EMEA)  |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+----------------------+

mysql> select * from offices  LIMIT 5;
+------------+---------------+-----------------+--------------------------+--------------+------------+---------+------------+-----------+
| officeCode | city          | phone           | addressLine1             | addressLine2 | state      | country | postalCode | territory |
+------------+---------------+-----------------+--------------------------+--------------+------------+---------+------------+-----------+
| 1          | San Francisco | +1 650 219 4782 | 100 Market Street        | Suite 300    | CA         | USA     | 94080      | NA        |
| 2          | Boston        | +1 215 837 0825 | 1550 Court Place         | Suite 102    | MA         | USA     | 02107      | NA        |
| 3          | NYC           | +1 212 555 3000 | 523 East 53rd Street     | apt. 5A      | NY         | USA     | 10022      | NA        |
| 4          | Paris         | +33 14 723 4404 | 43 Rue Jouffroy D'abbans | NULL         | NULL       | France  | 75017      | EMEA      |
| 5          | Tokyo         | +81 33 224 5000 | 4-1 Kioicho              | NULL         | Chiyoda-Ku | Japan   | 102-8578   | Japan     |
+------------+---------------+-----------------+--------------------------+--------------+------------+---------+------------+-----------+

mysql> show COLUMNS FROM employees
+----------------+--------------+------+-----+---------+-------+
| Field          | Type         | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| employeeNumber | int(11)      | NO   | PRI | NULL    |       |
| lastName       | varchar(50)  | NO   |     | NULL    |       |
| firstName      | varchar(50)  | NO   |     | NULL    |       |
| extension      | varchar(10)  | NO   |     | NULL    |       |
| email          | varchar(100) | NO   |     | NULL    |       |
| officeCode     | varchar(10)  | NO   | MUL | NULL    |       |
| reportsTo      | int(11)      | YES  | MUL | NULL    |       |
| jobTitle       | varchar(50)  | NO   |     | NULL    |       |
+----------------+--------------+------+-----+---------+-------+

mysql> show COLUMNS FROM offices ;
+--------------+-------------+------+-----+---------+-------+
| Field        | Type        | Null | Key | Default | Extra |
+--------------+-------------+------+-----+---------+-------+
| officeCode   | varchar(10) | NO   | PRI | NULL    |       |
| city         | varchar(50) | NO   |     | NULL    |       |
| phone        | varchar(50) | NO   |     | NULL    |       |
| addressLine1 | varchar(50) | NO   |     | NULL    |       |
| addressLine2 | varchar(50) | YES  |     | NULL    |       |
| state        | varchar(50) | YES  |     | NULL    |       |
| country      | varchar(50) | NO   |     | NULL    |       |
| postalCode   | varchar(15) | NO   |     | NULL    |       |
| territory    | varchar(10) | NO   |     | NULL    |       |
+--------------+-------------+------+-----+---------+-------+
9 rows in set (0.00 sec)

Exercise 1: select all employees from offices in USA only
 
mysql> SELECT * FROM employees as e, offices as o  where e.officeCode = o.officeCode and o.country='USA';
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+--------------------+------------+---------------+-----------------+----------------------+--------------+-------+---------+------------+-----------+
| employeeNumber | lastName  | firstName | extension | email                           | officeCode | reportsTo | jobTitle           | officeCode | city          | phone           | addressLine1         | addressLine2 | state | country | postalCode | territory |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+--------------------+------------+---------------+-----------------+----------------------+--------------+-------+---------+------------+-----------+
|           1002 | Murphy    | Diane     | x5800     | dmurphy@classicmodelcars.com    | 1          |      NULL | President          | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1056 | Patterson | Mary      | x4611     | mpatterso@classicmodelcars.com  | 1          |      1002 | VP Sales           | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1076 | Firrelli  | Jeff      | x9273     | jfirrelli@classicmodelcars.com  | 1          |      1002 | VP Marketing       | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1143 | Bow       | Anthony   | x5428     | abow@classicmodelcars.com       | 1          |      1056 | Sales Manager (NA) | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1165 | Jennings  | Leslie    | x3291     | ljennings@classicmodelcars.com  | 1          |      1143 | Sales Rep          | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1166 | Thompson  | Leslie    | x4065     | lthompson@classicmodelcars.com  | 1          |      1143 | Sales Rep          | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1188 | Firrelli  | Julie     | x2173     | jfirrelli@classicmodelcars.com  | 2          |      1143 | Sales Rep          | 2          | Boston        | +1 215 837 0825 | 1550 Court Place     | Suite 102    | MA    | USA     | 02107      | NA        |
|           1216 | Patterson | Steve     | x4334     | spatterson@classicmodelcars.com | 2          |      1143 | Sales Rep          | 2          | Boston        | +1 215 837 0825 | 1550 Court Place     | Suite 102    | MA    | USA     | 02107      | NA        |
|           1286 | Tseng     | Foon Yue  | x2248     | ftseng@classicmodelcars.com     | 3          |      1143 | Sales Rep          | 3          | NYC           | +1 212 555 3000 | 523 East 53rd Street | apt. 5A      | NY    | USA     | 10022      | NA        |
|           1323 | Vanauf    | George    | x4102     | gvanauf@classicmodelcars.com    | 3          |      1143 | Sales Rep          | 3          | NYC           | +1 212 555 3000 | 523 East 53rd Street | apt. 5A      | NY    | USA     | 10022      | NA        |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+--------------------+------------+---------------+-----------------+----------------------+--------------+-------+---------+------------+-----------+
10 rows in set (0.00 sec)

References

http://www.mysqltutorial.org/mysql-sample-database.aspx
http://en.wikipedia.org/wiki/Join_%28SQL%29
https://answers.yahoo.com/question/index?qid=20080520200936AAmD1Mt
http://www.cyberciti.biz/tips/mysql-auto-completion-for-database-table-names.html

Tuesday, May 20, 2014

What is the difference between XenServer vs Xen vx XCP vs XAPI

It wasn't clear to me at fist what the differences are between XenServer, Xen and XCP. To make it even more confusing the documentation in many place were referring to XAPI and its importance in managing the hypervisors.

To understand what the XAPI is and how it can be used please take a look at this demo I wrote: How to install ipython on XenServer and test XAPI. As we can see the XAPI is an elegant way on top of the hypervisor itself that exposes some more advance API operation to help to control and managed the VM and hypervisor live cycle.

In a very simplistic way you can think of Xen as a 'hypervisor kernel'. The kernel itself may be difficult to use so we need some management software bundled with it.

It is similar comparing Linux kernel and a distribution together. It is hard to use the kernel on its own, we need a more user friendly tools to do this and this is the place where GNU toolchain is coming into play.

Once we understand this it is now easy to understand this FAQ: What's the difference between Xen hypervisor (from xen.org) and Citrix XenServer or XCP?

If you understood what the last link is about please take a look at these for more advance comparisons:
Here is an example showing the differences between the XenServer and Xen management cli:

How to install ipython on XenServer and test XAPI

We've been using the more user friendly shell to interact with python before: ipython. The example below are showing first how to install and enable EPEL repository to be able to install ipython. Next we are going to write a simple XAPI demo program.

Install ipython
  • Find the distro your XenServer is based on
cat /etc/issue.net
CentOS release 5.7 (Final)
Kernel \r on an \m
  • Check enabled repository 
yum repolist
  • From the EPEL install the relevant rpm packets that will add new repository to your yum
# http://fedoraproject.org/wiki/EPEL
# http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/5/i386/repoview/epel-release.html

rpm --force -i http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
  • Update the repo info
yum list 
yum list | grep ipython
  • Install ipython
yum install ipython.noarch
  • Start and verify that ipython is working fine
ipython

In [3]: import sys
In [4]: sys.version
Out[4]: '2.4.3 (#1, Sep 21 2011, 20:06:00) \n[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)]'

Xapi example using ipython

References

XAPI:
http://blogs.citrix.com/2011/05/18/so-what-is-xenserver-xapi/
http://docs.vmd.citrix.com/XenServer/6.2.0/1.0/en_gb/sdk.html#language_bindings-python

Packages:
http://xmodulo.com/2012/05/how-to-install-additional-packages-in.html
http://thomas-cokelaer.info/blog/2012/01/installing-repositories-under-centos-6-2-to-get-ipython-r-and-other-packages/
http://fedoraproject.org/wiki/EPEL#How_can_I_use_these_extra_packages.3F


Monday, May 19, 2014

Create a VM on an isolated network

For experimenting and testing we want to have a VM that is attached to an isolated network.
In the script below in the part 1) :
  • Take a clone of an existing VM
  • Create a new private network
  • Create and attached new interface to our VM
 Next in part 2) we configure statically an IP of 192.168.32.1 on this new interface.

References

http://blogs.citrix.com/2013/03/18/virtual-hypervisor/
https://wiki.debian.org/NetworkConfiguration
http://docs.vmd.citrix.com/XenServer/4.0.1/reference/ch03s02.html

Saturday, May 17, 2014

Using xe how to boot a VM in XenServer

After we have installed XenServer  there is a time to spin up an VM to test. Below are is a litle script that create a VM (code is based on: http://wiki.xen.org/wiki/Installing_Linux_on_Kronos ).

This create and starts the VM. You need to connect over the console port and follow the installer questions. We don't need anything media or ISO files. The installer will automatically download all necessary files.

Once your VM is created you can shutdown it, export it to XVA image. Later on we can restore the VM by simply importing it back. We can use the XVA file as well to create further more VMs for testing.

References

http://wiki.xen.org/wiki/Installing_Linux_on_Kronos
http://krypted.com/tag/list_domains

Docs on Citrix site:

http://docs.vmd.citrix.com/XenServer/6.2.0/1.0/en_gb/

XenServer 6.2.0 Technical FAQ
XenServer 6.2.0 Release Notes
XenServer Product Documentation
/

Linux Bash cheat sheet

I've spend some time googling for bash shortcats using phrases like: bash readline shortcat, copy and paste text to bash clipboard, etc  ... I always forget how to do this, especially when I don't work on Linux for a while.

Below is a list of my favorite (hard to remember) bash shortcats and tricks I like to use.

Bash shortcats

Ctrl + w  Cut the Word before the cursor to the clipboard
Ctrl + y  Paste the last thing to be cut (yank)
Alt + r  Cancel the changes and put back the line as it was in the history (revert).

Bash tricks to speed up typing 
  • How to copy the last command 
  • How to copy and paste the last command output
This one is my favorite because it allows me to refer to a previous command output text without having to copy and paste it with mouse.

# readline function
shell-expand-line (M-C-e)

Example 1:
$ myvar="/etc/passwd"
$ echo $myvar
$ ls $(echo $myvar)

Before you press enter press now (M-C-e) and the line will turn into

ls /etc/passwd-rrr

Example 2:
$ ls -l /etc/passwd
$ echo !!

Before you press enter press now (M-C-e) and the line will turn into

echo ls -l /etc/passwd

Example 3:

$ ls -l /etc/passwd
$ echo $(!!)

Before you press enter press now (M-C-e) and the line will turn into

echo -rw-r--r-- 1 root root 1399 May 17 02:19 /etc/passwd

References

http://ss64.com/bash/syntax-keyboard.html
http://superuser.com/questions/304519/how-to-copy-the-results-from-a-grep-command-to-the-bash-clipboard
http://superuser.com/questions/421463/why-does-ctrl-v-notpaste-in-bash-linux-shell
http://unix.stackexchange.com/questions/15850/how-to-use-keyboard-instead-of-mouse-middle-click-for-copy-paste
http://stackoverflow.com/questions/749544/pipe-to-from-clipboard
https://wiki.archlinux.org/index.php/Keyboard_Shortcuts
http://rtomaszewski.blogspot.co.uk/2013/06/linux-and-bash-cheat-sheet.html



XenServer installation over iKVM and redirected ISO CD-ROM option in JViewer

I've had a problem using my XenServer on VMware Workstation. I needed instead to install it on my dedicated server i have. Some of my notes:
  • Enabled iKVM on the server: How to enable IPMI settings in BIOS on Tyan S8225 motherboard.
  • In BIOS make sure you have enabled the VGA graphic output and disabled the graphics output on the PCI-X bus (as per  the link above)
  • The XenServer installer was not able to recognize my AHCI disk driver so I needed to enable a regular 'Native IDE' driver in BIOS (see screen shots below)
  • Disable java security permission in Windows (search for 'configure java' under Windows menu)
  • Mount the ISO image from the JViewer (java iKVM from ASUS) restart the server and follow the installation instructions :)
  • At the beginning of installation you will see that the phase 'Loading /install.img' is taking very long time. Don't panic, watch the network card stats and wait for the installer to start. It can take even up to 10min.


References

http://www.davethijssen.nl/2013/07/install-citrix-xenserver-62-from-usb.html

Wednesday, May 14, 2014

How to install XenServer on VMware Workstation in Windows

How to install XenServer 6.2 on VMware Workstation 8

Note: 17 May 2014:
The installation was fine. The XenServer boots fine in the VM. Unfortunately it hangs once in a while constantly. Couldn't find out what causes it. I've installed XenServer on hw instead here.

It is possible to install Type 1 hypervisor like XenServer within a virtualized environment like VMware Workstation by using nested virtualization technology. It may be shocking at first look because this is the software stack we are going to create:
  • Regular operating system, I'm using Windows 7
  • Install in Windows (Type 2) hypervisor VMware Workstation
  • Create a VM within VMware 
  • Install XenServer with in VM
  • Boot VM and run (Type 1) hypervisor XenServer
Installation steps
  • Create a new VM and chose guest: OS VMware ESX and version: VMware ESXi 5 (my config for the VM can be found here: vmware-workstation-xenserver.conf)
  • Download the XenServer-6.2.0-install-cd.iso 
  • For the VM chose to boot from the iso above
  • Boot the VM and follow the installation instruction
  • Installer at the end will reboot the VM
  • Before new boots power off VM and deselect the ISO file
  • Boot the VM and enjoy your XenServer 
  • You may want to install XenServer-6.2.0-XenCenter.msi to graphically manged your XenServer or use CLI over SSH
 XenServer 6.2 doesn't boot after installation on VMware Workstation 8

You may run into the following issues when attempting the installation procedure above

  • Wrong VM guest type. I've seen this initially when I try to use the type: Other, version: Other 64-bit.
BUG: recent printk recursion!
clocksource/1: Time went backwards
PCI: Bar 13: no parent found for of bridge



  • You can boot the VM and from the Xen boot loader select 'safe'. Unfortunately the XenServer will report this time I/O error.
ata2.01: qc timeout (cmd 0xa0)
ata2.01: TEST_UNIT_READY failed (err_mask=0x4)
Unhanded error code
Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
end_request: I/O error, dev sda, sector 0
buffer I/O error on device sda, logical block 0




References

http://bjtechnews.org/2013/07/01/how-to-install-citrix-xenserver-6-2-0-on-vmware-workstation-9-0/
http://discussions.citrix.com/topic/329733-xenserver-freeze-on-reboot/
http://discussions.citrix.com/topic/324048-virtual-machines-in-xenserver-6-on-wmware-workstation/
http://www.vi-tips.com/2011/10/how-to-run-xenserver-60-on-vsphere-5.html
http://vstorage.wordpress.com/2010/06/06/running-xenserver-5-6-on-vmware-workstation/


Sunday, April 27, 2014

Overlay technologies in data center

Everyone speaks about SDN an the benefits its brings when deploying cloud or enterprise infrastructures. But do we actually know or have any understanding what this all SDN is about? If you want be fluent in the language of virtual networking and network overlays in modern data centers you need to understand at least the following concepts:
In the remaining of the post we will concentrate solely on existing overlay technologies. These information was extracted from Cisco doc: Cisco Nexus 9000 Series Switches - Data Center Overlay Technologies).

Network-Based Overlay Networks
  1. IEEE 802.1ad Provider Bridging or IEEE 802.1q Tunneling also known as IEEE 802.1QinQ or simply Q-in-Q
  2. IEEE 802.1ah Provider Backbone Bridges (PBB) or Mac-in-Mac Tunnels
  3. Cisco FabricPath allows multipath networking at Layer 2
  4. TRILL - IETF Transparent Interconnection of Lots of Links is a Layer 2 multipathing technology
  5. Shortest-Path Bridging (SPB) is defined in IEEE 802.1aq and is targeted as a replacement for Spanning Tree Protocol (example info based on Avaya documentation)
  6. Cisco Overlay Transport Virtualization (OTV) is a Layer 2-over-Layer 3 encapsulation "MAC-in-IP" technology
  7. The Cisco Location/Identifier Separation Protocol (LISP) is currently defined as a Layer 3 overlay scheme over a Layer 3 network
  8. Multiprotocol Label Switching (MPLS)
  9. Virtual Private LAN Service (VPLS) a Layer 2 tunneling protocols
  10. Virtual Private Routed Network (VPRN) also known as BGP/MPLS or IP-VPN provides IP VPN services
Host-Based Overlay Networks
    1. Virtual Extensible LAN (VXLAN) is a Layer 2 overlay scheme over a Layer 3 networ that uses IP/UDP encapsulation
    2. Network Virtualization Using Generic Routing Encapsulation (NVGRE) allows creation of virtual Layer 2 topologies on top of a physical Layer 3 network
    3. Stateless transport tunneling (STT) is an overlay encapsulation scheme over Layer 3 networks that use a TCP-like header

    You can use bash shell instead of Cisco CLI on Nexus Switches

    Every one who works on Linux and understand how to efficiently use Bash hates to work with the limited Cisco IOS CLI. The design objectives standing behind this CLI haven't changed for the last 20 years or so. It is obvious that this tools lacks plenty of features expected from a modern shell for many people.

    But the evolution or even revolution that is happening in networking thanks to SDN is changing this terrible static network configuration landscape. The new generation of network devises like Cisco Nexus platform are going to support in the Cisco NX-OS :
    • Bash shell
    • Python shell 
    • API access
    • Linux containers for custom applications
    For these who still don't believe you can read about this here:

    References

    http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/white-paper-listing.html
    http://rtomaszewski.blogspot.co.uk/search/label/sdn
    http://rtomaszewski.blogspot.co.uk/2013/09/cisco-cheat-sheet.html

    Sunday, April 13, 2014

    Description and demonstration of the Heartbleed bug in OpenSSL

    There is a ton of posts on the Internet about the new bug in OpenSSL. I'm not going to repeat what others wrote  but rather give us a small demonstration.

    Heartbeat packet description in SSL protocol suite

    This is excellent blog posts we can take a look at the openssl code analysis and see where exactly the bug was hidden: Diagnosis of the OpenSSL Heartbleed Bug.

    If you want to learn more how to build an potential exploid you can read and watch this: http://security.stackexchange.com/questions/55116/how-exactly-does-the-openssl-tls-heartbeat-heartbleed-exploit-work

    A working code for a prof of concept can be found here:
    http://www.garage4hackers.com/entry.php?b=2551
    http://nakedsecurity.sophos.com/2014/04/08/anatomy-of-a-data-leak-bug-openssl-heartbleed/

    Demonstration

    How do I know if my site is vulnerable?

    There are potentially many different ways how you can test if a site is vulnerable. As two extreme examples (a) we could write a simple SSL client and try to sent an hearbeat packet (not so trivial and requires some knowledge about the ssl protocol itself) or (b) search for a site on Internet that do the testing for us. I would definitively avoid (b). These sites can store the URL you provided and try to exploit you later.

    A more simple and elegant solution can be built using openssl cli client tool instead. By running as single line script you can test if a server supports heartbeat or not. Next you have to find if the version of the OpenSSL you use is vulnerable.
     
    $ openssl s_client -connect www.cloudflarechallenge.com:443 -tlsextdebug
    CONNECTED(00000003)
    TLS server extension "renegotiation info" (id=65281), len=1
    0001 - <SPACES/NULS>
    TLS server extension "EC point formats" (id=11), len=4
    0000 - 03 00 01 02                                       ....
    TLS server extension "session ticket" (id=35), len=0
    TLS server extension "heartbeat" (id=15), len=1
    0000 - 01                                                .
    depth=4 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
    verify error:num=19:self signed certificate in certificate chain
    verify return:0
    ---
    Certificate chain
     0 s:/OU=Domain Control Validated/OU=Free SSL/CN=cloudflarechallenge.com
       i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
     1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
       i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO Certification Authority
     2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO Certification Authority
       i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC
     3 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC
       i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
     4 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
       i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    MIIFLTCCBBWgAwIBAgIQSkGkHc+NJGGLqUs9YZlcxDANBgkqhkiG9w0BAQUFADBy
    MQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD
    VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDEYMBYGA1UE
    AxMPRXNzZW50aWFsU1NMIENBMB4XDTE0MDQxMDAwMDAwMFoXDTE0MDcwOTIzNTk1
    OVowWDEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRhdGVkMREwDwYDVQQL
    EwhGcmVlIFNTTDEgMB4GA1UEAxMXY2xvdWRmbGFyZWNoYWxsZW5nZS5jb20wggEi
    MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCbQBaRWcPHl945y10L3tm2C+13
    bm4oqaGMIekvJyYTF7VGJFKX+EYgvt/wWD+qJTO1Wbm5dknVQbt3PP7061M2H6/b
    sG3M+xTfKK8d6/AAHWZMy0/ps+5cGPOzFFwL3JVwEFakoExGc3jT6S9RlhU5q4I+
    q8Qd+jpHL7uKeklipCb8VIznRmtGKYI7H01kjyW8gwXYOrWKlKCHOIcR32LIxHfd
    fv72QjT2kGupne3TmXAY+6cEL12ZqS2HCYpGBa8QQaZ7/dggc1X5OJL1yrQP8Le9
    /faCOBHn0A4yzNp873BVMQ+7T+7k2PCSs7qAfB0TdvdfQFiPPFaTODDtPWClAgMB
    AAGjggHXMIIB0zAfBgNVHSMEGDAWgBTay+qtWwhdzP/8JlTOSeVVxjj0+DAdBgNV
    HQ4EFgQUbqyvF2sHtsjg5i82wBON35elvNQwDgYDVR0PAQH/BAQDAgWgMAwGA1Ud
    EwEB/wQCMAAwNAYDVR0lBC0wKwYIKwYBBQUHAwEGCCsGAQUFBwMCBgorBgEEAYI3
    CgMDBglghkgBhvhCBAEwTwYDVR0gBEgwRjA6BgsrBgEEAbIxAQICBzArMCkGCCsG
    AQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8uY29tL0NQUzAIBgZngQwBAgEw
    OwYDVR0fBDQwMjAwoC6gLIYqaHR0cDovL2NybC5jb21vZG9jYS5jb20vRXNzZW50
    aWFsU1NMQ0EuY3JsMG4GCCsGAQUFBwEBBGIwYDA4BggrBgEFBQcwAoYsaHR0cDov
    L2NydC5jb21vZG9jYS5jb20vRXNzZW50aWFsU1NMQ0FfMi5jcnQwJAYIKwYBBQUH
    MAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTA/BgNVHREEODA2ghdjbG91ZGZs
    YXJlY2hhbGxlbmdlLmNvbYIbd3d3LmNsb3VkZmxhcmVjaGFsbGVuZ2UuY29tMA0G
    CSqGSIb3DQEBBQUAA4IBAQBlN1564xpz0f0EnCh5dKOjo6uk+kbLzEhkfaGd5Ydi
    4diFQ9VYx3+Le1JCB/bDHMVUfwlqTpV0Eq8DZIWTO5wnP9BlRDiljVe7+y/jkQ/b
    /B88kmBr2jjR9Aet1l8hOrqJycw6Ack6F+5hd/lYIvZ/0YH+h/qu9/Z6ii6rcUCd
    UWERSKiTFsbM8PRmG/Cwb4Jm52N8ev6mcVYmxeBYIPmf51HBHEakN13oQcubCAjd
    V9/8CugEMrl56lUpt7BYZMET2h4NyCDrfTlbFcDqQC+YBr5dLDOvLpe7T7Dv+r1P
    wYJ+R0A4JC0F2RdUeIBWC5CycJcTx4h7ZSlNeWtFrZgJ
    -----END CERTIFICATE-----
    subject=/OU=Domain Control Validated/OU=Free SSL/CN=cloudflarechallenge.com
    issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 6784 bytes and written 376 bytes
    ---
    New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
    Server public key is 2048 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : TLSv1.1
        Cipher    : ECDHE-RSA-AES256-SHA
        Session-ID: EF16DB45C3D67F69A480645C5267C4FDC44F41FD4CF4911194E986FC21E72F62
        Session-ID-ctx:
        Master-Key: 9DF3223AAF1520D6437E643E83E4AD5B1A590776F375B7ED082E024F3EC9EB43617A0D1F7715DF299EA483F905095465
        Key-Arg   : None
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        TLS session ticket lifetime hint: 600 (seconds)
        TLS session ticket:
        0000 - c5 00 41 79 f6 38 12 30-bf 5f 85 54 f7 93 09 1c   ..Ay.8.0._.T....
        0010 - c1 60 e2 23 ca 90 8f 17-0c 4a 9f db cc 40 0e ea   .`.#.....J...@..
        0020 - 55 b0 f8 49 f1 7e b0 4e-78 0f 36 4a 58 3a 60 e2   U..I.~.Nx.6JX:`.
        0030 - b4 2b 22 a2 49 e8 c5 42-d0 00 ad a6 ec 49 b3 4d   .+".I..B.....I.M
        0040 - 28 b1 c3 ad 03 c6 53 de-a3 e7 ec c8 aa ed 5e 97   (.....S.......^.
        0050 - 75 12 5e 9f 5f eb cf a9-4a ab b7 85 bf cd e0 12   u.^._...J.......
        0060 - 2c ec 0b 05 4f cf ac 16-e9 65 40 1b a8 60 dc 3a   ,...O....e@..`.:
        0070 - 99 a0 cf 7a 65 0b 4c 74-a5 fc a5 16 11 48 e2 94   ...ze.Lt.....H..
        0080 - 19 0e 17 a8 03 d0 d0 4b-a4 14 7e 49 05 75 36 65   .......K..~I.u6e
        0090 - d4 70 63 fa a7 92 5a 14-63 97 00 cf 6b 5b 45 36   .pc...Z.c...k[E6
    
        Start Time: 1397426832
        Timeout   : 300 (sec)
        Verify return code: 19 (self signed certificate in certificate chain)
    ---
    GET /heartbleed HTTP/1.1
    Host: www.cloudflarechallenge.com
    
    HTTP/1.1 200 OK
    Server: nginx
    Date: Sun, 13 Apr 2014 22:02:32 GMT
    Content-Type: text/html; charset=utf-8
    Transfer-Encoding: chunked
    Connection: keep-alive
    Vary: Accept-Encoding
    Strict-Transport-Security: max-age=86400
    
    f61
    <!doctype html>
    <html>
    <head>
      <title>Heartbleed Challenge</title>
    

    From the output we can see that:
    • We connect to the server
    • There are many packages exchange between the client (our openssl cli tool) and the web server; the packets types and formats are defined in the relevant RFC documents for SSL/TLS
    • Option tlsextdebug instructs openssl to print out TLS extensions the server supports
    • We can immediately see if the option is supported by our www server; what we have o do next is to check if the version of OpenSSL that we run is vulnerable or not 
    • It is important to note that regardless if the www server supports the heartbeat extension or not you as a client can sent any legitimate HTTP requests; the whole problem is that if your client sent an heartbeat packet that was on purpose malicious the server in its response can reveal a lot more data that it should.
    References

    http://www.openssl.org/docs/apps/s_client.html
    http://www.theregister.co.uk/2014/04/09/heartbleed_explained/
    https://www.cloudflarechallenge.com/heartbleed


    Tuesday, April 8, 2014

    Can I use Shortest Path Bridging hardware to build my SDN network

    Recently I've come across a document that compares a number of existing network overlays in SDN architecture: The 2013 Guide to Network Visualization and SDN.

    What is new and interesting is the solution from Avaya. Instead of using VXLAN, STT and GRE like all other vendors they use SPB (we wrote about this here How does switch fabric network work) to build the SDN solution.

    How does switch fabric network work

    A network engineer can list a number of issues you can potentially run when using STP protocol  in your switch network. Over the years the network industry has created successor protocols like RSTP or MSTP. Both are improvements and offer much better convergence time and respond much quicker to switch topology changes. One of the major disadvantages for networks that relay on STP is the fact that they don't support multipathing. It means once network topology converges there will be blocked path between switches that are elected and managed by STP. This often redundant links can't be used because of a loop risk.

    But there are better solutions today on the market to design better layer 2 Ethernet networks (more scalable, with higher throughput and with active link redundancy as an example). The 2 most popular are based on SPB and TRILL protocols. Both of them are used as a foundation in switch fabrics products. To better understand both of them the pictures below provide a side by side comparison. This was taken from Avaya document: Compare and Contrast SPB and TRILL.

    Avaya is a SPB promoted so the comparison is a bit waited towards SPB but nevertheless it gives some inside view into both protocols.



    References

    http://cciethebeginning.wordpress.com/2008/11/20/differences-between-stp-and-rstp/
    http://etherealmind.com/spb-attention/
    http://en.wikipedia.org/wiki/IEEE_802.1aq
    http://en.wikipedia.org/wiki/TRILL_(computing)
    http://www.avaya.com/uk/resource/assets/whitepapers/SPB-TRILL_Compare_Contrast-DN4634.pdf
    http://nanog.org/meetings/nanog50/presentations/Monday/NANOG50.Talk63.NANOG50_TRILL-SPB-Debate-Roisman.pdf
    http://www.ebrahma.com/2012/06/trill-vs-spb-similarities-differences/
    http://wikibon.org/wiki/v/Network_Fabrics,_L2_Multipath_and_L3

    Monday, March 31, 2014

    How to list numbers next to ACL rules on Cisco

    How to list numbers next to the ACL rules on Cisco

    sh  access-list outside-acl | e \ \
    access-list 101; 86 elements; name hash: 0xe7d586b5
    access-list 101 line 1 extended permit ip object-group WHITELIST-IPS any 0xc4d2a54e
    access-list 101 line 2 extended permit icmp any any object-group ICMP-ALLOWED (hitcnt=576916) 0x994c9516
    access-list 101 line 3 extended deny ip any host 192.168.199.254 (hitcnt=31708) 0x8e8cc2a6
    access-list 101 line 5 remark !*!*!*!*!*!*!*!*!*!
    access-list 101 line 6 remark RULES CONTROLLED BY AUTOMATION
    access-list 101 line 7 remark !*!*!*!*!*!*!*!*!*!
    access-list 101 line 8 extended permit ip host 1.1.1.1 host 10.179.72.125 (hitcnt=0) 0xa9809ff7
    access-list 101 line 9 extended permit ip any host 10.179.72.125 (hitcnt=0) 0xa9809ff7
    

    Sunday, March 30, 2014

    How to automatically prefill command on the Linux bash

    Linux Bash is one of the most famous Linux shells. It offers a great number of features like for example spawning and controlling process, redirecting streams, supporting scripts and a flexible way to control you editing line.

    Problem

    How to automatically pre-populate a command on the shell after prompt.

    Solution description

    The shell has tree default streams: stdout, stdin and stderr. By manipulating the stdin of the process we can simulate typing a command.

    Reference implementation

    The original script can be found here: https://github.com/rtomaszewski/experiments/blob/master/type-command.c

    Demonstration
    • Compile first the program
    gcc -o type-command type-command.c
    • Run for the firs time
    # ./type-command
    type-command: the variable TYPE_CMD_ENABLED is not set, set it to 'no' to surpress this message; set the TYPE_CMD_TYPE for the command to type
    
    Example: export TYPE_CMD_ENABLED=yes; export TYPE_CMD_TYPE=date
    • Export the variable to controls if the program should try to type a command or not
    # export TYPE_CMD_ENABLED=yes
    # ./type-command
    #
    • Specify the command that you wish to be typed
    # export TYPE_CMD_ENABLED=yes; export TYPE_CMD_TYPE=date
    # ./type-command
    # date
    Sun Mar 30 19:27:55 UTC 2014>
    

    References

    http://stackoverflow.com/questions/10866005/bash-how-to-prefill-command-line-input
    http://stackoverflow.com/questions/11198603/inject-keystroke-to-different-process-using-bash
    http://unix.stackexchange.com/questions/48103/construct-a-command-by-putting-a-string-into-a-tty

    http://fossies.org/linux/misc/old/console-tools-0.3.3.tar.gz%3at/console-tools-0.3.3/vttools/writevt.c

    http://man7.org/linux/man-pages/man4/tty_ioctl.4.html
    http://man7.org/linux/man-pages/man3/tcflush.3.html
    http://www.tldp.org/LDP/lpg/node143.html

    Saturday, March 29, 2014

    How to create a sequence of replace commands to change your file

    Use existing plugin: RegReplace

    We could write a custom plugin using the Sublime API or try to use a plugin that promises to offer this functionality already: https://github.com/facelessuser/RegReplace

    Demonstration

    We have a following structured but not consistently formatted data that we would like to adjust so it is easier toread and work with.



    To reformat the text we can use the above plugin and define a series of regex that match and modify text.
    • Installed RegReplace plugin.
    • Create a reg_replace.sublime-settings in your Sublime2\Data\Packages\User\ directory and define the regex commands we want to use.
    {
        "replacements": {
            // add teh .<digit> when is missing
            "ig_order_add_dot_digit": {
                "find": "([0-9][0-9]) at",
                "replace": "\\1.0 at"
    //            "greedy": true,
    //            "case": false
            },
            "ig_order_add_dot_digit2": {
                "find": "([0-9][0-9]) *- ",
                "replace": "\\1.0 - ",
                "greedy": true
            },
            "ig_order_fix_spaces": {
                "find": "/(201[0-9]) *",
                "replace": "/\\1 "
            },
            "ig_order_fix_spaces2": {
                "find": "-   -    -  ",
                "replace": "-    -    -     "
            },
            "ig_order_change_android_str": {
                "find": "AndroidApp",
                "replace": "AndrAp"
            },
            "ig_order_remove_str": {
                "find": "/s ",
                "replace": " ",
                "greedy": true
            },
            "ig_order_fix_header": {
                "find": "(Date) *(Time) *(Activity) *(Market) *(Period) *(Channel) *(Currency) *(Size) *(Level) *(Stop) *(Type) *(Limit) *(Result)",
                "replace": "Date        Time    Activity Market                                               Period              Channel Cur Size Level  Stop Type Limit Result",
                "greedy": true
            },
    
    
    
            "ig_transactions_fix_header": {
                "find": "(Type) *(Date) *(Ref) *(Market) *(Period) *(Opening) *(Ccy) *(Size) *(Closing) *(P/L)",
                "replace": "Type    Date        Ref         Market                                                  Period            Opening Ccy Size    Closing P/L",
                "greedy": true
            },
           "ig_transactions_add_dot_digit": {
                "find": "([0-9][0-9]) +£",
                "replace": "\\1.0 £"
            },
            "ig_transactions_add_dot_digit2": {
                "find": "(£ +.*\\..* +)([0-9]+) +",
                "replace": "\\1\\2.0 "
            },
            "ig_transactions_fix_plus_minus_sign": {
                "find": "([0-9]+\\.[0-9]+ +[0-9]+\\.[0-9]+ +)([0-9]+\\.[0-9]+)",
                "replace": "\\1 \\2"
            }
    
        }
    }
    • Define the final  regex command to run and associate a a keyboard short in Default (Windows).sublime-keymap file
    [
    { 
        {
            "keys": ["alt+ctrl+t"],
            "command": "reg_replace",
            "args": {"replacements": [
                                        // orders
                                        "ig_order_add_dot_digit",
                                        "ig_order_add_dot_digit2",
                                        "ig_order_fix_spaces",
                                        "ig_order_fix_spaces2",
                                        "ig_order_change_android_str",
                                        "ig_order_remove_str",
                                        "ig_order_fix_header",
    
                                        // transactions
                                        "ig_transactions_fix_header",
                                        "ig_transactions_add_dot_digit",
                                        "ig_transactions_add_dot_digit2",
                                        "ig_transactions_fix_plus_minus_sign"
    
    
                                    ],  "find_only": true}
        }
    ]
    • When you activate the regex chain command it will first show what part of the file are going to be changed
    • Accept the "yes" option at the bottom and reformat the file

    How to write a plugin for Sublime editor

    Below is a list of links for Sublime API and Sublime commands if you want to write a custom plugins.

    Sublime API

    https://www.sublimetext.com/docs/api-reference
    https://www.sublimetext.com/docs/2/api_reference.html

    Commands

    http://sublimetext.info/docs/en/core/commands.html
    http://www.sublimetext.com/docs/commands

    Debug best practices

    Once you follow the steps below everything you do in the editor will be logged on the console.
    • Open Sublime console: Ctrl+~
    • Enable verbose and debug within the editor
    sublime.log_commands(True)
    sublime.log_input(True)
    • Example commands to try on the console 
    view.run_command("goto_line", {"line": 7})
    view.window().run_command("show_minimap", {"key": True})
    



    Friday, March 14, 2014

    Interface redundancy on the host with TCP Multipath

    TCP and UDP protocols are used exchange data between hosts. They have been used for a decade or longer and are very well documented how they work.

    Everyone knows the problem that when you lost your active link on the server all your TCP sessions are going to die as well. Let's say your server has 2 active interfaces. There is no way to move/migrate a TCP session to use another active interface (by default). The other link can't be used automatically as a fail back mechanism.

    There are couple of reasons behind why it isn't to works, the simplest one is that the new link used a different IP address. Even if the Linux kernel would start using the new interface and start sending IP/TCP packets sourced with the new IP address these packets wouldn't be recognized on the remote site. The remote site expect tcp segments from one and only one IP source.

    Problem

    How to provide a link level redundancy on the server to keep a TCP session alive even if one interface experience an error.

    Analysis and solution Demonstration

    The problem could be see as a more generic issue: how to implement multihoming or link redundancy. There are couple of working solution out there. The simplest example:
    • Link bonding(link aggregation) on the server; requires support and proper configuration on the switch and the server
    We will look at another one: TCP Multipath. What is cool about this is that it is transparent to your application. It visualizes a session and provide a single TCP session to the application that can benefit from built-in multipath redundancy on the kernel level.


    References

    http://multipath-tcp.org/
    Decoupled from IP, TCP is at last able to support multihomed hosts
    https://devcentral.f5.com/articles/multipath-tcp-mptcp
    https://devcentral.f5.com/articles/the-evolution-of-tcp


    Saturday, March 8, 2014

    How to build a high performance network appliance like routers using commodity hardware and off the shelf components

    You can assemble a server from off the shelf components that will be able to sent and receive traffic in multi Gigabit speed. Here is an example of an 10Gps net card from Intel.

    But can we turn this server into a high performance network appliance? Do we still need a dedicated hardware like for example ASIC, FPGA, low latency RAM and TCAM RAM in network devise so they can efficiently switch and forward packets with maximum wire speed.

    Router hardware design plan

    Looking at this presentation from 2012 https://ripe64.ripe.net/presentations/18-ripe-64-router-architecture-challenges.pdf you would think that yes. These would be the obvious reasons (screenshots taken from the presentation):





    Network processing unit (NPU) and new hardware design

    The key points listed above still hold. But the next generation network appliances will be rather build with a help of a multicore generic NPU using the power of parallel processing than expensive and purposely design ASIC. With the right software (OS - often Linux, drivers, firmware, SDK, and API libraries) you will be able to turn a conventional x86 server with a modern PCIe data bus into a high performance, low latency and high speed network appliance.

    Netronome Network Cards Accelerate SDN and NFV Designs
    100Gps FlowNIC-6xxx network card
    Hardware reference designs for FlowProcessor NPU chips



    Sunday, March 2, 2014

    How to do URL based load balancing on F5

    There are many load balancers out there. Some of them offer a great flexibility to control the traffic by allowing a user to upload a custom script that implement the load balancing algorithm to solve a particular problem.

    Problem

    How to do HTTP URL based load balancing on F5.

    Solution and demonstration

    This is an iRule script that inspects the HTTP GET URL string to decided where to load balance it: https://github.com/rtomaszewski/f5/blob/master/lb-based-on-url.tcl.

    Create default pool

    Create VIP

    Create custom pools

    Testing

    To verify that our iRule is working properly we can enable debugging by changing the iRule variable DEBUG to 1.

    Next we can simulate traffic

    curl -v http://vip/
    curl -v http://vip/url1
    curl -v http://vip/url2
    curl -v http://vip/url3

    And watch the logs on the lb.

    tail -f /var/log/ltm

    Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /
    Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80
    Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url1
    Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url1
    Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url2
    Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url2
    Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url3
    Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url3
    
    


    Reference

    https://devcentral.f5.com/wiki/iRules.HomePage.ashx

    Monday, February 24, 2014

    Dirty trick how to analysis ASA performance based on interface overruns and underruns


    There are number of firewall vendors on the market you can chose from (other links to Gartner magic quadrant for firewalls here and here). Every vendor has a product line ranging from the low to high end firewalls. An example product list for Cisco ASA can be seen here: http://www.cisco.com/c/en/us/products/security/asa-5500-series-next-generation-firewalls/models-comparison.html#~tab-b

    Problem

    I can see in my firewalls interface stats underruns and overrruns and the counters increase.

    Solution

    This is rather a dirty trick and your monitoring system should be able to graph the interface stats. But if you are in a position like me where you have no visibility to interface statistics like you could have in Zenoss, Cacti, Zabbix or other monitoring system you may need to manually check this...
    • We need to first start collecting data so we can look at it later.
    Run at least one a day the command and save in a file 1.txt, 2.txt, etc.

    sh clock
    sh int
    • After some time you should have a collection of files 
    $ ls -1 *.txt
    1.txt
    2.txt
    3.txt
    4.1.txt
    4.2.txt
    5.1.txt
    6.1.txt
    8.txt
    

    bash asa-interfaces.sh
    

    Base on the files you collected it will generate stats for every interface (time stamp is in the last column). Example output:

    Interface GigabitEthernet0/0 "outside", is up, line protocol is up
            31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
            719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            734 input errors, 0 CRC, 0 frame, 734 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            1502 input errors, 0 CRC, 0 frame, 1502 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            1794 input errors, 0 CRC, 0 frame, 1794 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            1881 input errors, 0 CRC, 0 frame, 1881 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            1921 input errors, 0 CRC, 0 frame, 1921 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            1971 input errors, 0 CRC, 0 frame, 1971 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
    Interface Management0/0 "lan-failover", is up, line protocol is up
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet1/0 "inside", is up, line protocol is up
            364 input errors, 0 CRC, 0 frame, 364 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            382 input errors, 0 CRC, 0 frame, 382 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            392 input errors, 0 CRC, 0 frame, 392 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            468 input errors, 0 CRC, 0 frame, 468 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            707 input errors, 0 CRC, 0 frame, 707 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            756 input errors, 0 CRC, 0 frame, 756 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet1/1 "app2", is up, line protocol is up
            640 input errors, 0 CRC, 0 frame, 640 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            658 input errors, 0 CRC, 0 frame, 658 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            683 input errors, 0 CRC, 0 frame, 683 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            797 input errors, 0 CRC, 0 frame, 797 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            811 input errors, 0 CRC, 0 frame, 811 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            863 input errors, 0 CRC, 0 frame, 863 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            984 input errors, 0 CRC, 0 frame, 984 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            1052 input errors, 0 CRC, 0 frame, 1052 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
            0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014



    Interface GigabitEthernet0/0 "outside", is up, line protocol is up
            646983182 packets output, 473597063148 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
            700155558 packets output, 509814505730 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
            753341661 packets output, 546026853810 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
            1025937535 packets output, 734304301602 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
            1054530605 packets output, 761409276094 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
            1105491616 packets output, 798565630885 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
            1264871240 packets output, 907739984962 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
            1315876113 packets output, 943680519398 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
            985243431 packets output, 309823858329 bytes, 459 underruns 07:48:45.631 cst Wed Feb 12 2014
            1070533450 packets output, 336205856058 bytes, 459 underruns 05:38:08.573 cst Thu Feb 13 2014
            1159894277 packets output, 366047579951 bytes, 483 underruns 03:37:26.853 cst Fri Feb 14 2014
            1596471490 packets output, 500836893219 bytes, 483 underruns 03:52:52.523 cst Wed Feb 19 2014
            1635530484 packets output, 511489408071 bytes, 483 underruns 10:03:08.799 cst Wed Feb 19 2014
            1722164227 packets output, 536769375853 bytes, 483 underruns 06:11:22.244 cst Thu Feb 20 2014
            2032554621 packets output, 636075162304 bytes, 2831 underruns 08:35:21.315 cst Sun Feb 23 2014
            2174454722 packets output, 688313076839 bytes, 2831 underruns 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
            1968362 packets output, 524301440 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
            1987058 packets output, 526612914 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
            2005883 packets output, 528940672 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
            4036852 packets output, 3167775775 bytes, 2831 underruns 03:52:52.523 cst Wed Feb 19 2014
            13676338 packets output, 15853359823 bytes, 2831 underruns 10:03:08.799 cst Wed Feb 19 2014
            23861856 packets output, 16649050850 bytes, 3052 underruns 06:11:22.244 cst Thu Feb 20 2014
            66743830 packets output, 20187731129 bytes, 5941 underruns 08:35:21.315 cst Sun Feb 23 2014
            80290600 packets output, 21286340673 bytes, 6860 underruns 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
            16582048 packets output, 17699836232 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
            17971640 packets output, 19234649922 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
            19380417 packets output, 20791969660 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
            26970739 packets output, 29162172960 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
            27259841 packets output, 29471830004 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
            29612954 packets output, 32141440890 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
            39077736 packets output, 42912691094 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
            42074827 packets output, 46322035220 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
    Interface Management0/0 "lan-failover", is up, line protocol is up
            1863787 packets output, 265441230 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
            1977505 packets output, 281732398 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
            2091970 packets output, 298145244 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
            2718733 packets output, 387975068 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
            2750523 packets output, 392530668 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
            2855417 packets output, 407567752 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
            3242848 packets output, 463113612 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
            3366749 packets output, 480878922 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet1/0 "inside", is up, line protocol is up
            229534738 packets output, 55157234973 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
            249682890 packets output, 59948227086 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
            272763726 packets output, 66350185657 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
            378020447 packets output, 91307448807 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
            384704374 packets output, 93165635304 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
            402556578 packets output, 97469565455 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
            492798902 packets output, 119550853698 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
            564346002 packets output, 137603523999 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet1/1 "app2", is up, line protocol is up
            142287604 packets output, 56966204294 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
            154809474 packets output, 62049309926 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
            167733332 packets output, 67152657884 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
            231962627 packets output, 93689642614 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
            235640974 packets output, 95384548398 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
            249769631 packets output, 103735197461 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
            290301462 packets output, 119748550482 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
            303003248 packets output, 125098088305 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
    Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
            0 packets output, 0 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
            0 packets output, 0 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
            0 packets output, 0 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
            0 packets output, 0 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
            0 packets output, 0 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
            0 packets output, 0 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
            0 packets output, 0 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
            0 packets output, 0 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014

    Once you have the data in-front of you can easily see how the stats were changing over time, over longer period of time like a week.

    In my case we suspected that the Firewall hit the capacity limit but further investigation confirmed that the device is doing well and no upgrade is necessary.

    References

    http://www.gossamer-threads.com/lists/cisco/nsp/152428
    http://ccna2ccnp.blogspot.co.uk/2012/12/ciscoasa-oversubcription-maximizing.html
    http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1904.html
    http://en.wikipedia.org/wiki/Buffer_underrun
    http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115985-asa-overrun-product-tech-note-00.html