Search This Blog

Saturday, March 30, 2013

How to host 2 ssl sites on a single public IP on F5 load balancer

As the pool of free IPv4 is getting lower it is important to efficiently mange our existing assigned public IPs. One thing that can be used is to use SSL offloading and hosting multiple HTTPS sites on single public IP using wild cart certificates.

The core of this solution is the configuration of the lb and the vhost setting on your server. The server reads the HTTP Host header and base on it decide what site the user try to access.

Problem

How to configure servers and load balancer to host 2 ssl sites on a single public IP.

Analisis and example configuration
  • Example F5 configuration
For SSL to work we need a wild card certificate.
We have to import the certificate and key to F5 and create client site ssl profile.

Example wild cart certificate:
 
# openssl x509 -in /var/tmp/wildcard.rado.net.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:9c:4a:4b:11:11:bc
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certificates.godaddy.com/repository, CN=Go Daddy Secure Certification Authority/serialNumber=111111
        Validity
            Not Before: Nov 21 11:57:16 2012 GMT
            Not After : Nov 20 09:31:37 2013 GMT
        Subject: O=*.rado.net, OU=Domain Control Validated, CN=*.rado.net
        ....

SSL profile:
 
# tmsh list ltm profile client-ssl  wildcard-client-ssl-profile
ltm profile client-ssl wildcard-client-ssl-profile {
    cert wildcard-rado.net.crt
    chain CA.crt
    defaults-from clientssl
    key wildcard-rado.net.key
}

Virtual server to terminate and load balance the traffic:
 
# tmsh list ltm virtual VS-1.1.1.1-443
ltm virtual VS-1.1.1.1-443 {
    destination 192.168.99.68:https
    ip-protocol tcp
    mask 255.255.255.255a
    pool POOL-192.168.99.68-80
    profiles {
        http { }
        tcp { }
        wildcard-client-ssl-profile {
            context clientside
        }
    }
}

Servers where the traffic is going to be load balanced:
 
# tmsh list ltm pool POOL-192.168.99.68-80
ltm pool POOL-192.168.99.68-80 {
    load-balancing-mode least-connections-member
    members {
        10.177.1.1:http {
            session monitor-enabled
        }
        10.177.1.2:http {
            session monitor-enabled
        }
    }
    monitor http
}
  • Apache configuration on one of the servers

    This is a simple example configuration for the http server.
     
    # cat vhost1.conf
    <VirtualHost *:80>
            ServerName vhost1.rado.net
            DocumentRoot /var/www/vhost1
    </VirtualHost>
    
    # cat vhost2.conf
    <VirtualHost *:80>
            ServerName vhost2.rado.net
            DocumentRoot /var/www/vhost2
    </VirtualHost>
    
    # cat /var/www/vhost1/index.html
    <html><body><h1>It works!</h1>
    <p>
    This is VHOST 1
    </p>
    </body></html>
    
    # a2ensite vhost1.conf 
    # a2ensite vhost2.conf 
    
    # service apache2 reload
    

    A quick local test on the server will confirm if the config is correct.
     
    # curl -v -H "Host: vhost1.rado.net" http://5.1.1.1
    * About to connect() to 5.1.1.1 port 80 (#0)
    *   Trying 5.1.1.1... connected
    > GET / HTTP/1.1
    > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
    > Accept: */*
    > Host: vhost1.rado.net
    >
    < HTTP/1.1 200 OK
    < Date: Sat, 30 Mar 2013 23:42:20 GMT
    < Server: Apache/2.2.22 (Ubuntu)
    < Last-Modified: Sat, 30 Mar 2013 23:19:21 GMT
    < ETag: "b46bb-47-4d92c9e17f040"
    < Accept-Ranges: bytes
    < Content-Length: 71
    < Vary: Accept-Encoding
    < Content-Type: text/html
    <
    <html><body><h1>It works!</h1>
    <p>
    This is VHOST 1
    </p>
    </body></html>
    * Connection #0 to host 5.79.21.166 left intact
    * Closing connection #0
    

How to run Sublime editor on remote cloud Linux server

Sublime is a graphical text editor that uses GTK libraries on Linux. If you would like to run it from your remote cloud servers session please follow the instruction below.
  1. Install the Sublime locally on your cloud server 
  2. Check what libraries you may be missing 

  3. The listing below shows that any libraries are missing.

     root@server:~/Sublime Text 2# ldd ./sublime_text
            linux-vdso.so.1 =>  (0x00007fff7d1ff000)
            libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f85d64d9000)
            librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f85d62d1000)
            libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f85d5fda000)
            libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f85d5dbd000)
            libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f85d5bb9000)
            libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f85d59b5000)
            libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f85d567d000)
            libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f85d5376000)
            libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f85d50f1000)
            libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f85d4edb000)
            libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f85d4b3c000)
            libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0 (0x00007f85d4936000)
            libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f85d472e000)
            libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f85d44f2000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f85d6731000)
            libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f85d42d5000)
            libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f85d40d2000)
            libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f85d3ecb000)
    

  4. Try to execute it and it should complain about some libraries now that are loaded dynamically after start.  
  5.  
    root@server:~/Sublime Text 2# ./sublime_text
    Unable to load libgdk-x11-2.0.so
    Unable to load gdk_cairo_create from libgdk-x11-2.0.so
    Unable to load gdk_cursor_new_for_display from libgdk-x11-2.0.so
    Unable to load gdk_cursor_unref from libgdk-x11-2.0.so
    Unable to load gdk_error_trap_pop from libgdk-x11-2.0.so
    Unable to load gdk_error_trap_push from libgdk-x11-2.0.so
    ...
    Unable to load gtk_window_unfullscreen from libgtk-x11-2.0.so
    Unable to load pango_font_description_free from libpango-1.0.so
    Unable to load pango_font_description_new from libpango-1.0.so
    Unable to load pango_font_description_set_family from libpango-1.0.so
    Unable to load pango_font_description_set_size from libpango-1.0.so
    Unable to load pango_cairo_show_layout from libpangocairo-1.0.so
    Unable to load pango_cairo_update_layout from libpangocairo-1.0.so
    Unable to load all required GTK functions
    Unable to init px
    

  6. Let's find the missing library file and install the necessary package   
  7.  
    root@server:~/Sublime Text 2# aptitude install apt-file
    root@server:~/Sublime Text 2# apt-file update
    root@server:~/Sublime Text 2# apt-file search libgdk-x11-2.0.so
    ia32-libs: /usr/lib32/libgdk-x11-2.0.so
    ia32-libs: /usr/lib32/libgdk-x11-2.0.so.0
    ia32-libs: /usr/lib32/libgdk-x11-2.0.so.0.2400.6
    libgtk2.0-0: /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
    libgtk2.0-0: /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0.2400.6
    libgtk2.0-0-dbg: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0.2400.6
    libgtk2.0-dev: /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so
    lsb-build-desktop3: /usr/lib/lsb3/libgdk-x11-2.0.so
    
    root@server:~/Sublime Text 2# aptitude install libgtk2.0-0
    
    root@server:~/Sublime Text 2# ./sublime_text
    (sublime_text:14069): Gtk-WARNING **: cannot open display:
    

  8. We need to enable X11 forwarding and set up a X11 server where the application output can be displayed

  9. If you connecting from Linux you need to use the -X option for ssh client.
    If you using Windows putty program you need to enable the X11 forwarding at Connection-SSH-X11-"Enable X11 forwarding". For the "X display location" option use the localhost:0.0

  10. Open new ssh session to the cloud server and test

  11. If client and server are configured properly you should be able to see DISPLAY variable being set
     
    root@urado2:~# set | grep DIS
    DISPLAY=localhost:10.0
    
    root@urado2:~# cat /etc/ssh/sshd_config | grep X11
    X11Forwarding yes
    X11DisplayOffset 10
    

  12. Test if we can start out sublime editor or if we get any new errors
  13.  
    root@urado2:~/Sublime Text 2# ./sublime_text
    (sublime_text:16851): Gtk-WARNING **: cannot open display: localhost:10.0
    

    To fix this we need to have an X11 windows server running on our local machine  If you are using Windows you can install for example Xming X Server for Windows

  14. Final test after the Xming server is up and running

  15. After lunch you should be able now to see the screen of your editor that was lunch on Linux cloud. 
    As this is a remote session there maybe some issues with lag and latency what will influence responsiveness and delays when editing.

Friday, March 29, 2013

Rackspace and Brocade pushing ADX platform

Brocade is Rackspace partner. The company provide load balancing as well as storage solution for the company. Since Brocade ADX load balancer became standard product Rackspace sells it got a bigger momentum and traction on the market. The Gartner magic quadrants for application delivery controllers show how the company progressed and improved.

Thursday, March 28, 2013

Good developer traits

I was searching for a way to use regex when searching in Chrome (found here Regular Expression Search Bookmarklet). Accidentally found this interesting blog entry and I though to repost it here.

How I read it is the fact that what really matters is to be good in what you are doing and specializing. The soft skill are nice to have but are of secondary requirements.

Taken from What Really Makes a Good Programmer blog:

Quick overview of AppFog PaaS platform

There is Openstack that aims to provide opensource IaaS solution for the Internet. But cloud has many faces and in a simplistic cloud taxonomy we can speak of IaaS, PaaS and SaaS as the main trends in the industry.

The video below provide a short introduction and comparison of Openstack and Appfog that is based on CloudFoundry. Both products embracing the open source developed paradigm. Both try to define standard and become leader in the respective domain.


As a bonus material for these who are interested in the overall technical system architecture this link  provide slides and video of Derek Collison (a lead technical architect for CloudFondry at Vmware) Cloud Foundry: Inside the Machine

Demonstration and example
  1. You need to create a basic account that is free (but has resource limitation)
  2. You can use the GUI to create your fist application
  3. Use the CLI af (docu here) to get a look and feel what it is 
This is the automatically generated demo.
 
$ curl -v http://rado-python.hp.af.cm/
* About to connect() to rado-python.hp.af.cm port 80 (#0)
*   Trying 15.185.166.79... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-cygwin) libcurl/7.22.0 OpenSSL/0.9.8r zlib/1.2.5 libidn/1.22 libssh2/1.2.7
> Host: rado-python.hp.af.cm
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: text/plain
< Keep-Alive: timeout=300
< Content-Length: 34
< Accept-Ranges: bytes
< Date: Thu, 28 Mar 2013 01:25:30 GMT
< X-Varnish: 816993201
< Age: 0
< Via: 1.1 varnish
< Connection: keep-alive
<
* Connection #0 to host rado-python.hp.af.cm left intact
* Closing connection #0
Hello World! AppFog Python Support

Some info about the application hosted on AppFog.
 
C:\Users\radoslaw>af login
Attempting login to [https://api.appfog.com]
Email: xxx
Password: ************
Successfully logged into [https://api.appfog.com]

C:\Users\radoslaw>af info
AppFog Free Your Cloud Edition
For support visit http://support.appfog.com

Target:   https://api.appfog.com (v0.999)
Client:   v0.3.18.12

User:     xxx
Usage:    Memory   (384.0M of 2.0G total)
          Services (1 of 8 total)
          Apps     (2 of 9999 total)

# download the source code 
C:\Users\radoslaw\workspace\appfog>af pull rado-python
Pulling last pushed source code: OK

The source code for our fist hello word application is very simple.
 
import bottle
import os

def application(environ, start_response):
    data = "Hello World! AppFog Python Support"
    start_response("200 OK", [
            ("Content-Type", "text/plain"),
            ("Content-Length", str(len(data)))
            ])
    return iter([data])

By default only one instance (cloud server?) is spin up to host our app.
 
C:\Users\radoslaw\workspace\appfog>af stats rado-python

+----------+-------------+----------------+--------------+---------------+
| Instance | CPU (Cores) | Memory (limit) | Disk (limit) | Uptime        |
+----------+-------------+----------------+--------------+---------------+
| 0        | 0% (4)      | 26.1M (128M)   | 7.1M (1G)    | 0d:0h:58m:18s |
+----------+-------------+----------------+--------------+---------------+

C:\Users\radoslaw\workspace\appfog>af instances rado-python

+-------+---------+--------------------+
| Index | State   | Start Time         |
+-------+---------+--------------------+
| 0     | RUNNING | 03/28/2013 12:34AM |
+-------+---------+--------------------+

On the GUI we can quickly and easily increase the number of instance. Once new instance is spun up and application code is cloned to it we see the changes on the CLI as well.
 
C:\Users\radoslaw\workspace\appfog>af stats rado-python

+----------+-------------+----------------+--------------+--------------+
| Instance | CPU (Cores) | Memory (limit) | Disk (limit) | Uptime       |
+----------+-------------+----------------+--------------+--------------+
| 0        | 0% (4)      | 26.1M (128M)   | 7.1M (1G)    | 0d:1h:2m:35s |
| 1        | 0.7% (4)    | 26.0M (128M)   | 0B (1G)      | 0d:0h:0m:58s |
+----------+-------------+----------------+--------------+--------------+

C:\Users\radoslaw\workspace\appfog>af instances rado-python

+-------+---------+--------------------+
| Index | State   | Start Time         |
+-------+---------+--------------------+
| 0     | RUNNING | 03/28/2013 12:34AM |
| 1     | RUNNING | 03/28/2013 01:35AM |
+-------+---------+--------------------+

After the testing we delete our application.
 
C:\Users\radoslaw\workspace\appfog>af delete rado-python

References
  1. https://docs.appfog.com/getting-started/af-cli#max-os-x-linux
  2. http://blog.appfog.com/getting-started-with-appfogs-command-line/
  3. http://net.tutsplus.com/articles/editorials/appfog-vs-heroku/

Wednesday, March 27, 2013

Custom 3 pane layout in Sublime

I work with many files and often I need to copy and paste various lines when editing them. What help me a lot was a nice 3 pane layout where I could move files around. One way to create it is to use the Origami plugin that you can install in Sublime. But to make it simpler I decided to create a keyborad shortcut/menu entry for this.

The layout looks like this:


This is the config you need to put:
  • create a config file Main.sublime-menu in your sublime2\Data\Packages\User directory
  • copy the config into the file
[{
    "id": "view",
    "children": [{
        "id": "layout",
        "children": [{
             "command": "set_layout",
             "caption" : "Custom: 3 Pane",
             "mnemonic": "C", 
             "args": {
        "cells": [[0, 0, 1, 1], [0, 1, 2, 2], [1, 0, 2, 1]],
     "rows": [0.0, 0.5, 1.0], 
     "cols": [0.0, 0.5, 1.0]
   }
  }]
 }]
}]
  • Before you save the file open console (view->show console) and see if the file is parsed and loaded properly
# on the console
found 11 files for base name Main.sublime-menu
  • If you need now to divide your screen you need to select View->Layout->Custom:3 Pane from menu
  • You can create a custom keyboard shortcut as well. To replace the original [alt+ctrl+3] open the Preferences -> Bindings -> User and copy this similar config there. 
  • {
        "keys": ["alt+shift+3"],
        "command": "set_layout",
        "args": 
        {
            "cells": [[0, 0, 1, 1], [0, 1, 2, 2], [1, 0, 2, 1]],
            "rows": [0.0, 0.5, 1.0], 
            "cols": [0.0, 0.5, 1.0]
        }
    }
    
References
  1. http://www.macdrifter.com/2012/07/sublime-text-working-with-multiple-panes.html
  2. http://stackoverflow.com/questions/13149800/custom-layouts-sublime-text-2
  3. http://stackoverflow.com/questions/10674030/in-sublime-text-2-reopen-build-output

Tuesday, March 26, 2013

Installation of Sublime Text editor on Ubuntu 12.10

Update 1:
added how to update alternatives in debian/ubuntu

An absolutely brilliant  installation manual how to install Sublime on Ubuntu:
How to install Sublime Text 2 on Ubuntu

If you want now type 'editor' in bash and start sublime as your default editor you have to update the alternative system that Debian introduces.
 
 update-alternatives --install /usr/bin/editor editor /opt/Sublime\ Text\ 2/sublime_text 55
 update-alternatives --display editor



Installing VirtualBox Extension Pack drivers on Ubuntu 12.10 guest system

After successful upgrade of my VirtualBox and its Extension Pack here I decided to spin up a new local virtual machine and install Ubuntu 12.10. All went fine except that I had an issue installing the Guest additional drivers from the Extension Pack. Below are my notes and missing steps from the official guide (*) Chapter 4. Guest Additions that I had to do in my Ubuntu Desktop.

The default installation of Ubuntu for your desktop doesn't install dkms package. Following the (*) guide you will have to install it. 

Additionally you have to install your kernel source header files that 

apt-get install aptitude
aptitude install  linux-headers-3.5.0-26-generic

After this the driver instillation goes well. All what you need to do is now to restart your VM.
 
root@rado-VirtualBox:/media/rado/VBOXADDITIONS_4.2.10_84104# sh ./VBoxLinuxAdditions.run  uninstall
root@rado-VirtualBox:/media/rado/VBOXADDITIONS_4.2.10_84104# sh ./VBoxLinuxAdditions.run 
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.2.10 Guest Additions for Linux..........
VirtualBox Guest Additions installer
Removing existing VirtualBox DKMS kernel modules ...done.
Removing existing VirtualBox non-DKMS kernel modules ...done.
Building the VirtualBox Guest Additions kernel modules ...done.
Doing non-kernel setup of the Guest Additions ...done.
Starting the VirtualBox Guest Additions ...done.
 Installing the Window System drivers
Installing X.Org Server 1.13 modules ...done.
Setting up the Window System to use the Guest Additions ...done.
You may need to restart the hal service and the Window System (or just restart
the guest system) to enable the Guest Additions.

Installing graphics libraries and desktop services components ...done.

VirtualBox upgrade error

I've decide to update my old version of VirtualBox on Windows 7. The installation of the software went fine but when I tried to update the  Extension Pack I run into this issue:
 
Failed to install the Extension Pack C:/Users/rado4752/.VirtualBox/Oracle_VM_VirtualBox_Extension_Pack-4.2.10.vbox-extpack.

The installer failed with exit code 1: VBoxExtPackHelperApp.exe: error: Failed to rename the extension pack directory: VERR_ALREADY_EXISTS

rcExit=1.

Result Code: E_FAIL (0x80004005)
Component: ExtPackManager
Interface: IExtPackManager {3295e6ce-b051-47b2-9514-2c588bfe7554}

As the error suggest this is most likely a file system security issue. As I don't have full Admin right on this host the solution for me was to clean and uninstall the old Extension Pack before trying to install the new one again (more info about it can be found here: https://forums.virtualbox.org/viewtopic.php?f=6&t=46861)
 
c:\Program Files\Oracle\VirtualBox>vboxmanage extpack cleanup
Successfully performed extension pack cleanup

# try to install the Extension Pack again
C:\Users\rado4752\.VirtualBox>Oracle_VM_VirtualBox_Extension_Pack-4.2.10.vbox-extpack

Monday, March 25, 2013

Example of a failing DNS request

I've run into an DNS issue today. I usually scan quickly the answer from dig as I'm interested in the actual A, PTR or MX only. Today the issue was different. Below in the two failing example DNS requests please note the SERVFAIL status code.

Example 1
 
dig @194.2.2.2 www.example.com A

; DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> @194.2.2.2 www.example.com A
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 5995
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.example.com. IN A

;; ANSWER SECTION:
www.example.com. 86400 IN CNAME buuu.example.com.

Example 2
 
dig @194.2.2.2 www.example.com

; DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> @83.138.151.80 www.example.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 55778
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.example.com. IN A

;; Query time: 12 msec
;; SERVER: 194.2.2.2#53(194.2.2.2)
;; WHEN: Mon Mar 25 17:41:28 2013
;; MSG SIZE  rcvd: 52

References
  1. http://networking.ringofsaturn.com/Unix/dnstroubleshooting.php

Is Connection header mandatory in HTTP 1.1

I've troubleshoot recently an F5 load balancer issue and show a traffic like below (example with mangled fields)

10.178.155.180 - F5 
10.178.100.64  - server

Nr      Time        Source                Destination    Protocol flags  Length vlan hsrc                  hdst                  sport  dport  Info                           
    1 9.970000   10.178.155.180        10.178.100.64      TCP      0x02   100    777  F5Networ_45:c9:05    Rackspac_08:2d:3d     41588  80     41588 > http [SYN] Seq=0 Win...
    1 9.970821   10.178.100.64         10.178.155.180     TCP      0x12   100    777  Rackspac_08:2d:3d    F5Networ_45:c9:05     80     41588  http > 41588 [SYN, ACK] Seq=...
    1 9.971842   10.178.155.180        10.178.100.64      TCP      0x10   92     777  F5Networ_45:c9:05    Rackspac_08:2d:3d     41588  80     41588 > http [ACK] Seq=1 Ack...
    1 9.971921   10.178.155.180        10.178.100.64      HTTP     0x18   179    777  F5Networ_45:c9:05    Rackspac_08:2d:3d     41588  80     GET /status.aspx HTTP/1.1   ...
    1 9.978979   10.178.100.64         10.178.155.180     HTTP     0x18   475    777  Rackspac_08:2d:3d    F5Networ_45:c9:05     80     41588  HTTP/1.1 200 OK  (text/html)...
    1 9.979898   10.178.155.180        10.178.100.64      TCP      0x10   92     777  F5Networ_45:c9:05    Rackspac_08:2d:3d     41588  80     41588 > http [ACK] Seq=88 Ac...
    1 9.980028   10.178.155.180        10.178.100.64      TCP      0x11   92     777  F5Networ_45:c9:05    Rackspac_08:2d:3d     41588  80     41588 > http [FIN, ACK] Seq=...
    1 9.980315   10.178.100.64         10.178.155.180     TCP      0x11   92     777  Rackspac_08:2d:3d    F5Networ_45:c9:05     80     41588  http > 41588 [FIN, ACK] Seq=...
    1 9.980950   10.178.155.180        10.178.100.64      TCP      0x10   92     777  F5Networ_45:c9:05    Rackspac_08:2d:3d     41588  80     41588 > http [ACK] Seq=89 Ac...

The HTTP request and response:

GET /status.aspx HTTP/1.1
Host: www.dummy.com
User-Agent: MyAgent :)

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=tbc1vlvsjxjx01xs2frend5z; path=/; HttpOnly
Date: Mon, 25 Mar 2013 09:59:33 GMT
Content-Length: 9

Status OK

Problem

The 'Host' header is mandatory in HTTP 1.1. Is 'Connection: keep-alive|close' HTTP header mandatory in HTTP 1.1 as well?

Analisis and results description

An example request from the Chrome browser that specifically provide a value for the 'Connection' header:

GET / HTTP/1.1
Host: www.google.com
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22
Accept-Encoding: gzip,deflate,sdch

HTTP/1.1 302 Found
Date: Mon, 25 Mar 2013 10:15:23 GMT
Server: gws
Location: https://www.google.com/
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Cache-Control: private
Content-Length: 220

We can see that the client specify what should happen to the underlying TCP session. In this example the TCP session should remain active and idle for a while - until either the client or server decide to close it.

Although in our example we can see that the client (F5)
  • didn't specify the 'Connection' header in the GET request
  • the session was closed by the F5 (the client) after receiving the response 

According to 8.1 Persistent Connections  of the rfc 2616 the persistent connections are the default behaviour in HTTP 1.1. The field is not mandatory and either sides can close it. In our example the  client decide to close the established tcp session after receiving the HTTP response.

References
  1. http://tools.ietf.org/html/rfc2616#section-8.1
  2. http://www8.org/w8-papers/5c-protocols/key/key.html

Sunday, March 24, 2013

Rackspace api-challenge summary

Below is a listing of some of my steps I went through when I participated in Rackspace  API Challenges: By Support, for Support contest. The final github repository with the code is here https://github.com/rtomaszewski/api-challenge
  1. Create initial repository on github
  2. In the OS of your choice install your favorite editor like Sublime (Installation of Sublime Text editor on Ubuntu 12.10) or other
  3. Install Python SDK for Rackspace cloud: How to install Rackspace Python SDK for cloud
  4. Authentication issue with London cloud account when using Pyrax module
  5. Howto add a file to api-challenge repository on github
  6. Challenge 1 script
  7. As my development efforts go on I wanted make sure I can always get back to the code I was initially released as challenge1.py script: How to create a branch on github to freeze your code in a particular version
  8. Challenge 2 script
  9. How to modify files in your branch on github
  10. bpython supports virtualenv
  11. How to develop and debug programs that use pyrax or python-novaclient libraries
  12. Challenge 3 script
  13. How to paste into Python interpreter a code snippet for testing
  14. Challenge 4 script
  15. Challenge 5 script
  16. How to merge changes from a custom branch into master branch on github
  17. Challenge 6 script
  18. Challenge 7 script
  19. Challenge 8 script
  20. Challenge 9 script
  21. How to use pyrax and ipython together to test and learn the library api
  22. Challenge 10 script
  23. Challenge 11 script

Create initial repository on github

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

This is in regards to the Rackspace API-challenge. More info can be found at Rackspace api-challenge summary.

All my steps to create my repository are listed below.

Install Git Shell for windows

mkdir api-challenge
git init

Create next a README.md file

git commit
git add README.md
git commit -m 'init'
git remote add origin git@github.com:rtomaszewski/api-challenge.git

git push -u origin master
Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of know
n hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Troubleshooting base on https://help.github.com/articles/error-permission-denied-publickey
 
ssh -vT git@github.com
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
....
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /c/Users/rado4752/.ssh/github_rsa
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

ssh-add -l
The agent has no identities.

I had to add and register my public ssh key github_rsa on github. After adding it using the webgui https://github.com/settings/ssh this test went fine.
 
ssh -vT git@github.com
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Applying options for github.com
debug1: Connecting to github.com [207.97.227.239] port 22.
debug1: Connection established.
debug1: identity file /c/Users/rado4752/.ssh/github_rsa type 1
...
Hi rtomaszewski! You've successfully authenticated, but GitHub does not provide
shell access.
debug1: channel 0: free: client-session, nchannels 1
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.5 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 1

git remote add origin git@github.com:rtomaszewski/api-challenge
fatal: remote origin already exists.

git remote -v
origin  git@github.com:rtomaszewski/api-challenge (fetch)
origin  git@github.com:rtomaszewski/api-challenge (push)

git push -u origin master
Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of know hosts.
ERROR: Repository not found.
Fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

git remote rm origin
git remote add origin git@github.com:rtomaszewski/api-challenge.git
git remote -v
origin  git@github.com:rtomaszewski/api-challenge.git (fetch)
origin  git@github.com:rtomaszewski/api-challenge.git (push)

As there is not repository with the name api-challenge I created it under my github account.
 
git push -u origin master

After all this trouble i could navigate to my newly created repository under https://github.com/rtomaszewski/api-challenge

At the end got to http://challenge.cldsrvr.com/register and register your repo.

Practical online github course

There are many modern distributed versioning systems and github is one of them. This link is going to take you to a simple online but practical course where you can practice the basic git commands.

Learning by doing: http://try.github.com/ (http://www.codeschool.com/courses/try-git)

Commands reference:
 
$ git init
$ git status
$ git add octocat.txt
$ git status
$ git commit -m "Add cute octocat story"
$ git add '*.txt'
$ git commit -m 'Add all the octocat txt files'
$ git log
$ git remote add origin git@github.com:rtomaszewski/try_git.git
$ git push -u origin master
$ git pull origin master
$ git diff HEAD
$ git add octofamily/octodog.txt
$ git diff --staged
$ git reset octofamily/octodog.txt
$ git checkout -- octocat.txt
$ git branch clean_up
$ git checkout clean_up
$ git rm '*.txt'
$ git commit -m "Remove all the cats"
$ git checkout master
$ git merge clean_up
$ git branch -d clean_up
$ git push


Friday, March 22, 2013

Code changes in Openstack Grizzly release

There is almost a new Openstack version called Grizzly around the corner. Some stats what can we expect base on [1] and [2].

nova
added lines 281036
removed lines 214574
total lines added 66462
commits 1889

https://github.com/openstack/nova/graphs/code-frequency
Quantum
added lines 92376
removed lines 41864
total lines added 50512
commits 602

https://github.com/openstack/quantum/graphs/code-frequency
Keystone
added lines 28488
removed lines 17265
total lines added 11223
commits 289
Glance
added lines 15717
removed lines 13163
total lines added 2554
commits 253
Cinder
added lines 73344
removed lines 72825
total lines added 519
commits 306

https://github.com/openstack/cinder/graphs/code-frequency
Horizon
added lines 187351
removed lines 125838
total lines added 61513
commits 160
Swift
added lines 17270
removed lines 6313
total lines added 10957
commits 222

https://github.com/openstack/swift/graphs/code-frequency
References
  1. http://www.slideshare.net/enovance/meetup-open-stackgrizzly-17372714
  2. https://github.com/openstack

Openstack deployment options

Openstack wants to be as flexible as possible. It means that each service that is developed under the umbrella of Openstack has to be written in a modular way to accept different backend system depending on user preferences. From a high level point of view it means that a service need to have a well defined internal and external API, and the more specific technical implementation details are left for backend systems.

The Openstack specific public (external) API is one milestone for every project. If the API is not reach, flexible and useful enough it will not empower uses when consuming the service. On the other side, if the internal service level API is badly design it may cause issues for example when integrating with other components, exchanging messages, causing bottlenecks or hinder vertical scalability to expand system capacity.

Below is a list of deployment options for Folsom and Grizzly Openstack release.
  • External API
    •  XML
    •  JSON
  • Possible hypervisors for OpenStack Compute
    • KVM
    • Xen
    • Citrix XenServer
    • Microsoft HyperV
    • VMware ESX
    • LXC
  • Possible OpenStack Block Storage drivers 
    • Coraid
    • EMC
    • GlusterFS
    • Huawei
    • LVM
    • NetApp
    • Nexenta
    • NFS
    • Ceph RBD
    • SAN/HP
    • SAN/Solaris
    • Scality
    • Sheepdog
    • SolidFire
    • Storwize
    • Windows
    • Xenapi
    • XIV
    • Zadara
  • Possible OpenStack Network backends 
    • Big Switch
    • Brocade
    • Cisco
    • Hyper-V
    • Linux Bridge
    • MidoNet
    • NEC
    • Nicira
    • Open vSwitch
    • PLUMgrid
    • Ryu
  • OpenStack Identity drivers
    • LDAP
    • SQL
    • PAM
    • KVS

How to manually create shadow password for a user in Linux

On Linux systems users passwords are stored in /etc/shadow file. An example line showing a password and account details for a 'demo' user on my system looks like this:
 
$ grep demo /etc/shadow
demo:$6$DdiZmmSe$eSXGHIB2gx.cHY.PR.Tfz8l00iStSgea0o7glv2ptBq8FpfSjz5XVU2GgCVzr72zAx4wG4gfYXucgoOGb3Rb7/:15786:0:99999:7:::

Problem

How to compute and generate a user password so it can be copied into the shadow file directly.

Solution and results description

The description of how the password is created can be found here:
 
$ man shadow
encrypted password
           Refer to crypt(3) for details on how this string is interpreted.


$ man 3 crypt
Glibc Notes
       The glibc2 version of this function supports additional encryption algorithms.

       If salt is a character string starting with the characters "$id$" followed by a string terminated by "$":

              $id$salt$encrypted

       then instead of using the DES machine, id identifies the encryption method used and this then determines how the rest of the password string is  interpreted.   The  following
       values of id are supported:

              ID  | Method
              ─────────────────────────────────────────────────────────

              1   | MD5
              2a  | Blowfish (not in mainline glibc; added in some
                  | Linux distributions)
              5   | SHA-256 (since glibc 2.7)
              6   | SHA-512 (since glibc 2.7)

       So $5$salt$encrypted is an SHA-256 encoded password and $6$salt$encrypted is an SHA-512 encoded one.

       "salt"  stands  for the up to 16 characters following "$id$" in the salt.  The encrypted part of the password string is the actual computed password.  The size of this string
       is fixed:

       MD5     | 22 characters
       SHA-256 | 43 characters
       SHA-512 | 86 characters


Analyzing the shadow line for the demo user we can see that his password:
  • uses SHA512 algorithm
  • it was generated with a salt string DdiZmmSe
  • it is 86 char long
  •  
    $ python -c "print len('eSXGHIB2gx.cHY.PR.Tfz8l00iStSgea0o7glv2ptBq8FpfSjz5XVU2GgCVzr72zAx4wG4gfYXucgoOGb3Rb7/')"
    86
    
The first impression that we could simply use a tool to generate an SHA digest isn't going to work unfortunately. The reason is that SHA512 generates only a 512 bit long message digest (that is 64 char string) and the password in shadow file is 86 char long.

Further researching found out that even though the 'crypt' function uses the standard SHA crypto function it varies in a number of ways to produce the 86 char long string. An interesting blog describing the algorithm can be found here: http://www.vidarholen.net/contents/blog/?p=33.

There are number of ways you generate our password:
  • we can use a bash script 
https://github.com/rtomaszewski/experiments/blob/master/shadow_pass.sh
 
$ ./shadow_pass.sh demo DdiZmmSe
$6$DdiZmmSe$eSXGHIB2gx.cHY.PR.Tfz8l00iStSgea0o7glv2ptBq8FpfSjz5XVU2GgCVzr72zAx4wG4gfYXucgoOGb3Rb7/
  • we can write a little script and call the crypt function directly to generate the password
http://serverfault.com/questions/330069/how-to-create-an-sha-512-hashed-password-for-shadow
 
$ python -c "import crypt, getpass, pwd; print crypt.crypt('demo', '\$6\$DdiZmmSe\$')"
$6$DdiZmmSe$eSXGHIB2gx.cHY.PR.Tfz8l00iStSgea0o7glv2ptBq8FpfSjz5XVU2GgCVzr72zAx4wG4gfYXucgoOGb3Rb7/

Further reading


Tuesday, March 12, 2013

Rackspace Private Cloud v3 available

In my previous post How to install Rackspace Private Cloud (Alamo) on a single physical server I described how to install Alamo version 2. Since then the software has evolved and a new versions 3 with new features is available for download and testing.

Instalation

Full instruction how to install it can be found here: http://www.rackspace.com/cloud/private/script/

A compact version looks like:

curl -L "http://sh.opencenter.rackspace.com/install.sh" | bash -s server
curl -L "http://sh.opencenter.rackspace.com/install.sh" | bash -s dashboard
curl -L "http://sh.opencenter.rackspace.com/install.sh" | bash -s agent

Architecture has changed

Alamo version 3 comes with a modified and extended architecture to support new features. A high level diagram can be found here: http://www.rackspace.com/cloud/private/openstack_software/

More documentation

Knowledge centre articles:
http://www.rackspace.com/knowledge_center/getting-started/rackspace-private-cloud

More instruction how to install and access to dedicated form for Rackspace Private Cloud
https://privatecloudforums.rackspace.com/viewtopic.php?f=4&t=345&start=0

Opencenter, installer and more about source code is here:
https://github.com/rcbops

Monday, March 11, 2013

ASA ssh login problem

Working for ISP is big fun. From all the work you do there is one routine like swapping of network devices (for example Cisco ASA firewall) that you are going to do. Not going into too much details the process is straight forward and requires:
  • copy the config to new device
  • rack the new device
  • make sure that the switches and VLANs are configured properly
  • change routing info if needed 
  Problem

After putting new ASA FW into rack you can connect using serial line but you can't access it over SSH. You getting this error message.
 
$ ssh 1.1.1.77
ssh_exchange_identification: Connection closed by remote host

Troubleshooting and solution

From serial console access enable debugging:
 
# debug ssh

Connect over ssh. You are going to see this logs on console:
 
Device ssh opened successfully.
SSH0: SSH client: IP = '212.100.225.42'  interface # = 2
SSH: unable to retrieve default host public key.  Please create a defauth RSA key pair before using SSH
SSH0: Session disconnected by SSH server - error 0x00 "Internal error"

Searching for 'unable to retrieve default host public key' finds the links in reference sections.  To fix this we need:
 
fw-asa(config)# crypto key generate rsa
INFO: The name for the keys will be: 
Keypair generation process begin. Please wait...

Once ASA has its own RSA key to use for SSH handshaking the logs from a sucessful SSH session looks like:
 
fw-asa# 
Device ssh opened successfully.
SSH0: SSH client: IP = '212.100.225.42'  interface # = 2
SSH: host key initialised
SSH: license supports 3DES: 2
SSH: license supports DES: 2
SSH0: starting SSH control process
SSH0: Exchanging versions - SSH-2.0-Cisco-1.25
SSH0: send SSH message: outdata is NULL
server version string:SSH-2.0-Cisco-1.25SSH0: receive SSH message: 83 (83)
SSH0: client version is - SSH-2.0-OpenSSH_4.3
client version string:SSH-2.0-OpenSSH_4.3SSH0: begin server key generation
SSH0: complete server key generation, elapsed time = 1830 ms
SSH2 0: SSH2_MSG_KEXINIT sent
SSH2 0: SSH2_MSG_KEXINIT received
SSH2: kex: client->server aes128-cbc hmac-md5 none
SSH2: kex: server->client aes128-cbc hmac-md5 none
SSH2 0: expecting SSH2_MSG_KEXDH_INIT
SSH2 0: SSH2_MSG_KEXDH_INIT received
SSH2 0: signature length 143
SSH2: kex_derive_keys complete
SSH2 0: newkeys: mode 1
SSH2 0: SSH2_MSG_NEWKEYS sent
SSH2 0: waiting for SSH2_MSG_NEWKEYSSSH0: TCP read failed, error code = 0x86300003 "TCP connection closed"
SSH0: receive SSH message: [no message ID: variable *data is NULL]

SSH2 0: Unexpected mesg type receivedSSH0: Session disconnected by SSH server - error 0x00 "Internal error"

References
  1. http://www.myteneo.net/blog/-/blogs/accessing-cisco-asa-using-ssh/
  2. http://ciscotalk.wordpress.com/2011/08/31/enabling-ssh-on-a-cisco-asa/

Sunday, March 10, 2013

Openstack Gerrit code review process details

Are you looking to contribute code to one of the Openstack projects (example include Nova, Quantum or Glance and many more). If the answer is yes, you definitely need to get familiar with the code review that Openstack enforces. A good overview of what this is can be found here Code review process in Openstack uses Zuul.

On a technical site the process has been implemented with a help of Gerrit system. From end user perspective you get access to a nice and good looking page in a browser that helps you to review, comment and approve code changes that should be committed to master repository. A quick and good introduction of what Gerrit do this and how it helps can be found here: Gerrit Code Review - A Quick Introduction.

Example how this interface looks like:

Code Review: https://review.openstack.org

When you select a link you dig into more details: (example) https://review.openstack.org/#/c/23878/


In this screen we can find:

  • linked Bug id if exists (https://code.launchpad.net/bugs/1131759
  • linked Blueprints if exists
  • reviewer list
  • one or more patches with proposed code changes
  • comments and suggestions before the code can be accepted in the mainstream repository
But the best part of Gerrit is its capability to pull the original file and create a diff to visual the changes. An example from the review #23878 is seen below.

https://review.openstack.org/#/c/23878/4/heat/tests/test_api_openstack_v1.py

Saturday, March 2, 2013

Text developer editor with Python API

I often need to work between Linux and Windows systems. Under every OS I have my favorite tools I like that help me to get the job done. But there has always been one tool that I wasn't very happy with: a good text editor.

Problem

What is a good cross platform editor with development features that is written and integrated with Python.

Analisis and discussion 

When I code I always like to know the editor so I can quickly and comfortably navigate in a single or multiple files at the same time. I've found recently one that I tend to use more: Sublime. The other one I was using for a long while was Notepad++ but it was only limited to Windows.

Why Sublime works for me:
My simple config

You can view and change all global settings under Menu - Preferences - Settings - Default but a better way is to create a local customized user preferences file.

To modify user settings open the following file under Menu - Preferences - Settings - User and copy or modify these options:
 
"fade_fold_buttons"        : false,
"highlight_line"           : true,
"auto_complete_size_limit" : 44194304,
"tree_animation_enabled"   : true,

My packages

Below are some of my packages I'm using

How to highlight a whole line in Sublime like in Notepad++ 

In notepad++ when editing your can enable whole like to be highlighted (here are some example screenshots). To achieve the same effect in Sublime you need to enable the highlight_line: true.

References and documentation

http://www.sublimetext.com/2
http://docs.sublimetext.info/en/latest/index.html