Challenge 11 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 11 script

Below is the output and results from script #11 from the api-challenge.

The script can be repeatedly run with the option -d otherwise it stops if any of the cloud objects it wants to create already exist.

    Challenge 11: Write an application that will: Create an SSL terminated load balancer 
    (Create self-signed certificate).
    Create a DNS record that should be pointed to the load balancer. Create Three servers 
    as nodes behind the LB.

    Each server should have a CBS volume attached to it. (Size and type are irrelevant.)
    All three servers should have a private Cloud Network shared between them.
    Login information to all three servers returned in a readable format as the result 
    of the script, including connection information. Worth 6 points


Depending on the obligatory -n options it will create cloud objects with predictable names.
Below is an example what objects it creates and what the naming conversion is.

# # python challenge11.py

    usage: challenge11.py [-h] [-v] [-d] [ -k key-key  ] [ -c cert ] [ -i image-id ][ -f flavor-id ] -n FQDN-name

    image-id -f flavor-id
        -h - usage help
        -v - verbose / debug output
        -d - delete objects if they existed in cloud before creating new one
        -k - certificate pritate key (see -c below)
        -c - public certificate ( see -k above)
        -n - FQDN name like www.myexample.com
        -i - specify image-id or use the default for Ubuntu 10.04
        -f - specify flavor-id or use the default for the smallest cloud server


# python challenge11.py -d -n www.challenge11.myrado.net

[01:04:17] Checked your cert/key pair www.challenge11.myrado.net.crt/www.challenge11.myrado.net.key, ok
[01:04:18] Deleting existing domain challenge11.myrado.net
[01:04:20] Checked your FQDN www.challenge11.myrado.net, ok
[01:04:20] Checked your DNS domains challenge11.myrado.net, there is none, ok
[01:04:26] Deleted existing cloud server www.challenge11.myrado.net-0
[01:04:29] Deleted existing cloud server www.challenge11.myrado.net-1
[01:04:33] Deleted existing cloud server www.challenge11.myrado.net-2
[01:04:33] Checked your existing cloud server www.challenge11.myrado.net-*, ok
[01:04:34] Checked your block images www.challenge11.myrado.net, there is none, ok
[01:04:36] Checked your lb www.challenge11.myrado.net, there is none, ok
[01:04:37] Checked your image id d4c7b93d-9f18-45dc-aa7c-3e3b126e3792, ok
[01:04:37] Checked your flavor id 2, ok
[01:04:38] Deleted existing network obj www.challenge11.myrado.net
[01:04:38] Checked your existing network objects www.challenge11.myrado.net, ok
[01:04:38] Checked your cert/key pair www.challenge11.myrado.net.crt/www.challenge11.myrado.net.key, ok
[01:04:39] Checked your FQDN www.challenge11.myrado.net, ok
[01:04:39] Checked your DNS domains challenge11.myrado.net, there is none, ok
[01:04:45] Deleted existing cloud server www.challenge11.myrado.net-1
[01:04:50] Deleted existing cloud server www.challenge11.myrado.net-2
[01:04:50] Checked your existing cloud server www.challenge11.myrado.net-*, ok
[01:04:50] Deleted existing block storage image  www.challenge11.myrado.net-0
[01:04:51] Deleted existing block storage image  www.challenge11.myrado.net-1
[01:04:51] Deleted existing block storage image  www.challenge11.myrado.net-2
[01:04:51] Checked your block images www.challenge11.myrado.net, there is none, ok
[01:05:04] Checked your image id d4c7b93d-9f18-45dc-aa7c-3e3b126e3792, ok
[01:05:04] Checked your flavor id 2, ok
[01:05:05] Checked your existing network objects www.challenge11.myrado.net, ok
[01:05:05] Building 3 cloud servers
.
.
.
.
[01:08:20] Building and configuring lb ...
.
.
.
.
.
.
.
[01:09:09] Building and configuring dns domain ...
[01:09:13] ----------------------------------------------------------------------
[01:09:13] vip name www.challenge11.myrado.net and ip 162.13.24.49
[01:09:13] cloud server www.challenge11.myrado.net-0 added to pool as 10.179.65.126
[01:09:13] cloud server www.challenge11.myrado.net-1 added to pool as 10.179.70.78
[01:09:13] cloud server www.challenge11.myrado.net-2 added to pool as 10.179.73.93
[01:09:13] ----------------------------------------------------------------------
Server # 0: ID 3227a512-3a1c-4f68-a82a-89d15ec74341 name www.challenge11.myrado.net-0 pub IP 95.138.173.80 priv IP 192.168.100.2 password SecretP@ss1
Server # 1: ID 662606f2-66a8-4281-8025-5161bf31754c name www.challenge11.myrado.net-1 pub IP 95.138.171.48 priv IP 192.168.100.1 password SecretP@ss1
Server # 2: ID f37d3551-60f5-40f3-92eb-4cbb1f7dbcfe name www.challenge11.myrado.net-2 pub IP 95.138.175.130 priv IP 192.168.100.3 password SecretP@ss1

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge11
2. http://docs.rackspace.com/loadbalancers/api/v1.0/clb-devguide/content/SSLTermination-d1e2479.html

How to use pyrax and ipython together to test and learn the library api

When testing pyrax module using interactive ipython session I find myself often trying to inspect a single list value. In practice as my lists usually were build out of the same objects what I wanted is to inspect a single object from the list to discover its attributes and functions.

To overcome the ipython object limitation when it comes to introspection or reflection features I use this little helper script described here Custom magic function in IPython session. Below is an example session howto use in practice (in line you press TAB to let the ipython introspect the object)

ipython -i bpython-pyrax.py
In [1]: cs.images.list()                                                                                                            19:51:28
Out[1]:
[<Image: Ubuntu 13.04 (Raring Ringtail)>,
 <Image: Gentoo 13.1>,
 <Image: Fedora 18 (Spherical Cow)>,
truncated ...

In [2]: rl
------> rl()
created variable l0 = <Image: Ubuntu 13.04 (Raring Ringtail)>
created variable l1 = <Image: Gentoo 13.1>
created variable l2 = <Image: Fedora 18 (Spherical Cow)>
created variable l3 = <Image: FreeBSD 9.1>
created variable l4 = <Image: Ubuntu 12.10 (Quantal Quetzal)>
created variable l5 = <Image: Ubuntu 12.04 LTS (Precise Pangolin)>
created variable l6 = <Image: Ubuntu 10.04 LTS (Lucid Lynx)>
created variable l7 = <Image: Red Hat Enterprise Linux 6.3>
list cs.images.list() has 45 elements but only 7 was printed

In [3]: l0. 
l0.HUMAN_ID              l0.created               l0.id                    l0.metadata              l0.progress
l0.NAME_ATTR             l0.delete                l0.is_loaded             l0.minDisk               l0.set_loaded
l0.OS-DCF:diskConfig     l0.get                   l0.links                 l0.minRam                l0.status
l0.OS-EXT-IMG-SIZE:size  l0.human_id              l0.manager               l0.name                  l0.updated

In [5]: l0.name
Out[5]: u'Ubuntu 13.04 (Raring Ringtail)'

References

1. http://rtomaszewski.blogspot.co.uk/2013/04/how-to-dynamically-create-variable-name.html
2. https://github.com/rtomaszewski/api-challenge

How to capture HTTP traffic using tcpdump

Everyone knows tcpdump but not everyone knows how to use it in efficient way. Below is a nice trick how to capture the HTTP GET request and the server response in plain text.

 
root@server:~# tcpdump -c100 -A -l -tttt -s0 -qpnni any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

2013-04-28 13:57:16.725851 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`....(.@*..Hx...........*..P@   ...........}.P/.Hp......8@.     .........
..b.........
2013-04-28 13:57:16.728057 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`....(.7*..P@   ..........*..Hx............P.}}.[./.Hq..7..{.........
^.....b.....
2013-04-28 13:57:16.728093 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.Hq}.[............
..b.^...
2013-04-28 13:57:16.728445 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 166
`......@*..Hx...........*..P@   ...........}.P/.Hq}.[............
..b.^...GET / HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: www.google.com
Accept: */*


2013-04-28 13:57:16.729989 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`.... .7*..P@   ..........*..Hx............P.}}.[./.I.....^E.....
^.....b.
2013-04-28 13:57:16.742677 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 988
`......7*..P@   ..........*..Hx............P.}}.[./.I......h.....
^.....b.HTTP/1.1 302 Found
Location: http://www.google.co.uk/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=0241f520b15aae6e:FF=0:TM=1367157332:LM=1367157332:S=5lVTK-ZqTfrki7HN; expires=Tue, 28-Apr-2015 13:55:32 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=OyloOHElfW8AKOcsRJ4DeQnfMhqfmnqJgcpYXlsSrN2ouREV9KHjS9boJZTfBoFZvYtVg0ugcBa2lJQKX-WrQ_uMxoIvPg-4JehPfFEdyGl_oh0RS37x_V6a_ozMElzJ; expires=Mon, 28-Oct-2013 13:55:32 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Sun, 28 Apr 2013 13:55:32 GMT
Server: gws
Content-Length: 221
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/">here</A>.
</BODY></HTML>

2013-04-28 13:57:16.742700 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.I.}._w...........
..b.^...
2013-04-28 13:57:16.743753 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.I.}._w...........
..b.^...
2013-04-28 13:57:16.745725 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`.... .7*..P@   ..........*..Hx............P.}}._w/.I.....ZS.....
^.....b.
2013-04-28 13:57:16.745743 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.I.}._x...........
..b.^...

10 packets captured
10 packets received by filter
0 packets dropped by kernel

Custom magic function in IPython session

I needed some additional functionality when running my interactive ipython session. As ipython session is easily extendable with a help of custom python base functions I decided to write one as an example.

The function searches for the last outputed list object and creates variables for easy access (latest version can be found on github). Instead of typing l[_index_] you can simple tame l__index_. See the example below.

 
~/.config/ipython/profile_default/startup# cat rl.py

def object_name(obj, ret=None):
    g=globals()
    ret=[]
    obj_id=id(obj)
    for i in g.keys() :
        if id(g[i]) == obj_id :
            print i, g[i]
            ret.append(i)
    return ret


def rl (mylist=None, prefix=None, limit=7):
    prefix = prefix if prefix else "l"
    limit = limit if limit else 7

    aux=list(Out.keys()) # we want to make a copy
    aux.sort()
    aux.reverse()
    last_index = aux[0]

    if mylist == None :
        mylist = Out[last_index]
        mylist_name=In[last_index]

        if type(mylist) is not  list :
            for i in aux :
                # print i, Out[i],
                if type(Out[i]) is list :
                    mylist = Out[i]
                    mylist_name=In[i]
                    print("using last list object: %s" % mylist_name)
                    break
    else :
        mylist_name = filter( lambda x: not x.startswith("_"), object_name(l))

    if type(mylist) is not  list :
        print("can't find any list or specified object is not a list")
        return

    glo=globals()

    for k, val in enumerate(mylist) :
        if k <= limit :
            name=prefix+str(k)
            glo[name]=val
            print("created variable %s = %s" % (name, val) )
        else :
            print("list %s has %d elements but only 7 was printed" % ( mylist_name, len(mylist) ) )
            break

def rla (**kargs) :
  rl(prefix="a", **kargs)

def rlb (**kargs) :
  rl(prefix="b", **kargs)

print
print("auxiliary functions `rl` `rla` and `rlb` have been defined")

Example how this works

 
$ ipython='ipython  --colors Linux --autocall=2

In [17]: l=[1,2,3,1,2,3,1,2,3,1,2,3]

# instead of typing l[0] or l[1] etc we can now use this variables
In [18]: rl
-------> rl()
created variable l0 = 1
created variable l1 = 2
created variable l2 = 3
created variable l3 = 1
created variable l4 = 2
created variable l5 = 3
created variable l6 = 1
created variable l7 = 2
list l has 24 elements but only 7 was printed

In [2]: a=l    

In [6]: a
Out[6]: [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]

In [7]: rl(a)
__ [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
_ [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
a [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
_6 [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
_4 [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
l [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
created variable l0 = 1
created variable l1 = 2
created variable l2 = 3
created variable l3 = 1
created variable l4 = 2
created variable l5 = 3
created variable l6 = 1
created variable l7 = 2
list ['a', 'l'] has 12 elements but only 7 was printed

In [10]: a
Out[10]: [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]

In [11]: rlb
-------> rlb()
created variable b0 = 1
created variable b1 = 2
created variable b2 = 3
created variable b3 = 1
created variable b4 = 2
created variable b5 = 3
created variable b6 = 1
created variable b7 = 2
list a has 12 elements but only 7 was printed

References

1. https://github.com/rtomaszewski/dotfiles
2. http://rtomaszewski.blogspot.co.uk/2013/04/how-to-find-list-variable-name.html
3. http://en.wikipedia.org/wiki/Reflection_(computer_science)
4. http://en.wikipedia.org/wiki/Type_introspection

How to find list variable name

I've found an interesting limitation or maybe this is my lack of knowledge about python. I was trying to write a function that takes a list as an argument and prints on stdout  the name of the list and its value. Unfortunately this wasn't possible as there isn't an available function neither from python builtin set or from the list object itself that retrieves the name of a list variable.

 
def example(mylist):
    try:
        print("The argument name %s and value %s" % (mylist.name, mylist))
    except Exception, e:
        print e
    try:
        print("The argument name %s and value %s" % (mylist.__name__, mylist))
    except Exception, e:
        print e    

Trying to run this generate these exceptions:

 
mylist=[1,2]
example(mylist)
'list' object has no attribute 'name'
'list' object has no attribute '__name__'

To solve this problem I write this little auxiliary function that iterates over all global symbols and try to find the variable name base on the value. Unfortunately it isn't perfect as list can have multiple references that share the save value so our function will return all variable names instead of only one.

 
def object_name(obj, ret=None):
    g=globals()
    ret=[]
    obj_id=id(obj)
    for i in g.keys() :
        if id(g[i]) == obj_id :
            print i, g[i]
            ret.append(i)
    return ret

# an example from ipython showing how the code works 
In [47]: l=[1,2,3]
In [48]: a=l 
In [49]: object_name(l)
l [1, 2, 3]
a [1, 2, 3]
Out[49]: ['l', 'a']

References

1. http://bytes.com/topic/python/answers/854009-how-get-name-list

How to dynamically create a variable name based on runtime conditions in Python

I needed a quick way to dynamically create local variables with a reference to my list elements. Below is an example code how to do it manually.

 
>>> l=["a","b","c","d"]
>>> l0=l[0]
>>> l0
'a'

But it is a monotone and boring task if you want to create l0, l1, l2 and so on variables. To solve this I try to play with the locals() and globals() built in functions in Python. The code below shows results.

 
>>> def f(x=1):
...  print globals()
...  print
...  print locals()
...  print
...  y=2
...  print x,y
...  print
...  print globals()
...  print
...  print locals()
...
>>>
>>>
>>> f()
{'loc': {...}, 'val': 'd', 'f': <function f at 0x2297050>, '__builtins__': <module '__builtin__' (built-in)>, 'k': 3, 'l': ['a', 'b', 'c', 'd'], '__package__': None, 'l2': 'c', 'l3': 'd', 'l0': 'a', 'l1': 'b', '__name__': '__main__', '__doc__': None}

{'x': 1}

1 2

{'loc': {...}, 'val': 'd', 'f': <function f at 0x2297050>, '__builtins__': <module '__builtin__' (built-in)>, 'k': 3, 'l': ['a', 'b', 'c', 'd'], '__package__': None, 'l2': 'c', 'l3': 'd', 'l0': 'a', 'l1': 'b', '__name__': '__main__', '__doc__': None}

{'y': 2, 'x': 1}

As we can see depending on the context where these functions are executed the return value is different and depends on the current runtime conditions.

The code below shows an example how to add a new variable (we could as well modify a variable value).

 
# python
>>> l=["a","b","c","d"]
>>> l0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'l0' is not defined
>>>
>>> loc=locals()
>>> loc["l0"] = l[0]
>>> loc["l1"] = l[1]
>>> loc["l2"] = l[2]
>>> loc["l3"] = l[3]
>>>
>>> l0
'a'
>>> l1
'b'
>>> l2
'c'
>>> l3
'd'

Solution

Once we put all the discovered elements together a solution to my problem is this simple code below.

 
>>> l=["a","b","c","d"]
>>> loc=locals()
>>> for k,val in enumerate(l) : loc["l"+str(k)]=val
...
>>> l0
'a'
>>> l1
'b'

References

1. http://en.wikipedia.org/wiki/Type_introspection
2. http://en.wikipedia.org/wiki/Reflection_(computer_science)
3. http://stackoverflow.com/questions/4015550/create-a-new-variable-as-named-from-input-in-python
4. http://stackoverflow.com/questions/932818/retrieving-a-variables-name-in-python-at-runtime
5. http://www.ibm.com/developerworks/library/l-pyint/index.html
6. http://stackoverflow.com/questions/8307612/how-to-create-class-variable-dynamically-in-python

How to use mutable and immutable objects as default values in Python

Python as a dynamic language allow us to define mutable and immutable objects and variables. There is a significant difference now what a function or class can do with its positional arguments depending if the arguments are immutable or not.

The example code below are taken from this blog post: Gotcha — Mutable default arguments. Do you know what is going to be printed on stdout and why (he explanation and results can be found in the link above)?

 
def foobar(arg_string="abc", arg_list = []): 
    print arg_string, arg_list 
    arg_string = arg_string + "xyz"
    arg_list.append("F")
 
for i in range(4): 
    foobar()

# (1) define a class for company employees 
class Employee:
    def __init__ (self, arg_name, arg_dependents=[]): 
        # an employee has two attributes: a name, and a list of his dependents 
        self.name = arg_name 
        self.dependents = arg_dependents
     
    def addDependent(self, arg_name): 
        # an employee can add a dependent by getting married or having a baby 
        self.dependents.append(arg_name)
     
    def show(self): 
        print
        print "My name is.......: ", self.name 
        print "My dependents are: ", str(self.dependents)
#--------------------------------------------------- 
#   main routine -- hire employees for the company 
#---------------------------------------------------
 
# (2) hire a married employee, with dependents 
joe = Employee("Joe Smith", ["Sarah Smith", "Suzy Smith"])
 
# (3) hire a couple of unmarried employess, without dependents 
mike = Employee("Michael Nesmith") 
barb = Employee("Barbara Bush")
 
# (4) mike gets married and acquires a dependent 
mike.addDependent("Nancy Nesmith")
 
# (5) now have our employees tell us about themselves 
joe.show() 
mike.show() 
barb.show()

Challenge 10 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 10 script

Below is the output and results from script #10 from the api-challenge.
The script shows more advance example how consume cloud resources like cloud servers, cloud load balances, cloud files and cloud dns. It does it by building a simple solution.

The script can be repeatedly run with the option -d otherwise it stops if any of the cloud objects it wants to create already exist.
Depending on the obligatory -n options it will create cloud objects with predictable names.
Below is an example what objects it creates and what the naming conversion is.

# python challenge10.py

    usage: challenge10.py [-h] [-v] [-d] [ -s ssh-key ] [ -e error-page ] [ -c container-name ] [ -i image-id ] [ -f flavor-id ] -n FQDN-name

    image-id -f flavor-id
        -h - usage help
        -v - verbose / debug output
        -d - delete objects if they existed in cloud before creating new one
        -n - FQDN name like www.myexample.com
        -s - path to your ssh public key (not priv!)
        -e - path to a html file that will be served from the LB when all pool members are down
        -c - name of cloud files container-name to store the backup data
        -i - specify image-id or use the default for Ubuntu 10.04
        -f - specify flavor-id or use the default for the smallest cloud server

# python challenge10.py -d -n www.challenge10.myrado.net
[22:50:46] Checked your ssh public key /root/.ssh/id_rsa.pub, ok
[22:50:46] Checked your error page error.html, ok
[22:50:50] Deleted existing cloud server www.challenge10.myrado.net-0
[22:50:54] Deleted existing cloud server www.challenge10.myrado.net-1
[22:50:54] Checked your existing cloud server www.challenge10.myrado.net-*, ok
[22:50:56] Deleting existing domain challenge10.myrado.net
[22:50:57] Checked your FQDN www.challenge10.myrado.net, ok
[22:50:57] Checked your DNS domains challenge10.myrado.net, there is none, ok
[22:50:57] Deleted container www.challenge10.myrado.net
[22:50:57] Checked your container www.challenge10.myrado.net where we are going to keep backup data, ok
[22:50:59] Checked your lb www.challenge10.myrado.net, there is none, ok
[22:51:01] Checked your image id d4c7b93d-9f18-45dc-aa7c-3e3b126e3792, ok
[22:51:01] Checked your flavor id 2, ok
[22:51:01] Building 2 cloud servers
.
.
.
.
[22:53:37] Building and configuring lb ...
.
.
.
.
[22:54:15] Building and configuring dns domain ...
[22:54:18] Backuping files to cloud files ...
[22:54:19] ----------------------------------------------------------------------
[22:54:19] vip name www.challenge10.myrado.net and ip 162.13.24.117
[22:54:19] cloud server www.challenge10.myrado.net-0 added to pool as 10.179.73.200
[22:54:19] cloud server www.challenge10.myrado.net-1 added to pool as 10.179.73.165
[22:54:19] Error page is stored in container www.challenge10.myrado.net under name error.html
[22:54:19] to check if the config works try to: curl -v http://162.13.24.117

These are some steps to verify the config.

 
# dig +short @69.20.95.4 www.challenge10.myrado.net
162.13.24.117

# nova list | grep challenge10                                                                          23:03:28
| 35e831a5-cfc8-44a7-98ff-1a003cc30901 | www.challenge10.myrado.net-0 | ACTIVE | public=2a00:1a48:7805:0113:8cfc:cf10:ff08:21ff, 95.138.174.137; private=10.179.73.200 |
| 06259b99-33d0-4537-94ab-e1be032337aa | www.challenge10.myrado.net-1 | ACTIVE | public=2a00:1a48:7805:0113:8cfc:cf10:ff08:3c44, 95.138.174.236; private=10.179.73.165 |

# curl -v http://162.13.24.117
* About to connect() to 162.13.24.117 port 80 (#0)
*   Trying 162.13.24.117... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 162.13.24.117
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Thu, 25 Apr 2013 23:26:45 GMT
< Connection: close
< Content-Type: text/html
<                                                                                                                                   23:28:27
* Closing connection #0
<html>
  <head>
    <title>Challenge 10 - Default error page on LB</title>
  </head>
  <body> Sorry but all pool members failing health checks. </body>
</html>

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge10

Challenge 9 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 9 script

Below is the output and results from script #9 from the api-challenge.
The script checks if the provided parameters are correct and doesn't collide with existing cloud objects.
First it creates cloud server. Once it is built it maps its public IP in DNS A record for new domain it created.

 
# python challenge9.py

    usage: challenge9.py [-h] [-v] -n FQDN-name -i image-id -f flavor-id
        -h - usage help
        -v - verbose / debug output
        -n - FQDN name
        -i - proper image id
        -f - flavor id

# python challenge9.py -n www.challenge9.myrado.net -i 88130782-11ec-4795-b85f-b55a297ba446 -f 2
[19:54:23] Creating new DNS zone challenge9.myrado.net
[19:54:24] Creating cloud server ...(please wait the build can take a while)
.
.
.
.
.
.
.
.
.
[19:59:34] DNS zone challenge9.myrado.net has been udpated and a new A record created
[19:59:34] rec A: www.challenge9.myrado.net -> 95.138.189.73

These are some steps to verify the config.

 
# dig +short @69.20.95.4  www.challenge9.myrado.net
95.138.189.73

# nova list | grep www.challenge9.myrado.net
| e7c3a258-13ff-426b-8c2d-d24649943947 | www.challenge9.myrado.net | ACTIVE | public=2a00:1a48:7805:0111:8cfc:cf10:ff08:3ac6, 95.138.189.73; private=10.178.199.189  |

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge9

Challenge 8 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 8 script

Below is the output and results from script #8 from the api-challenge.
The script checks that there isn't cloud files challenge8 container and no DNS domain rado-challenge.org.
As the script runs it create the container and DNS domain.
It uploads index.html file to the container and sets the meta data according to Serve A Static Website Fast, Without Servers.
Once the job is done it prints some info.

 
$ python challenge8.py
[16:26:00] Creating container challenge8 and uploading small index.html file.
[16:26:00] Creating DNS domain rado-challenge.org and setting CNAME records
[16:26:03] Static site details: (make sure you point your DNS to Rackspace Cloud DNS servers):
[16:26:03] ping challenge8.rado-challenge.org
[16:26:03] curl -v http://challenge8.rado-challenge.org/index.html
[16:26:03] curl -v http://8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com

With the default configuration on any of the cloud servers we will not be able to fully test it. Our domain is not registered anywhere and Rackspace cloud servers don't recursively resolve our CNAME record.

These are some testing to see how it works.

 
$ cat /etc/resolv.conf
# Automatically generated, do not edit
nameserver 83.138.151.81
nameserver 83.138.151.80

$ ping challenge8.rado-challenge.org   
ping: unknown host challenge8.rado-challenge.org

# the 65.61.188.4 is Rackspace cloud dns server (Google it)
# dig +short @65.61.188.4 challenge8.rado-challenge.org                   
8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com.

# curl -v http://8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com     
* About to connect() to 8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com port 80 (#0)
*   Trying 2001:668:108:b::216:3f40... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Last-Modified: Wed, 24 Apr 2013 16:25:58 GMT
< ETag: 60098552c51f8570fe0e59982f6a0869
< X-Timestamp: 1366820758.32217
< Content-Type: text/html
< X-Trans-Id: tx703c6bed688e40dcb184b3d5e3d6b084
< Cache-Control: public, max-age=900
< Expires: Wed, 24 Apr 2013 17:32:21 GMT
< Date: Wed, 24 Apr 2013 17:17:21 GMT
< Content-Length: 28
< Connection: keep-alive
<
Hello challenge8, it works!
* Connection #0 to host 8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com left intact
* Closing connection #0

root@server:~# cat /etc/resolv.conf 
# Automatically generated, do not edit
nameserver 69.20.95.4
#nameserver 83.138.151.81
#nameserver 83.138.151.80

# dig challenge8.rado-challenge.org  

; <<>> DiG 9.8.1-P1 <<>> challenge8.rado-challenge.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2047
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;challenge8.rado-challenge.org. IN      A

;; ANSWER SECTION:
challenge8.rado-challenge.org. 300 IN   CNAME   8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com.

;; Query time: 108 msec
;; SERVER: 69.20.95.4#53(69.20.95.4)
;; WHEN: Wed Apr 24 17:22:25 2013
;; MSG SIZE  rcvd: 134

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge8
2. http://serverfault.com/questions/196048/what-would-be-causing-a-recursion-requested-but-not-available-error-using-dbnd

How to find your public ip from bash

There are many sites that can show you your current public IP. But only a few of them are designed in a way so you can interact with them from command line using curl for example.

Below is a nice and elegant way to find out your public IP using curl

 
alias myip='curl --silent checkip.dyndns.org | egrep --only-matching "[0-9\.]+"'

root@server:~# myip
1.79.21.123

References

1. https://github.com/pex/config/blob/master/bash/shortcuts.sh
2. https://github.com/rtomaszewski/dotfiles
3. http://dotfiles.github.io/

Challenge 7 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 7 script

Below is the output and results from script #7 from the api-challenge.
The script first searches for an existing load balancer with a name of challenge7-vip. If it finds it stops. You have to delete the lb first.
If there is not lb with this name it start of creating 2 new cloud servers with names challenge7-0 and challenge7-1.
After the cloud servers are built it create a new LB and adds the servers as node.
At the end it prints some info what it did.
Neither the cloud servers or the lb are deleted after the script run.

 
root@manage2:~/api-challenge# python challenge7.py
[23:50:36] found lb challenge7-vip, please remove it before reruning the script

    usage: challenge7.py [-h] [-v]
        -h - usage help
        -v - verbose / debug output

$ python challenge7.py 
[23:39:31] Building 2 cloud servers
.
.
.
.
.
.
[23:43:13] Building and configuring lb ...
.
.
.
.
[23:43:39] lb name : challenge7-vip 
[23:43:39] lb status : ACTIVE 
[23:43:39] lb created : {u'time': u'2013-04-23T23:43:12Z'} 
[23:43:39] lb virtual_ips : [<VirtualIP type=PUBLIC, id=2525, address=5.79.37.137 version=IPV4>] 
[23:43:39] lb port : 80 
[23:43:39] lb protocol : HTTP 
[23:43:39] lb algorithm : RANDOM 
[23:43:39] lb nodeCount : 2 
[23:43:39] Node: {'port': 80, 'condition': 'ENABLED', 'address': u'10.179.6.186'}
[23:43:39] Node: {'port': 80, 'condition': 'ENABLED', 'address': u'10.179.5.147'}

With the default configuration we put in place we can run only a simple test. You can see that the lb VIP is responding to pings but not for HTTP requests. This is expected behavior as our cloud servers are not configured and there is not Apache listening on port 80.

 
# ping 5.79.37.137
PING 5.79.37.137 (5.79.37.137) 56(84) bytes of data.
64 bytes from 5.79.37.137: icmp_req=2 ttl=61 time=0.657 ms

# curl -v 5.79.37.137                                                                                  23:57:09
* About to connect() to 5.79.37.137 port 80 (#0)
*   Trying 5.79.37.137... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 5.79.37.137
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Tue, 23 Apr 2013 23:55:36 GMT
< Connection: close
< Content-Type: text/html
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<title>Service Unavailable</title>
<style type="text/css">
body, p, h1 {
  font-family: Verdana, Arial, Helvetica, sans-serif;
}
h2 {
  font-family: Arial, Helvetica, sans-serif;
  color: #b10b29;
}
</style>
</head>
<body>
<h2>Service Unavailable</h2>
<p>The service is temporarily unavailable. Please try again later.</p>
</body>
</html>
* Closing connection #0

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge7
2. https://github.com/rackspace/pyrax/tree/master/samples/cloud_loadbalancers

Challenge 6 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 6 script

Below is the output and results from script #6 from the api-challenge.
It search for the container name you provide as argument.
It enables CDN feature on the container.
It prints example files from the container for testing.

 
$ python challenge6.py
[16:42:12] Can't find container name None under your cloud account account_name

    usage: challenge6.py [-h] [-v] container-name
        -h - usage help
        -v - verbose / debug output

        container-name - name of the container

$ python challenge3.py tmp/challenge3-test.d/ mycont
$ python challenge6.py -v mycont
[17:16:47] --------------------------------------------
[17:16:47] Container mycont info before enabling CDN:
[17:16:48] cdn_enabled False
[17:16:48] cdn_ttl 86400
[17:16:48] cdn_uri None
[17:16:48] cdn_ssl_uri None
[17:16:48] cdn_streaming_uri None
[17:16:48] cdn_ios_uri None
[17:16:51] --------------------------------------------
[17:16:51] Container info after enabling CDN:
[17:16:51] cdn_enabled True
[17:16:51] cdn_ttl 1200
[17:16:51] cdn_uri http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com
[17:16:51] cdn_ssl_uri https://fb1002c7917e5ba2e2ec-2d8e05068e4fb2e0d8a6741787477e59.ssl.cf3.rackcdn.com
[17:16:51] cdn_streaming_uri http://4f23bda39d9e91b1dce8-2d8e05068e4fb2e0d8a6741787477e59.r33.stream.cf3.rackcdn.com
[17:16:51] cdn_ios_uri http://9ccd8d607dacdd984e50-2d8e05068e4fb2e0d8a6741787477e59.iosr.cf3.rackcdn.com
[17:16:51] --------------------------------------------
[17:16:51] Example files from the container mycont
[17:16:51] ============================================
[17:16:51] ch3f1.txt
[17:16:51] http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com/ch3f1.txt
[17:16:51] https://fb1002c7917e5ba2e2ec-2d8e05068e4fb2e0d8a6741787477e59.ssl.cf3.rackcdn.com/ch3f1.txt
[17:16:51] ============================================
[17:16:51] ch3f2.txt
[17:16:51] http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com/ch3f2.txt
[17:16:51] https://fb1002c7917e5ba2e2ec-2d8e05068e4fb2e0d8a6741787477e59.ssl.cf3.rackcdn.com/ch3f2.txt

After we enabled the CDN on the container level we can openly access our files using the above links

 
$ curl -o file.txt -v http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com/ch3f1.txt
* About to connect() to bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com port 80 (#0)
*   Trying 2001:668:108:b::216:3f19...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
> GET /ch3f1.txt HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 31
< Accept-Ranges: bytes
< Last-Modified: Tue, 23 Apr 2013 17:13:48 GMT
< ETag: 7f80594ffedbf15e5ade5f540eff6770
< X-Timestamp: 1366737228.82321
< Content-Type: text/plain
< X-Trans-Id: tx5109b62029b44600afbf6274e4bb9699
< Cache-Control: public, max-age=1187
< Expires: Tue, 23 Apr 2013 17:36:49 GMT
< Date: Tue, 23 Apr 2013 17:17:02 GMT
< Connection: keep-alive
<
{ [data not shown]
100    31  100    31    0     0    112      0 --:--:-- --:--:-- --:--:--   306
* Connection #0 to host bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com left intact
* Closing connection #0

$ cat file.txt

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge6
2. http://rtomaszewski.blogspot.co.uk/2013/04/challenge-3-script.html
3. https://github.com/rackspace/pyrax/blob/master/samples/cloudfiles/container_cdn.py

How to merge changes from a custom branch into master branch on github

As per our release policy for the api-challenge we have created a new branch called challenge5 to release the code that implements task #5. After the initial release we found some bugs. We fixed them and committed the changes back to our challenge5 repository using this method How to modify files in your branch on github. These are the changes made to the challenge5 branch:

https://github.com/rtomaszewski/api-challenge/commit/7872d14cc22e96b3b9f2049ae0d76a7697af22c6

Problem

The changes made on the challenge5 branch are not visible in master repository. How to copy the changes from challenge5 to master branch.

Solution

To copy the changes from one branch to another you can use github merging feature.

 
PS > git status .\challenge5.py
# On branch master
nothing to commit (working directory clean)

PS > git branch
  challenge2
  challenge3
  challenge4
  challenge5
* master

PS> git merge challenge5
Updating d609784..7872d14
Fast-forward
 challenge5.py |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

PS > git status .\challenge5.py
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
nothing to commit (working directory clean)

PS > git push
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:rtomaszewski/api-challenge
   d609784..7872d14  master -> master

References

1. http://learn.github.com/p/branching.html

Challenge 5 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 5 script

Below is the output and results from script #5 from the api-challenge.
The script first do some verification if the provided or the default options are correct.
It verifies if a data base instance exists and if run with option '-i' deletes it.
If there is no db instance it will create a new one.
Before proceeding it waits for the instance to be built. It can take some minutes (on average about 2 min).
Once the instance is created it connects to it and creates databases.
Once the data base(s) are created it creates user(s) that can connect to them.
At the end the script is not deleting the created db instance, databases or users. It has to be deleted manually.

 
$ python challenge5.py -h

    usage: challenge5.py [-h] [-v] [ -i | -d ] [ -u db-user ] [ instance-name ] [ db-name1 [db-name2] ]
        -h - usage help
        -v - verbose / debug output
        -i - if instance exists delete it and all its databases before proceeding
        -d - delete database(s) from under the instance

        instance-name - create instance with this name
        db-nameX - name of the database
        db-user - create user that can access the database; by default a user name is equal to db-name

$ python challenge5.py -i
[12:12:30] Found existing db instance challenge5-inst1, deleting it ...
challenge5-user2']
[12:12:35] Creating instance challenge5-inst1
[12:12:36] Waiting for the instance to be built ...
.
.
.
[12:14:38] Instance is created on host ad5e20f3ed07b5ab7592ede01f15316ef05fb0e2.rackspaceclouddb.com
[12:14:39] Creating databases ['challenge5-db1', 'challenge5-db2'] under the instance challenge5-inst1
[12:14:39] creating db: challenge5-db1
[12:14:40] creating db: challenge5-db2
[12:14:40] Creating users ['challenge5-user1', 'challenge5-user2']
[12:14:41] Created user challenge5-user1 pass pass123@ in db ['challenge5-db1', 'challenge5-db2']
[12:14:42] Created user challenge5-user2 pass pass123@ in db ['challenge5-db1', 'challenge5-db2']

Once the database is created we can connect to it from any existing cloud servers.

 
root@server:~/# mysql -h 71b4042b25136f6337ca013bb8525afae87a75a0.rackspaceclouddb.com --user=challenge5-user1 --password=pass123@ challenge5-db1
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 52
Server version: 5.1.66-0+squeeze1 (Debian)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| challenge5-db1     |
| challenge5-db2     |
+--------------------+
3 rows in set (0.00 sec)

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge5
2. https://github.com/rackspace/pyrax/tree/master/samples/cloud_databases
3. http://www.rackspace.com/knowledge_center/article/cloud-databases-how-to-articles-other-resources
4. http://docs.rackspace.com/cdb/api/v1.0/cdb-devguide/content/POST_createInstance__version___accountId__instances_.html
5. http://docs.rackspace.com/cdb/api/v1.0/cdb-devguide/content/POST_createDatabase__version___accountId__instances__instanceId__databases_.html
6. http://docs.rackspace.com/cdb/api/v1.0/cdb-devguide/content/POST_createUser__version___accountId__instances__instanceId__users_.html

Challenge 4 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 4 script

Below is the output and results from script #4 from the api-challenge.
The script first checks if a domain exists. By default it will terminated but if you specify the -d options it will delete the existing domain before proceeding.
Next adds a new A record according to the provided arguments.
Once the A record is added it tries to find the domain again and prints all info about it.
After it finishes it doesn't delete the created domain.

 
$ python challenge4.py
[18:58:06] missing params

    usage: challenge4.py [-h] [-v] [-d] FQDN ip-add
        -h - usage help
        -v - verbose / debug output
        -d - if a domain exists it will be deleted first

        FQDN - Fully Qualified Domain Name; exaple www.wikipedia.org
        ip-add - ip address

$ python challenge4.py challenge4.rado.com 100.2.3.4
[18:45:47] Domain created: challenge4.rado.com
[18:45:47] Adding records to our domain ...
[18:45:49] domain name: challenge4.rado.com created at 2013-04-22T18:44:06.000+0000
[18:45:50] record   A challenge4.rado.com -> 100.2.3.4
[18:45:50] record  NS challenge4.rado.com -> dns1.stabletransit.com
[18:45:50] record  NS challenge4.rado.com -> dns2.stabletransit.com

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge4
2. https://github.com/rackspace/pyrax/tree/master/samples/cloud_dns
3. http://www.rackspace.com/knowledge_center/article/troubleshooting-dns-with-dig
4. http://rtomaszewski.blogspot.co.uk/2013/04/how-to-paste-into-python-interpreter.html

How to paste into Python interpreter a code snippet for testing

While working on a bigger Python code module you would like to test a small part of it quickly. When you try to copy it directly to your Python interpreter (python, bpython, ipython) you get IndentationError: unexpected indent or SyntaxError: invalid syntax error messages.

Problem

How to copy a code snippet into ipython.

Solution

Example 1

 
a=True
if a :
    print "true"

else:
    print "false"

Example 2

 
class Test:
    def f(self):
        print("hello f")

    def f2(self):
        print("hello f2")

c=Test()
c.f()

To test example 1 or 2 code we are going to use ipython. To instruct the interpreter that it shouldn't  auto-format or auto-indent we use the %cpaste magic. An example below shows how to use it.

 
user@server:# ipython
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
Type "copyright", "credits" or "license" for more information.

IPython 0.12.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:a=True
:if a :
:    print "true"
:
:else:
:    print "false"
:
:--

true

References

1. http://stackoverflow.com/questions/10886946/how-does-ipythons-magic-paste-work

Practical tutorial on Python generators

Slides are usually very generic and lacking a lot of details. But the one below are of an extraordinary quality and tells you all what you need to know about Python generators. It is organized as a practical guide on how to write programs for data manipulation and processing using generators.

For these more familiar with design pattern is shows an idea on how to implement a chain of responsibility patter by using generator as a building block.

Python Generator Hacking from David Beazley (Dabeaz LLC)

References

1. https://python-3-patterns-idioms-test.readthedocs.org/en/latest/FunctionObjects.html#chain-of-responsibility
2. http://www.oodesign.com/chain-of-responsibility-pattern.html
3. http://java.dzone.com/articles/design-patterns-uncovered-chain-of-responsibility
4. http://java.dzone.com/articles/simple-example-illustrate
5. http://java.dzone.com/articles/two-java-implmentations-chain

Challenge 3 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 3 script

Below is the output and the results for the script #3 from the api-challenge.
The script first checks if the provided directory exists.
Next checks if there is a container with the name you provided.
If there is not it creates a new container.
It uploads all files and directories from the directory dir-name to cloud files and saves them in the container container-name.
After it finishes it neither deletes the local directory or the container it created.

 
$ python challenge3.py
[09:47:54] missing params

    usage: challenge3.py [-h] [-v] dir-name container-name
        -h - usage help
        -v - verbose / debug output
        dir-name - a path to a directory
        container-name - upload the dir-name to this cloud container

$ python challenge3.py tmp/challenge3-test.d/ cfdir
[09:47:01] Uploading directory tmp/challenge3-test.d/ to container cfdir
[09:47:01] container cfdir don't exists, create a new one ...
Total bytes to upload: 181
Progress: 0.00%
Progress: 16.57%
Progress: 73.48%
Progress: 100.00%     

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge3
2. https://github.com/rackspace/pyrax/tree/master/samples/cloudfiles

How to develop and debug programs that use pyrax or python-novaclient libraries

Python belongs to a category of high-level, dynamically typed programming languages. This has enormous impact on the way how you write the code and how it is interpreted and executed.

As an example let's take a look at Python Rackspace pyrax module. It was built and made available in a form of Python SDK for Rackspace cloud. Under the hood it is a wrapper and integrator of various python libraries that exists to interact with Openstack systems(cloud servers, files, networks, databases ...). As an example it uses python-novaclient library communicate and interact with Nova (Openstack Compute).

From developer point of view you can write a Python program to interact with Rackspace cloud using any of them. Both should work fine.

Working with any of these libraries you are going to face these challenges very soon:

• where is the latest API documentation 
• how well are function documented
• for a function what arguments it accepts and what it returns

In comparison to other statically typed languages like Java you quickly realize that Python definition of a function doesn't' fully answer the above questions. Below are couple of tricks to help you start with to play with them although.

Problem 1

How to start interactive Python session with loaded pyrax module for testing

We are going to use bpython that has a nice feature allowing it to show the available functions and variables a Python objects implements (for these who are interested how this works you can read about introspection here and there)

1. Create a file with credentials

 
root@server:~# cat > rackspace_cloud_credentials
[rackspace_cloud]
username = user
api_key = key



2. Create an auxiliary file for bpython

 
# https://github.com/rtomaszewski/api-challenge/blob/master/bpython-pyrax.py
cat > bpython-pyrax.py
import os
import sys
import json
import re
import time
import datetime, time

from pprint import pprint
from pprint import pformat

import pyrax

creds_file = os.path.expanduser("~/rackspace_cloud_credentials")
pyrax.set_credential_file(creds_file, "LON")
cs = pyrax.cloudservers

help_str="""
the pyrax module has been loaded, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account 
l=cs.servers.list()
print(l)

# example 2: show list of images 
pprint(cs.images.list())
"""

print(help_str)



3. Open the bpython interpreter

 
root@server:~# bpython -i  bpython-pyrax.py

the pyrax module has been loaded, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account
l=cs.servers.list()
print(l)

# example 2: show list of images
pprint(cs.images.list())

>>>



4. Test how it works

Problem 2

How to start interactive Python session with loaded python-novaclient for testing

1. Create bash file with variables

 
cat > nova.vars.sh
export OS_AUTH_SYSTEM="rackspace_uk"
export OS_AUTH_URL="https://lon.identity.api.rackspacecloud.com/v2.0/"
export OS_NO_CACHE="1"
export OS_PASSWORD="pass"
export OS_REGION_NAME="LON"
export OS_TENANT_NAME="user"
export OS_USERNAME="user"
export NOVA_RAX_AUTH=1
export OS_PROJECT_ID="user"



2. Create an auxiliary file for bpython

 
# https://github.com/rtomaszewski/api-challenge/blob/master/bpython-novaclient.py
cat > bpython-novaclient.py
import os
import sys
import json
import re
import time
import datetime, time

from pprint import pprint
from pprint import pformat

from novaclient.v1_1 import client

os.environ["OS_USERNAME"]
os.environ["OS_PASSWORD"]
os.environ["OS_NO_CACHE"]
os.environ["OS_TENANT_NAME"]
os.environ["OS_AUTH_URL"]
os.environ["OS_REGION_NAME"]
os.environ["OS_AUTH_SYSTEM"]
os.environ["NOVA_RAX_AUTH"]
os.environ["OS_PROJECT_ID"]

from novaclient import auth_plugin as _cs_auth_plugin
_cs_auth_plugin.discover_auth_systems()
auth_plugin = _cs_auth_plugin.load_plugin("rackspace_uk")

cs = client.Client(os.environ["OS_USERNAME"], os.environ["OS_PASSWORD"], os.environ["OS_TENANT_NAME"], auth_url=os.environ["OS_AUTH_URL"], auth_system="rackspace", region_name="LON",  service_type="compute", auth_plugin=auth_plugin)

help_str="""
the novaclient module has been loaded and INITIATED, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account 
l=cs.servers.list()
print(l)

# example 2: show list of images 
pprint(cs.images.list())
"""

print(help_str)



3. Open the bpython interpreter

 
root@manage2:~# bpython -i bpython-novaclient.py

the novaclient module has been loaded and INITIATED, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account
l=cs.servers.list()
print(l)

# example 2: show list of images
pprint(cs.images.list())
>>>



4. Test how it works

References

1. http://bpython-interpreter.org/screenshots/
2. http://stackoverflow.com/questions/546337/how-do-i-perform-introspection-on-an-object-in-python-2-x
3. http://docs.rackspace.com/servers/api/v2/cs-gettingstarted/content/section_gs_install_nova.html
4. https://github.com/openstack/python-novaclient
5. https://github.com/rackspace/pyrax/blob/master/samples/cloudservers/create_server.py

bpython supports virtualenv

With the help of virtualenv you can create different virtual environments for Python. Each environment can have installed different modules or have the same module but in different version.

Problem

Does bpython support virtualenv by default.

Solution

Yes, bpython supports virtualenv environments by default. Below is how you can verify it.

 
root@manage2:~# python
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/pymodules/python2.7']
>>>

Virtualenv local configuration (see the (novaclinet) string representing the name of the environment at the beginning)

 
root@manage2:~/virtualenv.d/novaclient/bin# source  activate
(novaclient)root@manage2:~/virtualenv.d/novaclient/bin# python
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/root/virtualenv.d/novaclient/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg', '/root/virtualenv.d/novaclient/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/root/virtualenv.d/novaclient/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg', '/root/virtualenv.d/novaclient/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/root/virtualenv.d/novaclient/lib/python2.7', '/root/virtualenv.d/novaclient/lib/python2.7/plat-linux2', '/root/virtualenv.d/novaclient/lib/python2.7/lib-tk', '/root/virtualenv.d/novaclient/lib/python2.7/lib-old', '/root/virtualenv.d/novaclient/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/root/virtualenv.d/novaclient/local/lib/python2.7/site-packages', '/root/virtualenv.d/novaclient/lib/python2.7/site-packages']

Global, system wide configuration

 
root@manage2:~# bpython
>>> import sys
>>> sys.path
['', '/usr/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/pymodules/python2.7']
>>>

Virtualenv local configuration

 
(novaclient)root@manage2:~/virtualenv.d/novaclient/bin# bpython
>>> import sys
>>> sys.path
['', '/root/virtualenv.d/novaclient/bin', '/root/virtualenv.d/novaclient/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg', '/root/virtualenv.d/novaclient/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/root/virtualenv.d/novaclient/lib/python2.7', '/root/virtualenv.d/novaclient/lib/python2.7/plat-linux2', '/root/virtualenv.d/novaclient/lib/python2.7/lib-tk', '/root/virtualenv.d/novaclient/lib/python2.7/lib-old', '/root/virtualenv.d/novaclient/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/root/virtualenv.d/novaclient/local/lib/python2.7/site-packages']
>>>

How long does a link goes down when changing speed on an interface

The switch and server are configured properly. There is an active link but because of recent flapping issues the link auto negotiate a lower speed than it should be running on.

When hard coding the new speed as outlined below the link went down for almost 3 to 5 seconds.

 
telnet switchA
conf t
interface GigabitEthernet0/21
speed 1000
end
write mem

Logs from the C2960G Cisco switch:

 
Apr 18 14:39:13.177 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/21, changed state to down
Apr 18 14:39:14.175 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/21, changed state to down
Apr 18 14:39:16.835 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/21, changed state to up
Apr 18 14:39:17.841 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/21, changed state to u

Status of the port from before and after the change.

 
# before
#sh interfaces status | i 0/21
Gi0/21                       connected    413        a-full a-1000 10/100/1000BaseTX

# after
#sh interfaces status | i 0/21
Gi0/21                       connected    413          full   1000 10/100/1000BaseTX

How to modify files in your branch on github

You created and released a new branch following the RERO development philosophy paradigm. Later on you discover that there are some bugs that you would like to fix in your branch and publish.

Problem

How to fix a file in a branch.

Solution

We need to checkout the branch from repository.
Modify the file and fix the issue.
Commit the changes back to our branch.

An example session is showed below.

 
git checkout challenge2
#now you modify the file, once done
git commit -m 'fixed delete_images func' -a
git push origin

# after modification we want to return in default branch
git checkout master
git branch

Challenge 2 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 2 script

Below is the output and the results for the script #2 from the api-challenge.
The script first create a single new cloud server with a name web0.
Next it waits for the cloud server to be created. This behavior is similar to challenge #1
Once the cloud is built it create a backup under the name web0-img, by cloning the live cloud server.
Once the image is saved it uses it to spin up a new cloud server with a name web0_m.dd.hh.mm.ss.
Once the new cloud image is built it shows the summary of the created objects.
Before the end of the work the created objects (2 cloud server, 1 image) are deleted.

 
$ python.exe challenge2.py
[00:41:59] Building 1 cloud server(s).
[00:41:59] building web0 cloud server
[00:42:00] Waiting for the servers to be built ...
.
.
.
.
.
.
.
.
.
.
[00:47:36] Preparing to clone 1 server(s) ...
[00:47:36] Creating cloud image file: 'web0-img' from web0 cloud server
[00:47:38] Waiting for the clone images to be created ...
.
.
.
[00:49:41] Building 1 new cloud server(s) from taken images.
[00:49:41] building new web0_4.17.1.49.41 cloud server from image web0-img
[00:49:42] Waiting for the servers to be built from the cloned images ...
.
.
.
.
.
.
[00:53:16] Created cloud server details:
Server # 0:  ID  fd9b996c-d034-4aca-9c73-2ef11b31707d name             web0 IP     162.13.2.151 password Gz7CjrjNtNU7
[00:53:16] Created images details:
Image  # 0:  ID  a0c1b05a-3ab0-414e-8d3c-663f58f5ab94 name         web0-img from server fd9b996c-d034-4aca-9c73-2ef11b31707d
[00:53:16] Created cloned server details:
Server # 0:  ID  de18d5f6-bb97-4e5f-a315-53698edffd0e name web0_4.17.1.49.41 IP     162.13.2.231 password FDoC8mMHtppo

References

1. https://github.com/rtomaszewski/api-challenge/tree/challenge2

Anonymous function in Python

Python is one of the very popular languages, especially when it comes to scripting and  automations. But it is a general purpose language whose syntax is able to express constructs from procedural or object or even functional languages.

Today we are going to look at the lambda syntax and how we can use it to generate on demand dynamic anonymous function objects. Some examples below.

def square_root(x): return math.sqrt(x)

We can code it like this as well

 
square_root = lambda x: math.sqrt(x)

Other simple examples

 
sum = lambda x, y:   x + y   #  def sum(x,y): return x + y

More complex example when the resulting code is more readable and cleaner

 
def map(f, s):
    result = []
    for x in s:
            result.append(f(x))
    return result

def square(x):
    return x*x

vals = [1, 2, 3, 4]
newvals = map(square,vals)

The same with the lambda symbol.

 
newvals = map( (lambda (x): (x * x)), [1, 2, 3, 4] )

References

1. http://pythonconquerstheuniverse.wordpress.com/2011/08/29/lambda_tutorial/
2. http://python-history.blogspot.co.uk/2009/04/origins-of-pythons-functional-features.html

Howto generate simple network diagrams

A picture is worth more than 1000 words one of the proverbs says. This is absolute truth when you  troubleshoot and need to understand a customer network environmental  Below is a very simple recipe (unfortunately) how to generate a network diagram with a tool called nwdiag.

Instalation

 
aptitude install python-dev
pip install nwdiag

# will generate a png file diag1.png
nwdiag  diag1.conf

Example 1: FW with one segment

 
{
  public [shape = cloud];
  public -- firewall;

  network inside {
      address = "192.168.100.x/24"

      firewall [address = "192.168.100.1"];
      web01 [address = ".11"];
      cachesrv [address = ".51"];
      db1 [address = ".101"];
  }
}

Example 2: FW with dmz and inside segment

 
{
  public [shape = cloud];
  public -- firewall;

  network dmz {
      address = "192.168.200.x/24";

      firewall [address = "192.168.200.1"];
      web01 [address = ".11"];
      cachesrv [address = ".51"];
  }
  network inside {
      address = "192.168.100.x/24"

      firewall [address = "192.168.100.1"];
      db1 [address = ".101"];
  }
}

How to enable IPMI settings in BIOS on Tyan S8225 motherboard

Often a server that supports IPMI supports as well remote KVM desktop like iKVM. The instruction below describe how to enable IPMI in BIOS for the Tyan S8225 motherboard.

1. Connect keyboard and monitor to our system. Before we can mange our server remotely we need to first set things up using a standard method
2. Get into BIOS





3. Go to advance settings





4. Set a fix IP for your IPMI console on the server








5. Enable remote access





6. Set a primary Video adapter to VGA if you have additional graphic card on broad 





7. Save BIOS changes and start your server
8. When the system boots you should see similar outputs








9. From another PC try to connect to your remote IPMI session by typing http://192.168.0.200 in the browser

The default user/password is root/superuser





10. Open remote iKVM session and if necessary reboot the Tyan server 





11. For better graphic results you can set this options in the JViewer

Howto start Ubuntu installation over IPMI session

A decent IPMI implementation comes not only with an iKVM  (KVM over IP) features but allows you as well to remotely mount local devices.

The software version available for mainboard Tyan S8225 support the following options from the JViewer terminal:

To start a remote installation from a downloaded ISO image all what you need to do is to mount it reboot t he system. Once the system starts it will try to boot from the ISO.

References

http://www.servethehome.com/tyan-ipmi-20-remote-management-webgui-tour/

Cisco Nexus NX-OS operating systems has a build in Python shell

The world is changing and Cisco doesn't want to stay behind. In its new NX-OS os that is targeting data centre and service provider customer they build in a python shell: Python API

More info about the Cisco Nexus can be found here:
http://www.cisco.com/en/US/products/ps9494/Products_Sub_Category_Home.html

Example:

 
switch# show clock                                           
23:54:55.872 UTC Wed May 16 2012

switch# python                          !-- Enter Python interpreter
switch# >>> cli("conf term ; interface loopback 1")          
switch(config-if)# >>> cli("ip address 1.1.1.1/24")          
switch(config-if)# >>> cli("exit")      !-- Exit the CLI interface mode
switch(config)# >>> cli("exit")                              
switch# >>> i=0                                              
switch# >>> while i<8:                                       
switch# ...   i=i+1                     !-- Composite command; prompt indicates more input
switch# ...   cmd = "show module %i" % i                     
switch# ...   r=clid(cmd)                                    
switch# ...   if "TABLE_modinfo/model" in r.keys():          
switch# ...     if r["TABLE_modinfo/model"] == "Nurburgring":
switch# ...       print "got a racer in slot %d" % i         
switch# ...                             !-- Empty input indicates end of loop
got a racer in slot 3                                        
switch# >>> exit                        ! -- Exit Python interpreter          
switch#

How to simulate F5 health check requests with empty Host header

The curl tool is one of the tool you use on a daily basis when verifying and checking health checks on the F5 load balancer. But it has its limitations. One of these is that it don't insert an empty Host header.

 
# curl -v -H 'Host:' http://192.168.1.138 -o tmp
* About to connect() to 192.168.1.138 port 80
*   Trying 192.168.1.138... connected
* Connected to 192.168.1.138 (192.168.1.138) port 80
> GET /id.aspx HTTP/1.1
> User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Accept: */*
>

This simple command will use nc tool to initiate a TCP session and sent the whole HTTP request as you define it in the string between the quotes. That way we can simulate an empty Host header if we need it. For more option how to troubleshoot monitoring issue check this one https://devcentral.f5.com/wiki/advdesignconfig.TroubleshootingLtmMonitors.ashx.

 
$ echo -e "GET / HTTP/1.1\r\nHost:\r\nConnection: Close\r\n\r\n" | tee req.txt | nc -v 192.168.1.186 80
$ cat req.txt
GET / HTTP/1.1
Host:
Connection: Close

Howto tune HTTP browser connection with Connection and Keep-Alive HTTP headers

I've come across this response today. This is diffident from all other I have seen so far. It uses the timeout and max options for the Keep-Alive header that should influences how the TCP connection should be handled.

 
HTTP/1.1 200 OK
Date: Wed, 10 Apr 2013 08:42:48 GMT
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Wed, 10 Apr 2013 08:42:48 GMT
Cache-Control: no-store,no-cache,must-revalidate
Expires: -1
Content-Length: 1567
Keep-Alive: timeout=5, max=10
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8

Looking at the recent HTTP 1.1 RFC rfc2616 we can see that this options is addressed in more details here:

The original HTTP/1.0 form of persistent connections (the Connection: Keep-Alive and Keep-Alive header) is documented in RFC 2068.

Looking at RFC 2068 we see that it specifies in the paragraph  19.7.1.1 The Keep-Alive Header

The Keep-Alive header itself is optional, and is used only if a parameter is being sent. HTTP/1.1 does not define any parameters.

It doesn't specify anything about max or timeout parameters  With Google help I found this RFC draft Hypertext Transfer Protocol (HTTP) Keep-Alive Header. In this one we can see that:

The value of the timeout parameter is a single integer in seconds.
The value of the max parameter counts the number of requests since the connection was created.

How to show Build Results panel in Sublime editor

If you like using Sublime editor when coding in Python you love the shortcut ctrl+b. It executes the python interpreter with your script as parameter and prints any stdout or error messages in a new Result panel at the bottom of the window. With a Esc you can instruct the editor to hide this new panel.

Unfortunately there isn't a default shortcut to bring the Result Panel back.

Problem

How to create a shortcut to open Tools -> Build Results -> Show Build Results.

From menu open Preferences -> Bindings -> User and copy this similar config there.

 
{ "keys": ["alt+b"], "command": "show_panel", "args": {"panel": "output.exec"} }

Now with every time you press alt+b the result panel will appear again with the previous output.

References

1. http://stackoverflow.com/questions/10674030/in-sublime-text-2-reopen-build-output