What cloud provider do you use

Every one knows that there is a number of cloud players there on the market. Some of them have been there from the beginning like Amazon some of the others are new like Google with the recent release of Google Compute Engine (GCE)[1].

These links gave you a list of some of the big names behind the ongoing cloud adoption and industry transformation:

• http://gigaom.com/cloud/amazon-is-no-1-whos-next-in-cloud-computing/
• http://www.networkworld.com/supp/2012/enterprise2/040912-ecs-iaas-companies-257611.html

But these information is only a listing of possible vendors to chose from. But how is the market shared between them. These questions are not answered there. There is a little bit of publicly available documents on the Internet that elaborate on it as well.

This is one of the links I found on the Openstack mailing list that give us a small inside view to this problem: Cloud tools survey

The most interesting results are listed in point # 4 and 5.

4. Which clouds do you use (choose all that are appropriate)?
Amazon: 47.4%
Rackspace: 36.8%
HP Cloud: 21.1%
Go Grid: 0%
Linode: 31.6%
Softlayer: 0%
Joyent: 0%
Azure: 5.3%
Google: 10.5%
Private Cloud: 42.1%
Other responses included AppFog, OrionVM (Australian provider), KT UCloud Biz (Korean provider), eNoCloud and Physical Servers.

5. How many cloud based servers do you manage (not physical servers)?
1-10: 45.5%
11-50: 27.3%
50-200: 9.1%
200-1000: 18.2%
1000+: 0%
A lot of fairly large deployments – many of them appear to be private clouds (who responded via the OpenStack list).

References

1. http://cloud.google.com/products/compute-engine.html

High Availability trends and architectures in cloud computing

Every now and than there is a shift and change in IT industry. The changes try to provide a solution to our old problems and try to as well as predict and introduce new ideas. Many times it is not only a change in hardware itself or a change in software only but a mixture of both.

The Cloud is today the buzz word that drives the changes and powers the transformation. In my attempts to embrace and understand what it is I have found couple if videos on YouTube that give us a very nice inside view into the cloud what it is, how this works and what ideas it brings with. 

This is a link to the repository of all videos: http://www.youtube.com/user/TheCloudcastNET

The one I particularly like because it is discussing concept of High Availability in Cloud:

How to execute remote commands from python over ssh connection

There are a number of possible solution for Python that allow you to execute a command remotely. Below is a list of libraries/modules I found when researching it.

1. paramiko
http://www.lag.net/paramiko/
http://pypi.python.org/pypi/paramiko/1.7.7.2


2. ssh module ( a wrapper for paramiko)
http://stackoverflow.com/questions/1939107/python-libraries-for-ssh-handling
http://media.commandline.org.uk/code/ssh.txt


3. pythion binding to C libssh library
http://www.no-ack.org/2010/11/python-bindings-for-libssh2.html


4. fabric
http://docs.fabfile.org/en/1.4.3/index.html>


5. python SSH module (base on the paramiko)
http://pypi.python.org/pypi/ssh/1.7.11 
https://github.com/bitprophet/ssh

An interesting introduction to the paramiko can be found here: SSH Programming with Paramiko

This is a simple python example as well.

cloudserver script doesn't work anylonger after installing novaclient

I have played with the FirsGen Rackspace cloud on my box. It was working fine. Some info about how to start with it can be found here rackspace-cloudservers .

At some point I have started using the Openstack nova tool to interact with the Rackspace NextGen cloud to run some tests. Once I was done I returned to the openstack tool and discoved that it stopped to work.

Problem

For every comamnd I run i got alwasys the same error message. An example output.

$ /usr/bin/cloudservers --username user --apikey 123  list
Traceback (most recent call last):
  File "/usr/bin/cloudservers", line 9, in module
    load_entry_point('python-cloudservers==1.0a5', 'console_scripts', 'cloudservers')()
  File "/usr/lib/pymodules/python2.7/cloudservers/shell.py", line 413, in main
    CloudserversShell().main(sys.argv[1:])
  File "/usr/lib/pymodules/python2.7/cloudservers/shell.py", line 127, in main
    args.func(args)
  File "/usr/lib/pymodules/python2.7/cloudservers/shell.py", line 279, in do_list
    print_list(self.cs.servers.list(), ['ID', 'Name', 'Status', 'Public IP', 'Private IP'])
  File "/usr/lib/pymodules/python2.7/cloudservers/shell.py", line 402, in print_list
    pt.printt(sortby=fields[0])
  File "/usr/local/lib/python2.7/dist-packages/prettytable.py", line 163, in __getattr__
    raise AttributeError(name)
AttributeError: printt

Solution

After debugin the prettytable.py code it turned out that there is not such a function like printt. Further research confirmed this [1]. The fix this I have changed the code of cloudserver module on my PC. The new code after changes is listed below.

# vim +399 /usr/lib/pymodules/python2.7/cloudservers/shell.py

def print_list(objs, fields):
    pt = prettytable.PrettyTable([f for f in fields], caching=False)
    pt.aligns = ['l' for f in fields]
    for o in objs:
        pt.add_row([getattr(o, f.lower().replace(' ', '_'), '') for f in fields])

    # pt.printt(sortby=fields[0])
    print pt.get_string(sortby=fields[0])

References

1. http://code.google.com/p/prettytable/issues/detail?id=14&q=printt
2. https://answers.launchpad.net/nova/+question/198709

Other examples where the code was broken as well

3. https://github.com/calebgroom/clb/issues/24
4. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=673790

Difference between class and instance variables in Python

Object oriented programming in Python can be tricky if your background comes from C++ or Java. First thing is that both C++ and Java are statically typed. This is different in Python that belongs to the class of dynamic scoped languages [4].

For Java and C++ for example if you forget to declare a variable before you try to access it it will generate an error during compilation phase. Below is an example code in Java that illustrates this.

Java

$ javac JavaExample.java 
JavaExample.java:7: cannot find symbol
symbol  : variable number
location: class JavaExample
        number=_number;
        ^
JavaExample.java:11: cannot find symbol
symbol  : variable number
location: class JavaExample
       System.out.println("[test " + number + "] list=%s");
                                     ^
2 errors

Python

When you start applying object oriented programming paradigms in Python and start to write classes and create instances you may encounter an interesting phenomena. The code below illustrate this.

The difference between the 2 classes is that in the class Test2ClassVariable we have defined this variable self.mylist=[] additionally.

When you run this code this is the output:

$ python object_variables_example.py
[test 1] list=one
[test 2] list=one two
[test 3] list=three
[test 4] list=four

The interesting part of the output is in line #3. The variable list has 2 entries where at first look we would expect it should have only one. Not knowing anything about objects in Python this would be a natrual intuition for everyone with Java or C++ background.

Explanation

We see this behavoiur because Python uses a concept of class and instance variables. For Java/C++ programmers a python class variable is simply a Java/C++ static class variable. In our example variable list is a class variable and it is shared between all class instances. That explains the output we see.

References

1. http://stackoverflow.com/questions/68645/static-class-variables-in-python
2. http://docs.python.org/tutorial/classes.html#instance-objects
3. http://en.wikipedia.org/wiki/Class_variable
4. http://en.wikipedia.org/wiki/Python_(programming_language)

Basic tutorial on how to use and debug Cloud API for the NextGene (Openstack based) or FirsGen (build by Mosso) Rackspace cloud infrastructure

Before we start writing code and play with API we have to install the necessary python libraries first. I used the Ubuntu 11.04 for my testing.

Tools installation

• Openstack API library

There is a packet with the Openstack libraries in repositories unfortunately this is not the latest code that is compatible with the NextGen Rackspace cloud release. We have to install the tools manually as described here [1].

# don't do this i Ubuntu 11.04 because it is based on older version of the library 

# and will not work

aptitude install python-novaclient

aptitude install python-setuptools
easy_install pip
pip install python-novaclient
pip install --upgrade  python-novaclient

# To verify that the files were installed 
find /usr -name novaclient
/usr/share/pyshared/novaclient
/usr/local/lib/python2.7/dist-packages/novaclient

• FirstGen

# to install the library
aptitude install python-rackspace-cloudservers

# to verify where the files were installed
find /usr -name cloudservers
/usr/share/pyshared/cloudservers
/usr/lib/pymodules/python2.7/cloudservers

Debugging Nova Openstack API

 As Rackspace lunched his cloud first in the USA and Europe is following a few weeks later the URL below is for a US base cloud account. More about the URLs can be found here [2].

nova_example.py

# cat nova_example.py 
import httplib2
httplib2.debuglevel = 1

OS_USERNAME="user"
OS_PASSWORD="api"
OS_AUTH_URL="https://identity.api.rackspacecloud.com/v2.0/"

from novaclient.v1_1 import client
nt=client.Client(OS_USERNAME,OS_PASSWORD,'',OS_AUTH_URL)
nt.flavors.list()

You can than run this like python -i nova_example.py or run python and copy the code into it like that:

$ python -u
>>> import httplib2
>>> httplib2.debuglevel = 1
>>> 
>>> OS_USERNAME="user"
>>> OS_PASSWORD="api"
>>> OS_AUTH_URL="https://identity.api.rackspacecloud.com/v2.0/"
>>> 
>>> from novaclient.v1_1 import client
>>> nt=client.Client(OS_USERNAME,OS_PASSWORD,'',OS_AUTH_URL)
>>> nt.flavors.list()
connect: (dfw.servers.api.rackspacecloud.com, 443)
send: 'GET /v2/672114/flavors/detail HTTP/1.1\r\nHost: dfw.servers.api.rackspacecloud.com\r\nx-auth-token: mytoken\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Tue, 07 Aug 2012 21:16:28 GMT
header: Content-Length: 2448
header: Content-Type: application/json
header: X-Compute-Request-Id: req-4e7dda9c-7eee-402f-acc8-b98dc08e21b5
header: Server: Jetty(8.0.y.z-SNAPSHOT)
[Flavor: 512MB Standard Instance, Flavor: 1GB Standard Instance, Flavor: 2GB Standard Instance, Flavor: 4GB Standard Instance, Flavor: 8GB Standard Instance, Flavor: 15GB Standard Instance, Flavor: 30GB Standard Instance]

Debugging FirstGen API

firstgen_example.py

# cat  firstgen_example.py
import httplib2
httplib2.debuglevel = 1

u='user'
k='api'

from cloudservers import CloudServers
cs=CloudServers(u,k)
cs.flavors.list()

An example output

$ python -u
>>> import httplib2
>>> httplib2.debuglevel = 1
>>>
>>> from cloudservers import CloudServers
>>> u='user'
>>> k='api'
>>> cs=CloudServers(u,k)
>>> cs.flavors.list()
connect: (lon.auth.api.rackspacecloud.com, 443)
send: 'GET /v1.0 HTTP/1.1\r\nHost: lon.auth.api.rackspacecloud.com\r\nx-auth-key: key\r\naccept-encoding: gzip, deflate\r\nx-auth-user: hugoalmeidauk\r\nuser-agent: python-cloudservers/1.0a1\r\n\r\n'
reply: 'HTTP/1.1 204 No Content\r\n'
header: Server: Apache/2.2.3 (Red Hat)
header: vary: X-Auth-User,X-Auth-Key,X-Storage-User,X-Storage-Pass
header: X-Storage-Url: https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_c99f3ebf-f27d-4933-94b7-9e9ebf2bc7cd
header: Cache-Control: s-maxage=60806
header: Content-Type: text/xml
header: Date: Tue, 07 Aug 2012 21:19:36 GMT
header: X-Auth-Token: token
header: X-Storage-Token: token2
header: X-Server-Management-Url: https://lon.servers.api.rackspacecloud.com/v1.0/10001641
header: Connection: Keep-Alive
header: X-CDN-Management-Url: https://cdn3.clouddrive.com/v1/MossoCloudFS_c99f3ebf-f27d-4933-94b7-9e9ebf2bc7cd
header: Content-Length: 0
connect: (lon.servers.api.rackspacecloud.com, 443)
send: 'GET /v1.0/10001641/flavors/detail?fresh HTTP/1.1\r\nHost: lon.servers.api.rackspacecloud.com\r\nx-auth-token: token\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-cloudservers/1.0a1\r\n\r\n'
reply: 'HTTP/1.1 203 OK\r\n'
header: Server: Apache-Coyote/1.1
header: vary:  Accept, Accept-Encoding, X-Auth-Token
header: Content-Encoding: gzip
header: Vary: Accept-Encoding
header: Last-Modified: Tue, 21 Jun 2011 21:09:45 GMT
header: X-PURGE-KEY: /flavors
header: Cache-Control: s-maxage=1800
header: Content-Type: application/json
header: Content-Length: 175
header: Date: Tue, 07 Aug 2012 21:19:36 GMT
header: X-Varnish: 1664913388 1664913324
header: Age: 44
header: Via: 1.1 varnish
header: Connection: keep-alive
[Flavor: 256 server, Flavor: 512 server, Flavor: 1GB server, Flavor: 2GB server, Flavor: 4GB server, Flavor: 8GB server, Flavor: 15.5GB server, Flavor: 30GB server]

References

1. http://docs.rackspace.com/servers/api/v2/cs-gettingstarted/content/section_gs_install_nova.html
2. http://docs.rackspace.com/servers/api/v2/cs-gettingstarted/content/section_gs_auth.html
3. http://pypi.python.org/pypi/python-novaclient
4. https://github.com/openstack/python-novaclient
5. http://www.rackspace.com/knowledge_center/article/cloud-servers-how-to-articles-other-resources

Links to Cloud provider web management consoles

For each cloud provider you have to learn what are the most important links and how to use their tools. Below is a short list in alfabetical order of these I came across.

• Amazon Web Service (AWS) management console

https://console.aws.amazon.com 

• HP Cloud based on Openstack management console 

https://console.hpcloud.com 

• Rackspace Cloud Control Panel (FirstGen Cloud) 

https://lon.manage.rackspacecloud.com 
https://manage.rackspacecloud.com 

• Rackspace Cloud Control Panel (NextGen Cloud base on Openstack)

https://mycloud.rackspace.com