How does the GIL affect my python program when threads are used

This is in relation to the previous post: "What does the GIL lock means when I use threads in Python" .

I have been reading more about the GIL problem in Python and found out that this limitation only exists for CPython. It is not present in the Java base implementation of the python interpreter [1].

Still not convinced how much it affects the execution time I run couple of tests.

A test program

Cloud server provisioning

I have provisioned a 32GB cloud server for the tests. These are some of the server details.

$ cat /proc/cpuinfo  | egrep 'proc|model name' | sort | uniq 
model name : Quad-Core AMD Opteron(tm) Processor 2374 HE
processor : 0
...
processor : 7

$ free -m
            total       used       free     shared    buffers     cached
Mem:         30041       8364      21676          0         34        580
-/+ buffers/cache:       7749      22291
Swap:        61436          0      61436

$ python -V
Python 2.7.3

$ jython -V
Jython 2.5.1+

$ ls -al /etc/alternatives/java
lrwxrwxrwx 1 root root 46 Jul 31 19:36 /etc/alternatives/java -> /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java

$ /usr/bin/java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.3) (6b24-1.11.3-1ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Test results

I run 4 tests. Two with the python and two with the jython interpreter. To start the tests i used:

$ /usr/bin/time -v jython gil_threads_test.py | tee log 
$ /usr/bin/time -v python gil_threads_test.py | tee log

The results for the python after filtering out the unnecessary lines ( results | grep -v ': 0' )

Test1 
  Command being timed: "python gil_threads_test.py"
  User time (seconds): 759.93
  System time (seconds): 516.46
  Percent of CPU this job got: 170%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 12:27.25
  Maximum resident set size (kbytes): 76240000
  Minor (reclaiming a frame) page faults: 5806046
  Voluntary context switches: 44574718 
  Involuntary context switches: 3311991
 
Test2
  Command being timed: "python gil_threads_test.py"
  User time (seconds): 760.37
  System time (seconds): 515.21
  Percent of CPU this job got: 170%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 12:30.16
  Maximum resident set size (kbytes): 76066992
  Minor (reclaiming a frame) page faults: 5817319
  Voluntary context switches: 44574153
  Involuntary context switches: 3318754
  Page size (bytes): 4096

The results for the jython after filtering out the unnecessary lines ( results | grep -v ': 0' )

Test3
  Command being timed: "jython gil_threads_test.py"
  User time (seconds): 955.74
  System time (seconds): 16.31
  Percent of CPU this job got: 397%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 4:04.75
  Maximum resident set size (kbytes): 30018032
  Minor (reclaiming a frame) page faults: 2034262
  Voluntary context switches: 83887
  Involuntary context switches: 110640
  File system outputs: 328
  Page size (bytes): 4096
  
Test4
  Command being timed: "jython gil_threads_test.py"
  User time (seconds): 984.16
  System time (seconds): 16.22
  Percent of CPU this job got: 385%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 4:19.69
  Maximum resident set size (kbytes): 30718496
  Minor (reclaiming a frame) page faults: 2070011
  Voluntary context switches: 97457
  Involuntary context switches: 109420
  File system outputs: 344
  Page size (bytes): 4096

Analisis and results description

• The execution of the test{1,2} was a lot longer than for test{3,4}. The CPython interpreter needed 12:27.25/12:30.16 versus 4:04.75/4:19.69 for the Jython.

• The gil_threads_test.py script CPU utilization in user space is about 20% higher for Jython and there is a big difference how much time the interpreters spent in kernel mode.
  
  Looking at the output from top during the tests we saw that java was able to use 100% all 7 CPUs where python managed only to use have utilization of 13-20%.
  
  The increaded number of context swithces for python has a direct impact on performance what the execution time results show clearly.

• One interesting thing. The memory usage is about 100% higher when python is used. Looks like java is able to better manged the memory allocation and memory management. But I'm not sure as well as how much we can trust the output from the utime -v command for this.

References

1. http://www.jython.org/jythonbook/en/1.0/Concurrency.html

What does the GIL lock means when I use threads in Python

Looking for a way to introduce parallelism to your code you are quickly going to find that there are two different solutions: either you can use threads or processes.

Although python supports both [1] and [2] models the support for threads has a following warning:

[1] CPython implementation detail: Due to the Global Interpreter Lock, in CPython only one thread can execute Python code at once.

It took me a while to understand what this actualy means. This is a serious limitation for a multithreaded programming and it still exists even in the latest python version 3.x [3]. To better understanding what this means this is an example from one of the links below I found:

The GIL enforces Python's requirement that only a single bytecode operation is executed at a time.

References

1. http://docs.python.org/library/threading.html
2. http://docs.python.org/library/multiprocessing.html#module-multiprocessing
3. http://docs.python.org/py3k/library/threading.html


4. Google search about GIL has many link, I personaly found these very useful
http://www.grouplens.org/node/244
http://linuxgazette.net/107/pai.html

The section "Write multithreaded or multiprocess code"
http://www.scipy.org/ParallelProgramming

http://stackoverflow.com/questions/10344529/is-the-max-thread-limit-actually-a-non-relevant-issue-for-python-linux
http://stackoverflow.com/questions/7053284/what-is-more-suitable-in-performance-aspect-multithreading-or-multiprocessing


5. These below are a little longer to read but they talk about the problem and around it on a very low level
http://www.jeffknupp.com/blog/2012/03/31/pythons-hardest-problem/
http://jessenoller.com/2009/02/01/python-threads-and-the-global-interpreter-lock/

How to clean and delete multiple cloud servers after a failed test

The best think about open cloud API is that it is easy accessible and can be easily scripted around. For exmaple during one of my tests I created multiple cloud servers but my job failed and didn't delete them before the exception was thrownd.

Problem

How to extract cloud server names from the log file and how to delted all of them from the accout.

$ python performance-single-cs.py-t 1 -s 25 -u user -k key  run | tee log.$(date +%s).txt
$ cat log*.txt
[ 1][  ] starting test nr 1, creating 25 cloud server, please wait ...
[ 1][ 1] created image: {'flavor': 1, 'image': 112, 'name': 'test7945'}
[ 1][ 2] created image: {'flavor': 1, 'image': 112, 'name': 'test7948'}
[ 1][ 3] created image: {'flavor': 1, 'image': 112, 'name': 'test7951'}
[ 1][ 4] created image: {'flavor': 1, 'image': 112, 'name': 'test7954'}
[ 1][ 5] created image: {'flavor': 1, 'image': 112, 'name': 'test7958'}
[ 1][ 6] created image: {'flavor': 1, 'image': 112, 'name': 'test7961'}
[ 1][ 7] created image: {'flavor': 1, 'image': 112, 'name': 'test7965'}
[ 1][ 8] created image: {'flavor': 1, 'image': 112, 'name': 'test7969'}
[ 1][ 9] created image: {'flavor': 1, 'image': 112, 'name': 'test7972'}
[ 1][10] created image: {'flavor': 1, 'image': 112, 'name': 'test7976'}
[ 1][11] created image: {'flavor': 1, 'image': 112, 'name': 'test8050'}
[ 1][12] created image: {'flavor': 1, 'image': 112, 'name': 'test8054'}
[ 1][13] created image: {'flavor': 1, 'image': 112, 'name': 'test8059'}
[ 1][14] created image: {'flavor': 1, 'image': 112, 'name': 'test8063'}
[ 1][15] created image: {'flavor': 1, 'image': 112, 'name': 'test8068'}
[ 1][16] created image: {'flavor': 1, 'image': 112, 'name': 'test8072'}
[ 1][17] created image: {'flavor': 1, 'image': 112, 'name': 'test8077'}
[ 1][18] created image: {'flavor': 1, 'image': 112, 'name': 'test8082'}
[ 1][19] created image: {'flavor': 1, 'image': 112, 'name': 'test8086'}
[ 1][20] created image: {'flavor': 1, 'image': 112, 'name': 'test8091'}
[ 1][21] created image: {'flavor': 1, 'image': 112, 'name': 'test8117'}
[ 1][22] created image: {'flavor': 1, 'image': 112, 'name': 'test8192'}
[ 1][23] created image: {'flavor': 1, 'image': 112, 'name': 'test8197'}
[ 1][24] created image: {'flavor': 1, 'image': 112, 'name': 'test8202'}
[ 1][25] created image: {'flavor': 1, 'image': 112, 'name': 'test8208'}
[ 1][ 1] cloud server build [test7945] created in 298.427304 seconds / 4.9737884 minutes
[ 1][ 2] cloud server build [test7948] created in 298.331735 seconds / 4.97219558333 minutes
[ 1][ 3] cloud server build [test7951] created in 298.268271 seconds / 4.97113785 minutes
[ 1][ 4] cloud server build [test7954] created in 298.469954 seconds / 4.97449923333 minutes
[ 1][ 5] cloud server build [test7958] created in 298.202301 seconds / 4.97003835 minutes
[ 1][ 6] cloud server build [test7961] created in 297.702382 seconds / 4.96170636667 minutes
[ 1][ 7] cloud server build [test7965] created in 298.051012 seconds / 4.96751686667 minutes
[ 1][ 8] cloud server build [test7969] created in 297.3658 seconds / 4.95609666667 minutes
[ 1][ 9] cloud server build [test7972] created in 296.993362 seconds / 4.94988936667 minutes
[ 1][10] cloud server build [test7976] created in 296.810522 seconds / 4.94684203333 minutes
[ 1][11] cloud server build [test8050] created in 226.269396 seconds / 3.7711566 minutes
[ 1][12] cloud server build [test8054] created in 226.051247 seconds / 3.76752078333 minutes
[ 1][14] cloud server build [test8063] created in 225.04139 seconds / 3.75068983333 minutes
[ 1][15] cloud server build [test8068] created in 224.326799 seconds / 3.73877998333 minutes
[ 1][16] cloud server build [test8072] created in 223.051956 seconds / 3.7175326 minutes
[ 1][17] cloud server build [test8077] created in 221.830032 seconds / 3.6971672 minutes
[ 1][18] cloud server build [test8082] created in 219.514883 seconds / 3.65858138333 minutes
[ 1][19] cloud server build [test8086] created in 218.35139 seconds / 3.63918983333 minutes
[ 1][20] cloud server build [test8091] created in 216.172884 seconds / 3.6028814 minutes
[ 1][21] cloud server build [test8117] created in 193.915105 seconds / 3.23191841667 minutes
[ 1][13] cloud server build [test8059] created in 328.421036 seconds / 5.47368393333 minutes
[ 1][22] cloud server build [test8192] created in 197.893329 seconds / 3.29822215 minutes
[ 1][23] cloud server build [test8197] created in 196.71745 seconds / 3.27862416667 minutes
[ 1][24] cloud server build [test8202] created in 263.305984 seconds / 4.38843306667 minutes
[ 1][25] cloud server build [test8208] created in 260.425359 seconds / 4.34042265 minutes

Solution

For a single log file

$ cat log.*.txt | grep "image':" | cut -d':' -f5 | tr '}' ' ' | grep -v created > tmp
echo > 'set -x' >  delete-all-cs.sh
cat tmp | xargs -I cs_name echo 'cloudservers --username user --apikey key delete cs_name' >> delete-all-cs.sh
bash delete-all-cs.sh

When we have multiple log files

$ cat << END > aux_script.sh
cat $1 | grep "image':" | cut -d':' -f5 | tr '}' ' ' | grep -v created > tmp
cat tmp | xargs -I cs_name echo 'cloudservers --username user --apikey key delete cs_name' >> delete-all-cs.sh
END

$ for i in log.*.txt ; do echo $i; ./aux_script.sh $i;  done
$ bash -x delete-all-cs.sh

Summary and results discussion

The solution with 'xargs' works pretty well for relatively small number of servers to delete. As each cloud server is deleted in a single cloudserver run there is no parallelism involved.

An interesting solution could be built with a help of a parallel tool [3]. It could allow us to execute multiple commands in parallel and achieve a much better timing results. Of course to make it work we would have to take into consideration the API limitations and design some workarounds it.

References

1. https://github.com/rtomaszewski/cloud-performance
2. http://www.cyberciti.biz/faq/linux-unix-bsd-xargs-construct- argument-lists-utility/
3. https://savannah.gnu.org/projects/parallel/

How to avoid Rackspace cloud API limits when creating cloud servers (cloudservers.exceptions.OverLimit)

The Rackspace Cloud API has number of limitations. One of them is how many cloud server you can create in 1 minute. The current limitations [1] looks like:

Verb	URI		RegEx		Default
POST	*		.*		10/min
POST	*/servers		^/servers		50/

When your program [2] tries to create more than 10 cloud servers a minute you are going to run into error and start seeing this cloudservers.exceptions.OverLimit exception.

$ python performance-single-cs.py -v -t 1 -s 13 -u user -k key  run | tee log.$(date +%s).txt

Traceback (most recent call last):
  File "performance-single-cs.py", line 349, in module
    Main().run()
  File "performance-single-cs.py", line 346, in run
    self.test_performance(user,key, sample, cs_count, timeout)
  File "performance-single-cs.py", line 298, in test_performance
    t.test_multi_cs_perf(sample, cs_count, timeout)
  File "performance-single-cs.py", line 250, in test_multi_cs_perf
    cs_records=self.cs_create_all(cs_count, i+1)
  File "performance-single-cs.py", line 225, in cs_create_all
    server=self.cs_create(k, sample_nr)
  File "performance-single-cs.py", line 178, in cs_create
    server=sm.create(name, image, flavor)
  File "/usr/lib/pymodules/python2.7/cloudservers/servers.py", line 172, in create
    return self._create("/servers", body, "server")
  File "/usr/lib/pymodules/python2.7/cloudservers/base.py", line 26, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/pymodules/python2.7/cloudservers/client.py", line 69, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/pymodules/python2.7/cloudservers/client.py", line 50, in _cs_request
    resp, body = self.request(self.management_url + url, method, **kwargs)
  File "/usr/lib/pymodules/python2.7/cloudservers/client.py", line 37, in request
    raise exceptions.from_response(resp, body)
cloudservers.exceptions.OverLimit: Too many requests... (HTTP 413)

A naive code that was used in the first version of my program was simply trying to sent as many create requests as possible. This of course is not going to work if we increase servers above the allowed limit.


Code demonstration

An exaple of the naive code.

A code that is more inteligent looks like this.



References

1. http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/Rate_Limits-d1e1017.html
2. https://github.com/rtomaszewski/cloud-performance

My python script buffers the output and it causes delays before the text appiers on the console

Linux bash is exelent tool for every day use. It allows you to combine tools and chain them toggethr to achieve remarkable results. As one of my favorite I use this one when testing:

$ python some_script.py | tee log.$(date +%s).txt 

Problem

The problem is that althoug I get all the output on the console it appers to be bufffered and I can't monitor the logs in live when my script runs. An example code can be seen below.
 
Solution

You have to tell python to stop buffering the data sent the the stream you are using (stdin, stdout, stderr). On on the way I found convenient is by using the command line '-u' options.

References

• http://stackoverflow.com/questions/107705/python-output-buffering
• http://stackoverflow.com/questions/3515757/python-print-statements-being-buffered-with-output-redirection
• http://www.pixelbeat.org/programming/stdio_buffering/

How long does a cloud API may return a cached results

I have been playing with the Rackspace first generation cloud API library/tool (python-cloudservers [1] ) and simulated API calls to create and destroy and list cloud servers.

Problem

At the time of this writing I haven't run into any issues with the create cloud API call. Although I have found some strange behavior when I tried to check status of the newly created cloud server after.

Playing with the tool I wrote [2] for testing I saw that the cloud API call to get a list of all cloud servers immensely relays on cached data that we often get.

GET /servers/detail  # List all servers (all details) [3]

Test

When testing and trying to determine how long it takes to create a cloud sever I got thes results:

# during a week 
$ python performance-single-cs.py -s 2 -u  user -k key  run 
[ 0] creating image: {'flavor': 1, 'image': 112, 'name': 'csperform1343469777'}
[ 0] cloud server build [csperform1343469777] created in 226.428314 seconds / 3.77380523333 minutes
[ 1] creating image: {'flavor': 1, 'image': 112, 'name': 'csperform1343470005'}
[ 1] cloud server build [csperform1343470005] created in 288.115938 seconds / 4.8019323 minutes
[ 2] creating image: {'flavor': 1, 'image': 112, 'name': 'csperform1343471011'}
[ 2] cloud server build [csperform1343471011] created in 1841.598305 seconds / 30.6933050833 minutes

# on Sunday 
$ python performance-single-cs.py -t 1 -s 13 -u hugoalmeidauk -k 391c77192480cbb9969d0514b3daebe1  run
[ 1][  ] starting test nr 1, creating 13 cloud server, please wait ...
[ 1][ 1] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582734'}
[ 1][ 2] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582738'}
[ 1][ 3] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582741'}
[ 1][ 4] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582744'}
[ 1][ 5] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582747'}
[ 1][ 6] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582751'}
[ 1][ 7] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582754'}
[ 1][ 8] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582758'}
[ 1][ 9] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582761'}
[ 1][10] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582835'}
[ 1][11] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582839'}
[ 1][12] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582843'}
[ 1][13] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582847'}
[ 1][ 1] cloud server build [csperform1343582734] created in 149.491952 seconds / 2.49153253333 minutes
[ 1][ 2] cloud server build [csperform1343582738] created in 148.614413 seconds / 2.47690688333 minutes
[ 1][ 3] cloud server build [csperform1343582741] created in 147.810013 seconds / 2.46350021667 minutes
[ 1][ 4] cloud server build [csperform1343582744] created in 147.27588 seconds / 2.454598 minutes
[ 1][ 5] cloud server build [csperform1343582747] created in 145.799047 seconds / 2.42998411667 minutes
[ 1][ 6] cloud server build [csperform1343582751] created in 144.760795 seconds / 2.41267991667 minutes
[ 1][ 7] cloud server build [csperform1343582754] created in 143.3738 seconds / 2.38956333333 minutes
[ 1][ 8] cloud server build [csperform1343582758] created in 141.850991 seconds / 2.36418318333 minutes
[ 1][ 9] cloud server build [csperform1343582761] created in 140.177985 seconds / 2.33629975 minutes
[ 1][10] cloud server build [csperform1343582835] created in 157.222253 seconds / 2.62037088333 minutes
[ 1][11] cloud server build [csperform1343582839] created in 155.36349 seconds / 2.5893915 minutes
[ 1][12] cloud server build [csperform1343582843] created in 153.28229 seconds / 2.55470483333 minutes
[ 1][13] cloud server build [csperform1343582847] created in 151.02232 seconds / 2.51703866667 minutes

Results

In average the time between the library API call to create a cloud server and then multiple API calls to verify that this server is successfully built took about 3 to 4 minutes during the week and 2 to 3 min at the weekend. In the worst case when the cache was involved the cloud server API took up to 30min before the data were invalidated and refreshed.

Analysis  

I don't think it is cache problem only. When looking at the python-cloudservers [1] library we can notice that:

• The documentation has a warning [4]

find(**kwargs) Find a single item with attributes matching **kwargs.
This isn’t very efficient: it loads the entire list then filters on the Python side.

• The python-cloudservers code shows that it uses only the 'GET /servers/detail' API call instead of  'GET /servers/id' [5]

A code analyses will not be provided but for interested readers they can take a lock at these files to confirm:
/usr/lib/pymodules/python2.7/cloudservers/__init__.py
/usr/lib/pymodules/python2.7/cloudservers/base.py
/usr/lib/pymodules/python2.7/cloudservers/servers.py


References
[1] https://github.com/rackspace/python-cloudservers
[2] https://github.com/rtomaszewski/cloud-performance
[3] http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/List_Servers-d1e1730.html
[4] http://packages.python.org/python-cloudservers/ref/servers.html#classes
[5] http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/Get_Server_Details-d1e2143.html

Stale data and caching problem when dealing with cloud API responses

Because the Rackspace cloud API is using caching extensively you have to write a code that is able to handle a stale HTTP data and be able to resent the request if needed.

Example 

A simple example that sometimes works and another time may not work because the API HTTP response contains 404 status code.

from cloudservers import CloudServers
cs=CloudServers(user, key)
cs.authenticate()
sm=cs.servers
server=sm.create(name, image, flavor)
sm.find( name=_server.name )

The cloud 'find' function [1] raises exception NotFound even though we know that the server is there and it shouldn't. 

Traceback (most recent call last):
  File "performance-single-cs.py", line 175, in module
    Main().run()
  File "performance-single-cs.py", line 172, in run
    self.test_performance(user,key)
  File "performance-single-cs.py", line 134, in test_performance
    t.test_perf_single_cs(1)
  File "performance-single-cs.py", line 105, in test_perf_single_cs
    status=self.check_cs_status(server)
  File "performance-single-cs.py", line 45, in check_cs_status
    server=sm.find( name=_server.name )
  File "/usr/lib/pymodules/python2.7/cloudservers/base.py", line 50, in find
    raise NotFound(404, "No %s matching %s." % (self.resource_class.__name__, kwargs))

References

[1] http://packages.python.org/python-cloudservers/ref/servers.html#classes

Rackspace python-cloudservers debugging

For the first generation of Rackspace cloud there is a cli tool and a python library [1].
The source code can the tool can be found on github [2].

Problem description

After successful installation of this tool I run into the follow error below.

 
# cat /etc/issue
Ubuntu 11.04 \n \l
# aptitude install python-rackspace-cloudservers

$ cloudservers --username user --apikey 391c7...be1  show 10197279
Traceback (most recent call last):
  File "/usr/bin/cloudservers", line 9, in 
    load_entry_point('python-cloudservers==1.0a5', 'console_scripts', 'cloudservers')()
  File "/usr/lib/pymodules/python2.7/cloudservers/shell.py", line 409, in main
    CloudserversShell().main(sys.argv[1:])
  File "/usr/lib/pymodules/python2.7/cloudservers/shell.py", line 123, in main
    self.cs.authenticate()
  File "/usr/lib/pymodules/python2.7/cloudservers/__init__.py", line 60, in authenticate
    self.client.authenticate()
  File "/usr/lib/pymodules/python2.7/cloudservers/client.py", line 79, in authenticate
    resp, body = self.request(self.AUTH_URL, 'GET', headers=headers)
  File "/usr/lib/pymodules/python2.7/cloudservers/client.py", line 34, in request
    body = json.loads(body) if body else None
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 360, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 378, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Troubleshooting


Investigating the stacktrace we see that there are various method called before the error was reported. Looking at the stack from bottom up we see that the the code gets interested around lines 12 and 13.

$ sed -n 79p  /usr/lib/pymodules/python2.7/cloudservers/client.py
        resp, body = self.request(self.AUTH_URL, 'GET', headers=headers)

$ grep AUTH_URL /usr/lib/pymodules/python2.7/cloudservers/client.py 
    AUTH_URL = 'https://auth.api.rackspacecloud.com/v1.0'
        resp, body = self.request(self.AUTH_URL, 'GET', headers=headers)

We see that the URL is hard coded in python code. Because my cloud account is base in London (my cloud files and cloud server provisioning regions are in London ) this URL is wrong. More info about the authentication point can be found in the official doc [3].

An easy fix for this requires to change the URL to a new one:

https://lon.auth.api.rackspacecloud.com/v1.0

References

[1] http://packages.python.org/python-cloudservers/index.html
[2] https://github.com/rackspace/python-cloudservers
[3]
http://docs.rackspace.com/api/
http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/Overview-d1e70.html

[4] http://www.dimitrioskouzisloukas.com/blog/index.php?blog=2&title=authenticate_with_cloudservers_to_the_lo&more=1&c=1&tb=1&pb=1

How to create a Cloud Server using Rackspace API

Problem description

How to create a cloud server using cloud API.

Solution

A small script available at gitub [1] that uses available cloud API to login to the Rackspace cloud infrastructure. After login it issues necessary create request and creates a new cloud server. At the end prints the result on stdout.

Demonstration

• Standard output

# python rackspace_cloudserver.py -u user -k key run
creating new cloud server with a name: cstest1343077883
{
 "server": {
  "status": "BUILD", 
  "hostId": "50a21504e046275fa53877dc937ceab6", 
  "name": "cstest1343077883", 
  "adminPass": "KnB1Cu7t4cstest1343077883", 
  "metadata": {}, 
  "imageId": 112, 
  "progress": 0, 
  "flavorId": 1, 
  "id": 10196007, 
  "addresses": {
   "public": [
    "5.79.1.17"
   ], 
   "private": [
    "10.178.4.241"
   ]
  }
 }
}

• Verbose output 

# python rackspace_cloudserver.py -u user -k key -v run
debug[ 1]: user:  key: key;
debug[ 1]: http request
  lon.auth.api.rackspacecloud.com GET /v1.0
  headers: 
    X-Auth-Key: 391....aebe1
    X-Auth-User: user
debug[ 1]: http response
  204 No Content
  headers
    content-length: 0
    x-server-management-url: https://lon.servers.api.rackspacecloud.com/v1.0/10001641
    x-storage-token: 7e901...67a
    vary: X-Auth-Token,X-Auth-Key,X-Storage-User,X-Storage-Pass
    x-cdn-management-url: https://cdn3.clouddrive.com/v1/MossoCloudFS_c99f3ebf-f27d-4933-94b7-9e9ebf2bc7cd
    server: Apache/2.2.3 (Red Hat)
    connection: Keep-Alive
    cache-control: s-maxage=85050
    date: Mon, 23 Jul 2012 20:59:40 GMT
    x-storage-url: https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_c99f3ebf-f27d-4933-94b7-9e9ebf2bc7cd
    x-auth-token: 7e90....1bc67a
    content-type: text/xml
debug[ 1]: read 0 from the server:
creating new cloud server with a name: cstest1343077180
debug[ 1]: http request
  lon.servers.api.rackspacecloud.com POST /v1.0/10001641/servers
  headers: 
    Content-Type: application/json
    X-Auth-Token: 7e901...bc67a
body:
   {"server": {"flavorId": 1, "name": "cstest1343077180", "imageId": 112}}
debug[ 1]: http response
  202 Accepted
  headers
    content-length: 272
    x-varnish: 1662239131
    age: 0
    vary: Accept, Accept-Encoding, X-Auth-Token
    server: Apache-Coyote/1.1
    connection: keep-alive
    via: 1.1 varnish
    cache-control: no-cache
    date: Mon, 23 Jul 2012 20:59:45 GMT
    content-type: application/json
debug[ 1]: read 272 from the server:
{
 "server": {
  "status": "BUILD", 
  "hostId": "155a42778c64cf881c41ae5dc8a8bd0c", 
  "name": "cstest1343077180", 
  "adminPass": "4KUt5u3oXcstest1343077180", 
  "metadata": {}, 
  "imageId": 112, 
  "progress": 0, 
  "flavorId": 1, 
  "id": 10195995, 
  "addresses": {
   "public": [
    "5.79.1.129"
   ], 
   "private": [
    "10.178.18.97"
   ]
  }
 }
}

References
[1]  https://github.com/rtomaszewski/experiments/blob/master/rackspace_cloudserver.py  

How to use the first generation of Rackspace Cloud Server API

Problem description

Using the Rackspace Cloud API [1] for the Cloud Servers how to authenticate yourself.
Being authenticated how to execute and test the various API calls that are listed in the API specification [2]

Solution

• Authentication

To authenticate you need the user name and the API key. You can find these on your cloud control (CC) portal [3] under the "Your Accout" -> "API Access" section.

If you use the new CC Panel you will find it under the user name on the top - right section of the screen. Once clicked you will see the "API Keys" menu that lists the key and the user name again.

Example output:

curl --verbose -H "X-Auth-User: user" -H "X-Auth-Key: key"  https://lon.auth.api.rackspacecloud.com/v1.0
* About to connect() to lon.auth.api.rackspacecloud.com port 443 (#0)
*   Trying 94.236.107.224... connected
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*        subject: C=US; ST=Texas; L=San Antonio; O=Rackspace Managed Hosting; OU=The Rackspace Cloud; OU=Terms of use at www.verisign.com/rpa (c)05; CN=lon.auth.api.rackspacecloud.com
*        start date: 2010-11-02 00:00:00 GMT
*        expire date: 2015-11-01 23:59:59 GMT
*        common name: lon.auth.api.rackspacecloud.com (matched)
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*        SSL certificate verify ok.
> GET /v1.0 HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-cygwin) libcurl/7.22.0 OpenSSL/0.9.8r zlib/1.2.5 libidn/1.22 libssh2/1.2.7
> Host: lon.auth.api.rackspacecloud.com
> Accept: */*
> X-Auth-User: user
> X-Auth-Key: key
>
< HTTP/1.1 204 No Content
< Server: Apache/2.2.3 (Red Hat)
< vary: X-Auth-Token,X-Auth-Key,X-Storage-User,X-Storage-Pass
< X-Storage-Url: https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_c99f3ebf-f27d-4933-94b7-9e9ebf2bc7cd
< Cache-Control: s-maxage=79412
< Content-Type: text/xml
< Date: Sun, 22 Jul 2012 20:33:45 GMT
< X-Auth-Token: a27...e
< X-Server-Management-Url: https://lon.servers.api.rackspacecloud.com/v1.0/10001641
< X-Storage-Token: a27...9de
< Connection: Keep-Alive
< X-CDN-Management-Url: https://cdn3.clouddrive.com/v1/MossoCloudFS_c99f3ebf-f27d-4933-94b7-9e9ebf2bc7cd
< Content-Length: 0
<
* Connection #0 to host lon.auth.api.rackspacecloud.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

The most important values we have to copy from the output for further work are:

X-Auth-Token: a27....b9de
X-Server-Management-Url: https://lon.servers.api.rackspacecloud.com/v1.0/10001641

• Issuing more requests, example of how to list available cloud flavors

curl --verbose -H "X-Auth-Token: a27....9de"  https://lon.servers.api.rackspacecloud.com/v1.0/10001641/flavors
* About to connect() to lon.servers.api.rackspacecloud.com port 443 (#0)
*   Trying 212.64.148.15... connected
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
*        subject: C=US; ST=Texas; L=San Antonio; O=Rackspace Managed Hosting; OU=The Rackspace Cloud; OU=Terms of use at www.verisign.com/rpa (c)05; CN=lon.servers.api.rackspacecloud.com
*        start date: 2010-11-02 00:00:00 GMT
*        expire date: 2015-11-01 23:59:59 GMT
*        common name: lon.servers.api.rackspacecloud.com (matched)
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*        SSL certificate verify ok.
> GET /v1.0/10001641/flavors HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-cygwin) libcurl/7.22.0 OpenSSL/0.9.8r zlib/1.2.5 libidn/1.22 libssh2/1.2.7
> Host: lon.servers.api.rackspacecloud.com
> Accept: */*
> X-Auth-Token: a277...9de
>
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< vary:  Accept, Accept-Encoding, X-Auth-Token
< Last-Modified: Tue, 21 Jun 2011 21:09:45 GMT
< X-PURGE-KEY: /flavors
< Cache-Control: s-maxage=1800
< Content-Type: application/json
< Content-Length: 249
< Date: Sun, 22 Jul 2012 20:34:35 GMT
< X-Varnish: 1603286324
< Age: 0
< Via: 1.1 varnish
< Connection: keep-alive
<
* Connection #0 to host lon.servers.api.rackspacecloud.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
{
  flavors => [
    { id => 1, name => "256 server" },
    { id => 2, name => "512 server" },
    { id => 3, name => "1GB server" },
    { id => 4, name => "2GB server" },
    { id => 5, name => "4GB server" },
    { id => 6, name => "8GB server" },
    { id => 7, name => "15.5GB server" },
    { id => 8, name => "30GB server" },
  ],
}

• Example of how to list available image types (type of operating systems/Linux distributions)

curl -H "X-Auth-Token: a2772b...b9de" https://lon.servers.api.rackspacecloud.com/v1.0/10001641/images | json_xs -t dump 
{
  images => [
    { id => 24,  name => "Windows Server 2008 SP2 x64" },
    { id => 31,  name => "Windows Server 2008 SP2 x86" },
    { id => 56,  name => "Windows Server 2008 SP2 x86 + SQL Server 2008 R2 Standard" },
    { id => 57,  name => "Windows Server 2008 SP2 x64 + SQL Server 2008 R2 Standard"},
    { id => 85,  name => "Windows Server 2008 R2 x64" },
    { id => 86,  name => "Windows Server 2008 R2 x64 + SQL Server 2008 R2 Standard" },
    { id => 89,  name => "Windows Server 2008 R2 x64 + SQL Server 2008 R2 Web"},
    { id => 91,  name => "Windows Server 2008 R2 x64 + SQL Server 2012 Standard" },
    { id => 92,  name => "Windows Server 2008 R2 x64 + SQL Server 2012 Web" },
    { id => 100, name => "Arch 2011.10" },
    { id => 103, name => "Debian 5 (Lenny)" },
    { id => 104, name => "Debian 6 (Squeeze)" },
    { id => 108, name => "Gentoo 11.0" },
    { id => 109, name => "openSUSE 12" },
    { id => 110, name => "Red Hat Enterprise Linux 5.5" },
    { id => 111, name => "Red Hat Enterprise Linux 6" },
    { id => 112, name => "Ubuntu 10.04 LTS" },
    { id => 114, name => "CentOS 5.6" },
    { id => 115, name => "Ubuntu 11.04" },
    { id => 116, name => "Fedora 15" },
    { id => 118, name => "CentOS 6.0" },
    { id => 119, name => "Ubuntu 11.10" },
    { id => 120, name => "Fedora 16" },
    { id => 121, name => "CentOS 5.8" },
    { id => 122, name => "CentOS 6.2" },
    { id => 125, name => "Ubuntu 12.04 LTS" },
    { id => 126, name => "Fedora 17" },
    { id => 10776442, name => "test3" },
    { id => 10992583, name => "test2" },
    { id => 11019155, name => "Mytest1" },
    { id => 11208053, name => "windows2008R2x64Image" },
    { id => 11236707, name => "w2k8x86sp2-090712-IIS" },
  ],
}

The above outputs has been changed a little bit to include the HTTP headers as well as nice formatted server payload. The payload was generated using this command:

curl -H "X-Auth-Token: a27...de" https://lon.servers.api.rackspacecloud.com/v1.0/10001641/flavors | json_xs -t dump

• Issuing a request to create a cloud server

And our important request to create a cloud server with a help of the API request.

cat create-cloud.txt 
{
    "server" : {
        "name" : "rc-test1",
        "imageId" : 112,
        "flavorId" : 1,
        "metadata" : {
            "key1" : "value1"
        }
    }
}

curl --verbose -d @create-cloud.txt -H "Content-Type: application/json" -H "X-Auth-Token: a2772b...9de" https://lon.servers.api.rackspacecloud.com/v1.0/10001641/servers | json_xs -t dump 
* About to connect() to lon.servers.api.rackspacecloud.com port 443 (#0)
*   Trying 212.64.148.15...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to lon.servers.api.rackspacecloud.com (212.64.148.15) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*   subject: C=US; ST=Texas; L=San Antonio; O=Rackspace Managed Hosting; OU=The Rackspace Cloud; OU=Terms of use at www.verisign.com/rpa (c)05; CN=lon.servers.api.rackspacecloud.com
*   start date: 2010-11-02 00:00:00 GMT
*   expire date: 2015-11-01 23:59:59 GMT
*   common name: lon.servers.api.rackspacecloud.com (matched)
*   issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*   SSL certificate verify ok.
> POST /v1.0/10001641/servers HTTP/1.1
> User-Agent: curl/7.21.3 (i686-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: lon.servers.api.rackspacecloud.com
> Accept: */*
> Content-Type: application/json
> X-Auth-Token: a2772b...
> Content-Length: 158
> 
} [data not shown]
100   158    0     0  100   158      0     49  0:00:03  0:00:03 --:--:--    50< HTTP/1.1 202 Accepted
< Server: Apache-Coyote/1.1
< vary:  Accept, Accept-Encoding, X-Auth-Token
< Cache-Control: no-cache
< Content-Type: application/json
< Content-Length: 272
< Date: Sun, 22 Jul 2012 21:33:26 GMT
< X-Varnish: 1603293323
< Age: 0
< Via: 1.1 varnish
< Connection: keep-alive
< 
 36   430    0     0  100   158      0     43  0:00:03  0:00:03 --:--:--    44{ [data not shown]
100   430  100   272  100   158     74     42  0:00:03  0:00:03 --:--:--    75* Connection #0 to host lon.servers.api.rackspacecloud.com left intact

* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
{
  server => {
    addresses => { private => ["10.178.14.178"], public => ["5.79.0.178"] },
    adminPass => "vS2Ec60xDrc-test1",
    flavorId => 1,
    hostId => "0fa7e40134ef21228895618b21e7090b",
    id => 10195258,
    imageId => 112,
    metadata => { key1 => "value1" },
    name => "rc-test1",
    progress => 0,
    status => "BUILD",
  },
}

References
http://docs.rackspace.com/api/
[2] http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/Overview-d1e70.html
[3] https://lon.manage.rackspacecloud.com
[4] http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/Auth-Request.html

Python IDE using Eclipse and PyDev/Aptana

There are numbers of options when you want to code in Python or other languages and you expect some level of support from your editor and IDE. The environment I use is definitely not the simplest and easy to set up but if you like using one single tool to code in various languages it works than relatively well.

Problem description

How to use a modern IDE to develop programs in Python and potentially use it with other languages like Java and C.

Solution


Eclipse as a single IDE with a support for multiple languages.
PyDev as an Eclipse plugin to support programming in Python. But instead of installing it manualy I decide to use the Aptana Studio 3.x

Installation

1. We need java first http://java.com/en/download/faq/java_6.xml
2. Nest install the Eclipse http://www.eclipse.org/downloads/. I used the most popular branch Eclipse IDE for Java EE Developers. It comes with a number of preinstalled modules.
3. Install Python http://www.python.org/getit/. I needed 2.7.x.
4. Every one uses some source of software version control tools. I've started to use   http://msysgit.github.com/
5. This is optional, but because I wanted to have a working Ruby environment as well i installed
1. Ruby from  http://rubyinstaller.org/downloads/
2. From the some link than ' RubyInstaller Development Kit (DevKit)'  DevKit
6. The ruby comes with a powerful Rails application framework  http://rubyonrails.org/download/
7. At the end we installed our Python plugin from  http://aptana.com/products/studio3/download using the option 'Eclipse Plug-in Version'. The installation is described here:  http://aptana.com/downloads/start

After all the steps the hello world looks like this :)