How long does a cloud API may return a cached results

I have been playing with the Rackspace first generation cloud API library/tool (python-cloudservers [1] ) and simulated API calls to create and destroy and list cloud servers.

Problem

At the time of this writing I haven't run into any issues with the create cloud API call. Although I have found some strange behavior when I tried to check status of the newly created cloud server after.

Playing with the tool I wrote [2] for testing I saw that the cloud API call to get a list of all cloud servers immensely relays on cached data that we often get.

GET /servers/detail  # List all servers (all details) [3]

Test

When testing and trying to determine how long it takes to create a cloud sever I got thes results:

# during a week 
$ python performance-single-cs.py -s 2 -u  user -k key  run 
[ 0] creating image: {'flavor': 1, 'image': 112, 'name': 'csperform1343469777'}
[ 0] cloud server build [csperform1343469777] created in 226.428314 seconds / 3.77380523333 minutes
[ 1] creating image: {'flavor': 1, 'image': 112, 'name': 'csperform1343470005'}
[ 1] cloud server build [csperform1343470005] created in 288.115938 seconds / 4.8019323 minutes
[ 2] creating image: {'flavor': 1, 'image': 112, 'name': 'csperform1343471011'}
[ 2] cloud server build [csperform1343471011] created in 1841.598305 seconds / 30.6933050833 minutes

# on Sunday 
$ python performance-single-cs.py -t 1 -s 13 -u hugoalmeidauk -k 391c77192480cbb9969d0514b3daebe1  run
[ 1][  ] starting test nr 1, creating 13 cloud server, please wait ...
[ 1][ 1] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582734'}
[ 1][ 2] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582738'}
[ 1][ 3] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582741'}
[ 1][ 4] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582744'}
[ 1][ 5] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582747'}
[ 1][ 6] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582751'}
[ 1][ 7] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582754'}
[ 1][ 8] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582758'}
[ 1][ 9] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582761'}
[ 1][10] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582835'}
[ 1][11] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582839'}
[ 1][12] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582843'}
[ 1][13] created image: {'flavor': 1, 'image': 112, 'name': 'csperform1343582847'}
[ 1][ 1] cloud server build [csperform1343582734] created in 149.491952 seconds / 2.49153253333 minutes
[ 1][ 2] cloud server build [csperform1343582738] created in 148.614413 seconds / 2.47690688333 minutes
[ 1][ 3] cloud server build [csperform1343582741] created in 147.810013 seconds / 2.46350021667 minutes
[ 1][ 4] cloud server build [csperform1343582744] created in 147.27588 seconds / 2.454598 minutes
[ 1][ 5] cloud server build [csperform1343582747] created in 145.799047 seconds / 2.42998411667 minutes
[ 1][ 6] cloud server build [csperform1343582751] created in 144.760795 seconds / 2.41267991667 minutes
[ 1][ 7] cloud server build [csperform1343582754] created in 143.3738 seconds / 2.38956333333 minutes
[ 1][ 8] cloud server build [csperform1343582758] created in 141.850991 seconds / 2.36418318333 minutes
[ 1][ 9] cloud server build [csperform1343582761] created in 140.177985 seconds / 2.33629975 minutes
[ 1][10] cloud server build [csperform1343582835] created in 157.222253 seconds / 2.62037088333 minutes
[ 1][11] cloud server build [csperform1343582839] created in 155.36349 seconds / 2.5893915 minutes
[ 1][12] cloud server build [csperform1343582843] created in 153.28229 seconds / 2.55470483333 minutes
[ 1][13] cloud server build [csperform1343582847] created in 151.02232 seconds / 2.51703866667 minutes

Results

In average the time between the library API call to create a cloud server and then multiple API calls to verify that this server is successfully built took about 3 to 4 minutes during the week and 2 to 3 min at the weekend. In the worst case when the cache was involved the cloud server API took up to 30min before the data were invalidated and refreshed.

Analysis  

I don't think it is cache problem only. When looking at the python-cloudservers [1] library we can notice that:

• The documentation has a warning [4]

find(**kwargs) Find a single item with attributes matching **kwargs.
This isn’t very efficient: it loads the entire list then filters on the Python side.

• The python-cloudservers code shows that it uses only the 'GET /servers/detail' API call instead of  'GET /servers/id' [5]

A code analyses will not be provided but for interested readers they can take a lock at these files to confirm:
/usr/lib/pymodules/python2.7/cloudservers/__init__.py
/usr/lib/pymodules/python2.7/cloudservers/base.py
/usr/lib/pymodules/python2.7/cloudservers/servers.py


References
[1] https://github.com/rackspace/python-cloudservers
[2] https://github.com/rtomaszewski/cloud-performance
[3] http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/List_Servers-d1e1730.html
[4] http://packages.python.org/python-cloudservers/ref/servers.html#classes
[5] http://docs.rackspace.com/servers/api/v1.0/cs-devguide/content/Get_Server_Details-d1e2143.html