How to clean and delete multiple cloud servers after a failed test

The best think about open cloud API is that it is easy accessible and can be easily scripted around. For exmaple during one of my tests I created multiple cloud servers but my job failed and didn't delete them before the exception was thrownd.

Problem

How to extract cloud server names from the log file and how to delted all of them from the accout.

$ python performance-single-cs.py-t 1 -s 25 -u user -k key  run | tee log.$(date +%s).txt
$ cat log*.txt
[ 1][  ] starting test nr 1, creating 25 cloud server, please wait ...
[ 1][ 1] created image: {'flavor': 1, 'image': 112, 'name': 'test7945'}
[ 1][ 2] created image: {'flavor': 1, 'image': 112, 'name': 'test7948'}
[ 1][ 3] created image: {'flavor': 1, 'image': 112, 'name': 'test7951'}
[ 1][ 4] created image: {'flavor': 1, 'image': 112, 'name': 'test7954'}
[ 1][ 5] created image: {'flavor': 1, 'image': 112, 'name': 'test7958'}
[ 1][ 6] created image: {'flavor': 1, 'image': 112, 'name': 'test7961'}
[ 1][ 7] created image: {'flavor': 1, 'image': 112, 'name': 'test7965'}
[ 1][ 8] created image: {'flavor': 1, 'image': 112, 'name': 'test7969'}
[ 1][ 9] created image: {'flavor': 1, 'image': 112, 'name': 'test7972'}
[ 1][10] created image: {'flavor': 1, 'image': 112, 'name': 'test7976'}
[ 1][11] created image: {'flavor': 1, 'image': 112, 'name': 'test8050'}
[ 1][12] created image: {'flavor': 1, 'image': 112, 'name': 'test8054'}
[ 1][13] created image: {'flavor': 1, 'image': 112, 'name': 'test8059'}
[ 1][14] created image: {'flavor': 1, 'image': 112, 'name': 'test8063'}
[ 1][15] created image: {'flavor': 1, 'image': 112, 'name': 'test8068'}
[ 1][16] created image: {'flavor': 1, 'image': 112, 'name': 'test8072'}
[ 1][17] created image: {'flavor': 1, 'image': 112, 'name': 'test8077'}
[ 1][18] created image: {'flavor': 1, 'image': 112, 'name': 'test8082'}
[ 1][19] created image: {'flavor': 1, 'image': 112, 'name': 'test8086'}
[ 1][20] created image: {'flavor': 1, 'image': 112, 'name': 'test8091'}
[ 1][21] created image: {'flavor': 1, 'image': 112, 'name': 'test8117'}
[ 1][22] created image: {'flavor': 1, 'image': 112, 'name': 'test8192'}
[ 1][23] created image: {'flavor': 1, 'image': 112, 'name': 'test8197'}
[ 1][24] created image: {'flavor': 1, 'image': 112, 'name': 'test8202'}
[ 1][25] created image: {'flavor': 1, 'image': 112, 'name': 'test8208'}
[ 1][ 1] cloud server build [test7945] created in 298.427304 seconds / 4.9737884 minutes
[ 1][ 2] cloud server build [test7948] created in 298.331735 seconds / 4.97219558333 minutes
[ 1][ 3] cloud server build [test7951] created in 298.268271 seconds / 4.97113785 minutes
[ 1][ 4] cloud server build [test7954] created in 298.469954 seconds / 4.97449923333 minutes
[ 1][ 5] cloud server build [test7958] created in 298.202301 seconds / 4.97003835 minutes
[ 1][ 6] cloud server build [test7961] created in 297.702382 seconds / 4.96170636667 minutes
[ 1][ 7] cloud server build [test7965] created in 298.051012 seconds / 4.96751686667 minutes
[ 1][ 8] cloud server build [test7969] created in 297.3658 seconds / 4.95609666667 minutes
[ 1][ 9] cloud server build [test7972] created in 296.993362 seconds / 4.94988936667 minutes
[ 1][10] cloud server build [test7976] created in 296.810522 seconds / 4.94684203333 minutes
[ 1][11] cloud server build [test8050] created in 226.269396 seconds / 3.7711566 minutes
[ 1][12] cloud server build [test8054] created in 226.051247 seconds / 3.76752078333 minutes
[ 1][14] cloud server build [test8063] created in 225.04139 seconds / 3.75068983333 minutes
[ 1][15] cloud server build [test8068] created in 224.326799 seconds / 3.73877998333 minutes
[ 1][16] cloud server build [test8072] created in 223.051956 seconds / 3.7175326 minutes
[ 1][17] cloud server build [test8077] created in 221.830032 seconds / 3.6971672 minutes
[ 1][18] cloud server build [test8082] created in 219.514883 seconds / 3.65858138333 minutes
[ 1][19] cloud server build [test8086] created in 218.35139 seconds / 3.63918983333 minutes
[ 1][20] cloud server build [test8091] created in 216.172884 seconds / 3.6028814 minutes
[ 1][21] cloud server build [test8117] created in 193.915105 seconds / 3.23191841667 minutes
[ 1][13] cloud server build [test8059] created in 328.421036 seconds / 5.47368393333 minutes
[ 1][22] cloud server build [test8192] created in 197.893329 seconds / 3.29822215 minutes
[ 1][23] cloud server build [test8197] created in 196.71745 seconds / 3.27862416667 minutes
[ 1][24] cloud server build [test8202] created in 263.305984 seconds / 4.38843306667 minutes
[ 1][25] cloud server build [test8208] created in 260.425359 seconds / 4.34042265 minutes

Solution

For a single log file

$ cat log.*.txt | grep "image':" | cut -d':' -f5 | tr '}' ' ' | grep -v created > tmp
echo > 'set -x' >  delete-all-cs.sh
cat tmp | xargs -I cs_name echo 'cloudservers --username user --apikey key delete cs_name' >> delete-all-cs.sh
bash delete-all-cs.sh

When we have multiple log files

$ cat << END > aux_script.sh
cat $1 | grep "image':" | cut -d':' -f5 | tr '}' ' ' | grep -v created > tmp
cat tmp | xargs -I cs_name echo 'cloudservers --username user --apikey key delete cs_name' >> delete-all-cs.sh
END

$ for i in log.*.txt ; do echo $i; ./aux_script.sh $i;  done
$ bash -x delete-all-cs.sh

Summary and results discussion

The solution with 'xargs' works pretty well for relatively small number of servers to delete. As each cloud server is deleted in a single cloudserver run there is no parallelism involved.

An interesting solution could be built with a help of a parallel tool [3]. It could allow us to execute multiple commands in parallel and achieve a much better timing results. Of course to make it work we would have to take into consideration the API limitations and design some workarounds it.

References

1. https://github.com/rtomaszewski/cloud-performance
2. http://www.cyberciti.biz/faq/linux-unix-bsd-xargs-construct- argument-lists-utility/
3. https://savannah.gnu.org/projects/parallel/