The Systems Engineer organized chaos: api-challenge

Showing posts with label api-challenge. Show all posts

Monday, April 29, 2013

Challenge 11 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 11 script

Below is the output and results from script #11 from the api-challenge.

The script can be repeatedly run with the option -d otherwise it stops if any of the cloud objects it wants to create already exist.

Challenge 11: Write an application that will: Create an SSL terminated load balancer
(Create self-signed certificate).
Create a DNS record that should be pointed to the load balancer. Create Three servers
as nodes behind the LB.

Each server should have a CBS volume attached to it. (Size and type are irrelevant.)
All three servers should have a private Cloud Network shared between them.
Login information to all three servers returned in a readable format as the result
of the script, including connection information. Worth 6 points

Depending on the obligatory -n options it will create cloud objects with predictable names.
Below is an example what objects it creates and what the naming conversion is.

# # python challenge11.py

    usage: challenge11.py [-h] [-v] [-d] [ -k key-key  ] [ -c cert ] [ -i image-id ][ -f flavor-id ] -n FQDN-name

    image-id -f flavor-id
        -h - usage help
        -v - verbose / debug output
        -d - delete objects if they existed in cloud before creating new one
        -k - certificate pritate key (see -c below)
        -c - public certificate ( see -k above)
        -n - FQDN name like www.myexample.com
        -i - specify image-id or use the default for Ubuntu 10.04
        -f - specify flavor-id or use the default for the smallest cloud server


# python challenge11.py -d -n www.challenge11.myrado.net

[01:04:17] Checked your cert/key pair www.challenge11.myrado.net.crt/www.challenge11.myrado.net.key, ok
[01:04:18] Deleting existing domain challenge11.myrado.net
[01:04:20] Checked your FQDN www.challenge11.myrado.net, ok
[01:04:20] Checked your DNS domains challenge11.myrado.net, there is none, ok
[01:04:26] Deleted existing cloud server www.challenge11.myrado.net-0
[01:04:29] Deleted existing cloud server www.challenge11.myrado.net-1
[01:04:33] Deleted existing cloud server www.challenge11.myrado.net-2
[01:04:33] Checked your existing cloud server www.challenge11.myrado.net-*, ok
[01:04:34] Checked your block images www.challenge11.myrado.net, there is none, ok
[01:04:36] Checked your lb www.challenge11.myrado.net, there is none, ok
[01:04:37] Checked your image id d4c7b93d-9f18-45dc-aa7c-3e3b126e3792, ok
[01:04:37] Checked your flavor id 2, ok
[01:04:38] Deleted existing network obj www.challenge11.myrado.net
[01:04:38] Checked your existing network objects www.challenge11.myrado.net, ok
[01:04:38] Checked your cert/key pair www.challenge11.myrado.net.crt/www.challenge11.myrado.net.key, ok
[01:04:39] Checked your FQDN www.challenge11.myrado.net, ok
[01:04:39] Checked your DNS domains challenge11.myrado.net, there is none, ok
[01:04:45] Deleted existing cloud server www.challenge11.myrado.net-1
[01:04:50] Deleted existing cloud server www.challenge11.myrado.net-2
[01:04:50] Checked your existing cloud server www.challenge11.myrado.net-*, ok
[01:04:50] Deleted existing block storage image  www.challenge11.myrado.net-0
[01:04:51] Deleted existing block storage image  www.challenge11.myrado.net-1
[01:04:51] Deleted existing block storage image  www.challenge11.myrado.net-2
[01:04:51] Checked your block images www.challenge11.myrado.net, there is none, ok
[01:05:04] Checked your image id d4c7b93d-9f18-45dc-aa7c-3e3b126e3792, ok
[01:05:04] Checked your flavor id 2, ok
[01:05:05] Checked your existing network objects www.challenge11.myrado.net, ok
[01:05:05] Building 3 cloud servers
.
.
.
.
[01:08:20] Building and configuring lb ...
.
.
.
.
.
.
.
[01:09:09] Building and configuring dns domain ...
[01:09:13] ----------------------------------------------------------------------
[01:09:13] vip name www.challenge11.myrado.net and ip 162.13.24.49
[01:09:13] cloud server www.challenge11.myrado.net-0 added to pool as 10.179.65.126
[01:09:13] cloud server www.challenge11.myrado.net-1 added to pool as 10.179.70.78
[01:09:13] cloud server www.challenge11.myrado.net-2 added to pool as 10.179.73.93
[01:09:13] ----------------------------------------------------------------------
Server # 0: ID 3227a512-3a1c-4f68-a82a-89d15ec74341 name www.challenge11.myrado.net-0 pub IP 95.138.173.80 priv IP 192.168.100.2 password SecretP@ss1
Server # 1: ID 662606f2-66a8-4281-8025-5161bf31754c name www.challenge11.myrado.net-1 pub IP 95.138.171.48 priv IP 192.168.100.1 password SecretP@ss1
Server # 2: ID f37d3551-60f5-40f3-92eb-4cbb1f7dbcfe name www.challenge11.myrado.net-2 pub IP 95.138.175.130 priv IP 192.168.100.3 password SecretP@ss1

References

Friday, April 26, 2013

Challenge 10 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 10 script

Below is the output and results from script #10 from the api-challenge.
The script shows more advance example how consume cloud resources like cloud servers, cloud load balances, cloud files and cloud dns. It does it by building a simple solution.

The script can be repeatedly run with the option -d otherwise it stops if any of the cloud objects it wants to create already exist.
Depending on the obligatory -n options it will create cloud objects with predictable names.
Below is an example what objects it creates and what the naming conversion is.

# python challenge10.py

    usage: challenge10.py [-h] [-v] [-d] [ -s ssh-key ] [ -e error-page ] [ -c container-name ] [ -i image-id ] [ -f flavor-id ] -n FQDN-name

    image-id -f flavor-id
        -h - usage help
        -v - verbose / debug output
        -d - delete objects if they existed in cloud before creating new one
        -n - FQDN name like www.myexample.com
        -s - path to your ssh public key (not priv!)
        -e - path to a html file that will be served from the LB when all pool members are down
        -c - name of cloud files container-name to store the backup data
        -i - specify image-id or use the default for Ubuntu 10.04
        -f - specify flavor-id or use the default for the smallest cloud server

# python challenge10.py -d -n www.challenge10.myrado.net
[22:50:46] Checked your ssh public key /root/.ssh/id_rsa.pub, ok
[22:50:46] Checked your error page error.html, ok
[22:50:50] Deleted existing cloud server www.challenge10.myrado.net-0
[22:50:54] Deleted existing cloud server www.challenge10.myrado.net-1
[22:50:54] Checked your existing cloud server www.challenge10.myrado.net-*, ok
[22:50:56] Deleting existing domain challenge10.myrado.net
[22:50:57] Checked your FQDN www.challenge10.myrado.net, ok
[22:50:57] Checked your DNS domains challenge10.myrado.net, there is none, ok
[22:50:57] Deleted container www.challenge10.myrado.net
[22:50:57] Checked your container www.challenge10.myrado.net where we are going to keep backup data, ok
[22:50:59] Checked your lb www.challenge10.myrado.net, there is none, ok
[22:51:01] Checked your image id d4c7b93d-9f18-45dc-aa7c-3e3b126e3792, ok
[22:51:01] Checked your flavor id 2, ok
[22:51:01] Building 2 cloud servers
.
.
.
.
[22:53:37] Building and configuring lb ...
.
.
.
.
[22:54:15] Building and configuring dns domain ...
[22:54:18] Backuping files to cloud files ...
[22:54:19] ----------------------------------------------------------------------
[22:54:19] vip name www.challenge10.myrado.net and ip 162.13.24.117
[22:54:19] cloud server www.challenge10.myrado.net-0 added to pool as 10.179.73.200
[22:54:19] cloud server www.challenge10.myrado.net-1 added to pool as 10.179.73.165
[22:54:19] Error page is stored in container www.challenge10.myrado.net under name error.html
[22:54:19] to check if the config works try to: curl -v http://162.13.24.117

These are some steps to verify the config.

 
# dig +short @69.20.95.4 www.challenge10.myrado.net
162.13.24.117

# nova list | grep challenge10                                                                          23:03:28
| 35e831a5-cfc8-44a7-98ff-1a003cc30901 | www.challenge10.myrado.net-0 | ACTIVE | public=2a00:1a48:7805:0113:8cfc:cf10:ff08:21ff, 95.138.174.137; private=10.179.73.200 |
| 06259b99-33d0-4537-94ab-e1be032337aa | www.challenge10.myrado.net-1 | ACTIVE | public=2a00:1a48:7805:0113:8cfc:cf10:ff08:3c44, 95.138.174.236; private=10.179.73.165 |

# curl -v http://162.13.24.117
* About to connect() to 162.13.24.117 port 80 (#0)
*   Trying 162.13.24.117... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 162.13.24.117
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Thu, 25 Apr 2013 23:26:45 GMT
< Connection: close
< Content-Type: text/html
<                                                                                                                                   23:28:27
* Closing connection #0
<html>
  <head>
    <title>Challenge 10 - Default error page on LB</title>
  </head>
  <body> Sorry but all pool members failing health checks. </body>
</html>

References

https://github.com/rtomaszewski/api-challenge/tree/challenge10

Wednesday, April 24, 2013

Challenge 9 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 9 script

Below is the output and results from script #9 from the api-challenge.
The script checks if the provided parameters are correct and doesn't collide with existing cloud objects.
First it creates cloud server. Once it is built it maps its public IP in DNS A record for new domain it created.

 
# python challenge9.py

    usage: challenge9.py [-h] [-v] -n FQDN-name -i image-id -f flavor-id
        -h - usage help
        -v - verbose / debug output
        -n - FQDN name
        -i - proper image id
        -f - flavor id

# python challenge9.py -n www.challenge9.myrado.net -i 88130782-11ec-4795-b85f-b55a297ba446 -f 2
[19:54:23] Creating new DNS zone challenge9.myrado.net
[19:54:24] Creating cloud server ...(please wait the build can take a while)
.
.
.
.
.
.
.
.
.
[19:59:34] DNS zone challenge9.myrado.net has been udpated and a new A record created
[19:59:34] rec A: www.challenge9.myrado.net -> 95.138.189.73

These are some steps to verify the config.

 
# dig +short @69.20.95.4  www.challenge9.myrado.net
95.138.189.73

# nova list | grep www.challenge9.myrado.net
| e7c3a258-13ff-426b-8c2d-d24649943947 | www.challenge9.myrado.net | ACTIVE | public=2a00:1a48:7805:0111:8cfc:cf10:ff08:3ac6, 95.138.189.73; private=10.178.199.189  |

References

https://github.com/rtomaszewski/api-challenge/tree/challenge9

Challenge 8 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 8 script

Below is the output and results from script #8 from the api-challenge.
The script checks that there isn't cloud files challenge8 container and no DNS domain rado-challenge.org.
As the script runs it create the container and DNS domain.
It uploads index.html file to the container and sets the meta data according to Serve A Static Website Fast, Without Servers.
Once the job is done it prints some info.

 
$ python challenge8.py
[16:26:00] Creating container challenge8 and uploading small index.html file.
[16:26:00] Creating DNS domain rado-challenge.org and setting CNAME records
[16:26:03] Static site details: (make sure you point your DNS to Rackspace Cloud DNS servers):
[16:26:03] ping challenge8.rado-challenge.org
[16:26:03] curl -v http://challenge8.rado-challenge.org/index.html
[16:26:03] curl -v http://8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com

With the default configuration on any of the cloud servers we will not be able to fully test it. Our domain is not registered anywhere and Rackspace cloud servers don't recursively resolve our CNAME record.

These are some testing to see how it works.

 
$ cat /etc/resolv.conf
# Automatically generated, do not edit
nameserver 83.138.151.81
nameserver 83.138.151.80

$ ping challenge8.rado-challenge.org   
ping: unknown host challenge8.rado-challenge.org

# the 65.61.188.4 is Rackspace cloud dns server (Google it)
# dig +short @65.61.188.4 challenge8.rado-challenge.org                   
8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com.

# curl -v http://8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com     
* About to connect() to 8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com port 80 (#0)
*   Trying 2001:668:108:b::216:3f40... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Last-Modified: Wed, 24 Apr 2013 16:25:58 GMT
< ETag: 60098552c51f8570fe0e59982f6a0869
< X-Timestamp: 1366820758.32217
< Content-Type: text/html
< X-Trans-Id: tx703c6bed688e40dcb184b3d5e3d6b084
< Cache-Control: public, max-age=900
< Expires: Wed, 24 Apr 2013 17:32:21 GMT
< Date: Wed, 24 Apr 2013 17:17:21 GMT
< Content-Length: 28
< Connection: keep-alive
<
Hello challenge8, it works!
* Connection #0 to host 8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com left intact
* Closing connection #0

root@server:~# cat /etc/resolv.conf 
# Automatically generated, do not edit
nameserver 69.20.95.4
#nameserver 83.138.151.81
#nameserver 83.138.151.80

# dig challenge8.rado-challenge.org  

; <<>> DiG 9.8.1-P1 <<>> challenge8.rado-challenge.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2047
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;challenge8.rado-challenge.org. IN      A

;; ANSWER SECTION:
challenge8.rado-challenge.org. 300 IN   CNAME   8d6b88807b981953cc63-629fa6387722ad10d7413e8f778f5452.r54.cf3.rackcdn.com.

;; Query time: 108 msec
;; SERVER: 69.20.95.4#53(69.20.95.4)
;; WHEN: Wed Apr 24 17:22:25 2013
;; MSG SIZE  rcvd: 134

References

Challenge 7 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 7 script

Below is the output and results from script #7 from the api-challenge.
The script first searches for an existing load balancer with a name of challenge7-vip. If it finds it stops. You have to delete the lb first.
If there is not lb with this name it start of creating 2 new cloud servers with names challenge7-0 and challenge7-1.
After the cloud servers are built it create a new LB and adds the servers as node.
At the end it prints some info what it did.
Neither the cloud servers or the lb are deleted after the script run.

 
root@manage2:~/api-challenge# python challenge7.py
[23:50:36] found lb challenge7-vip, please remove it before reruning the script

    usage: challenge7.py [-h] [-v]
        -h - usage help
        -v - verbose / debug output

$ python challenge7.py 
[23:39:31] Building 2 cloud servers
.
.
.
.
.
.
[23:43:13] Building and configuring lb ...
.
.
.
.
[23:43:39] lb name : challenge7-vip 
[23:43:39] lb status : ACTIVE 
[23:43:39] lb created : {u'time': u'2013-04-23T23:43:12Z'} 
[23:43:39] lb virtual_ips : [<VirtualIP type=PUBLIC, id=2525, address=5.79.37.137 version=IPV4>] 
[23:43:39] lb port : 80 
[23:43:39] lb protocol : HTTP 
[23:43:39] lb algorithm : RANDOM 
[23:43:39] lb nodeCount : 2 
[23:43:39] Node: {'port': 80, 'condition': 'ENABLED', 'address': u'10.179.6.186'}
[23:43:39] Node: {'port': 80, 'condition': 'ENABLED', 'address': u'10.179.5.147'}

With the default configuration we put in place we can run only a simple test. You can see that the lb VIP is responding to pings but not for HTTP requests. This is expected behavior as our cloud servers are not configured and there is not Apache listening on port 80.

 
# ping 5.79.37.137
PING 5.79.37.137 (5.79.37.137) 56(84) bytes of data.
64 bytes from 5.79.37.137: icmp_req=2 ttl=61 time=0.657 ms

# curl -v 5.79.37.137                                                                                  23:57:09
* About to connect() to 5.79.37.137 port 80 (#0)
*   Trying 5.79.37.137... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 5.79.37.137
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Tue, 23 Apr 2013 23:55:36 GMT
< Connection: close
< Content-Type: text/html
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<title>Service Unavailable</title>
<style type="text/css">
body, p, h1 {
  font-family: Verdana, Arial, Helvetica, sans-serif;
}
h2 {
  font-family: Arial, Helvetica, sans-serif;
  color: #b10b29;
}
</style>
</head>
<body>
<h2>Service Unavailable</h2>
<p>The service is temporarily unavailable. Please try again later.</p>
</body>
</html>
* Closing connection #0

References

Tuesday, April 23, 2013

Challenge 6 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 6 script

Below is the output and results from script #6 from the api-challenge.
It search for the container name you provide as argument.
It enables CDN feature on the container.
It prints example files from the container for testing.

 
$ python challenge6.py
[16:42:12] Can't find container name None under your cloud account account_name

    usage: challenge6.py [-h] [-v] container-name
        -h - usage help
        -v - verbose / debug output

        container-name - name of the container

$ python challenge3.py tmp/challenge3-test.d/ mycont
$ python challenge6.py -v mycont
[17:16:47] --------------------------------------------
[17:16:47] Container mycont info before enabling CDN:
[17:16:48] cdn_enabled False
[17:16:48] cdn_ttl 86400
[17:16:48] cdn_uri None
[17:16:48] cdn_ssl_uri None
[17:16:48] cdn_streaming_uri None
[17:16:48] cdn_ios_uri None
[17:16:51] --------------------------------------------
[17:16:51] Container info after enabling CDN:
[17:16:51] cdn_enabled True
[17:16:51] cdn_ttl 1200
[17:16:51] cdn_uri http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com
[17:16:51] cdn_ssl_uri https://fb1002c7917e5ba2e2ec-2d8e05068e4fb2e0d8a6741787477e59.ssl.cf3.rackcdn.com
[17:16:51] cdn_streaming_uri http://4f23bda39d9e91b1dce8-2d8e05068e4fb2e0d8a6741787477e59.r33.stream.cf3.rackcdn.com
[17:16:51] cdn_ios_uri http://9ccd8d607dacdd984e50-2d8e05068e4fb2e0d8a6741787477e59.iosr.cf3.rackcdn.com
[17:16:51] --------------------------------------------
[17:16:51] Example files from the container mycont
[17:16:51] ============================================
[17:16:51] ch3f1.txt
[17:16:51] http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com/ch3f1.txt
[17:16:51] https://fb1002c7917e5ba2e2ec-2d8e05068e4fb2e0d8a6741787477e59.ssl.cf3.rackcdn.com/ch3f1.txt
[17:16:51] ============================================
[17:16:51] ch3f2.txt
[17:16:51] http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com/ch3f2.txt
[17:16:51] https://fb1002c7917e5ba2e2ec-2d8e05068e4fb2e0d8a6741787477e59.ssl.cf3.rackcdn.com/ch3f2.txt

After we enabled the CDN on the container level we can openly access our files using the above links

 
$ curl -o file.txt -v http://bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com/ch3f1.txt
* About to connect() to bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com port 80 (#0)
*   Trying 2001:668:108:b::216:3f19...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
> GET /ch3f1.txt HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 31
< Accept-Ranges: bytes
< Last-Modified: Tue, 23 Apr 2013 17:13:48 GMT
< ETag: 7f80594ffedbf15e5ade5f540eff6770
< X-Timestamp: 1366737228.82321
< Content-Type: text/plain
< X-Trans-Id: tx5109b62029b44600afbf6274e4bb9699
< Cache-Control: public, max-age=1187
< Expires: Tue, 23 Apr 2013 17:36:49 GMT
< Date: Tue, 23 Apr 2013 17:17:02 GMT
< Connection: keep-alive
<
{ [data not shown]
100    31  100    31    0     0    112      0 --:--:-- --:--:-- --:--:--   306
* Connection #0 to host bd558d3e1d93f7c9aa5e-2d8e05068e4fb2e0d8a6741787477e59.r33.cf3.rackcdn.com left intact
* Closing connection #0

$ cat file.txt

References

How to merge changes from a custom branch into master branch on github

As per our release policy for the api-challenge we have created a new branch called challenge5 to release the code that implements task #5. After the initial release we found some bugs. We fixed them and committed the changes back to our challenge5 repository using this method How to modify files in your branch on github. These are the changes made to the challenge5 branch:

https://github.com/rtomaszewski/api-challenge/commit/7872d14cc22e96b3b9f2049ae0d76a7697af22c6

Problem

The changes made on the challenge5 branch are not visible in master repository. How to copy the changes from challenge5 to master branch.

Solution

To copy the changes from one branch to another you can use github merging feature.

 
PS > git status .\challenge5.py
# On branch master
nothing to commit (working directory clean)

PS > git branch
  challenge2
  challenge3
  challenge4
  challenge5
* master

PS> git merge challenge5
Updating d609784..7872d14
Fast-forward
 challenge5.py |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

PS > git status .\challenge5.py
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
nothing to commit (working directory clean)

PS > git push
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:rtomaszewski/api-challenge
   d609784..7872d14  master -> master

References

http://learn.github.com/p/branching.html

Challenge 5 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 5 script

Below is the output and results from script #5 from the api-challenge.
The script first do some verification if the provided or the default options are correct.
It verifies if a data base instance exists and if run with option '-i' deletes it.
If there is no db instance it will create a new one.
Before proceeding it waits for the instance to be built. It can take some minutes (on average about 2 min).
Once the instance is created it connects to it and creates databases.
Once the data base(s) are created it creates user(s) that can connect to them.
At the end the script is not deleting the created db instance, databases or users. It has to be deleted manually.

 
$ python challenge5.py -h

    usage: challenge5.py [-h] [-v] [ -i | -d ] [ -u db-user ] [ instance-name ] [ db-name1 [db-name2] ]
        -h - usage help
        -v - verbose / debug output
        -i - if instance exists delete it and all its databases before proceeding
        -d - delete database(s) from under the instance

        instance-name - create instance with this name
        db-nameX - name of the database
        db-user - create user that can access the database; by default a user name is equal to db-name

$ python challenge5.py -i
[12:12:30] Found existing db instance challenge5-inst1, deleting it ...
challenge5-user2']
[12:12:35] Creating instance challenge5-inst1
[12:12:36] Waiting for the instance to be built ...
.
.
.
[12:14:38] Instance is created on host ad5e20f3ed07b5ab7592ede01f15316ef05fb0e2.rackspaceclouddb.com
[12:14:39] Creating databases ['challenge5-db1', 'challenge5-db2'] under the instance challenge5-inst1
[12:14:39] creating db: challenge5-db1
[12:14:40] creating db: challenge5-db2
[12:14:40] Creating users ['challenge5-user1', 'challenge5-user2']
[12:14:41] Created user challenge5-user1 pass pass123@ in db ['challenge5-db1', 'challenge5-db2']
[12:14:42] Created user challenge5-user2 pass pass123@ in db ['challenge5-db1', 'challenge5-db2']

Once the database is created we can connect to it from any existing cloud servers.

 
root@server:~/# mysql -h 71b4042b25136f6337ca013bb8525afae87a75a0.rackspaceclouddb.com --user=challenge5-user1 --password=pass123@ challenge5-db1
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 52
Server version: 5.1.66-0+squeeze1 (Debian)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| challenge5-db1     |
| challenge5-db2     |
+--------------------+
3 rows in set (0.00 sec)

References

Monday, April 22, 2013

Challenge 4 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 4 script

Below is the output and results from script #4 from the api-challenge.
The script first checks if a domain exists. By default it will terminated but if you specify the -d options it will delete the existing domain before proceeding.
Next adds a new A record according to the provided arguments.
Once the A record is added it tries to find the domain again and prints all info about it.
After it finishes it doesn't delete the created domain.

 
$ python challenge4.py
[18:58:06] missing params

    usage: challenge4.py [-h] [-v] [-d] FQDN ip-add
        -h - usage help
        -v - verbose / debug output
        -d - if a domain exists it will be deleted first

        FQDN - Fully Qualified Domain Name; exaple www.wikipedia.org
        ip-add - ip address

$ python challenge4.py challenge4.rado.com 100.2.3.4
[18:45:47] Domain created: challenge4.rado.com
[18:45:47] Adding records to our domain ...
[18:45:49] domain name: challenge4.rado.com created at 2013-04-22T18:44:06.000+0000
[18:45:50] record   A challenge4.rado.com -> 100.2.3.4
[18:45:50] record  NS challenge4.rado.com -> dns1.stabletransit.com
[18:45:50] record  NS challenge4.rado.com -> dns2.stabletransit.com

References

Challenge 3 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 3 script

Below is the output and the results for the script #3 from the api-challenge.
The script first checks if the provided directory exists.
Next checks if there is a container with the name you provided.
If there is not it creates a new container.
It uploads all files and directories from the directory dir-name to cloud files and saves them in the container container-name.
After it finishes it neither deletes the local directory or the container it created.

 
$ python challenge3.py
[09:47:54] missing params

    usage: challenge3.py [-h] [-v] dir-name container-name
        -h - usage help
        -v - verbose / debug output
        dir-name - a path to a directory
        container-name - upload the dir-name to this cloud container

$ python challenge3.py tmp/challenge3-test.d/ cfdir
[09:47:01] Uploading directory tmp/challenge3-test.d/ to container cfdir
[09:47:01] container cfdir don't exists, create a new one ...
Total bytes to upload: 181
Progress: 0.00%
Progress: 16.57%
Progress: 73.48%
Progress: 100.00%

References

Saturday, April 20, 2013

How to develop and debug programs that use pyrax or python-novaclient libraries

Python belongs to a category of high-level, dynamically typed programming languages. This has enormous impact on the way how you write the code and how it is interpreted and executed.

As an example let's take a look at Python Rackspace pyrax module. It was built and made available in a form of Python SDK for Rackspace cloud. Under the hood it is a wrapper and integrator of various python libraries that exists to interact with Openstack systems(cloud servers, files, networks, databases ...). As an example it uses python-novaclient library communicate and interact with Nova (Openstack Compute).

From developer point of view you can write a Python program to interact with Rackspace cloud using any of them. Both should work fine.

Working with any of these libraries you are going to face these challenges very soon:

where is the latest API documentation
how well are function documented
for a function what arguments it accepts and what it returns

In comparison to other statically typed languages like Java you quickly realize that Python definition of a function doesn't' fully answer the above questions. Below are couple of tricks to help you start with to play with them although.

Problem 1

How to start interactive Python session with loaded pyrax module for testing

We are going to use bpython that has a nice feature allowing it to show the available functions and variables a Python objects implements (for these who are interested how this works you can read about introspection here and there)

Create a file with credentials

 
root@server:~# cat > rackspace_cloud_credentials
[rackspace_cloud]
username = user
api_key = key

Create an auxiliary file for bpython

 
# https://github.com/rtomaszewski/api-challenge/blob/master/bpython-pyrax.py
cat > bpython-pyrax.py
import os
import sys
import json
import re
import time
import datetime, time

from pprint import pprint
from pprint import pformat

import pyrax

creds_file = os.path.expanduser("~/rackspace_cloud_credentials")
pyrax.set_credential_file(creds_file, "LON")
cs = pyrax.cloudservers

help_str="""
the pyrax module has been loaded, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account 
l=cs.servers.list()
print(l)

# example 2: show list of images 
pprint(cs.images.list())
"""

print(help_str)

Open the bpython interpreter

 
root@server:~# bpython -i  bpython-pyrax.py

the pyrax module has been loaded, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account
l=cs.servers.list()
print(l)

# example 2: show list of images
pprint(cs.images.list())

>>>

Test how it works

Problem 2

How to start interactive Python session with loaded python-novaclient for testing

Create bash file with variables

 
cat > nova.vars.sh
export OS_AUTH_SYSTEM="rackspace_uk"
export OS_AUTH_URL="https://lon.identity.api.rackspacecloud.com/v2.0/"
export OS_NO_CACHE="1"
export OS_PASSWORD="pass"
export OS_REGION_NAME="LON"
export OS_TENANT_NAME="user"
export OS_USERNAME="user"
export NOVA_RAX_AUTH=1
export OS_PROJECT_ID="user"

Create an auxiliary file for bpython

 
# https://github.com/rtomaszewski/api-challenge/blob/master/bpython-novaclient.py
cat > bpython-novaclient.py
import os
import sys
import json
import re
import time
import datetime, time

from pprint import pprint
from pprint import pformat

from novaclient.v1_1 import client

os.environ["OS_USERNAME"]
os.environ["OS_PASSWORD"]
os.environ["OS_NO_CACHE"]
os.environ["OS_TENANT_NAME"]
os.environ["OS_AUTH_URL"]
os.environ["OS_REGION_NAME"]
os.environ["OS_AUTH_SYSTEM"]
os.environ["NOVA_RAX_AUTH"]
os.environ["OS_PROJECT_ID"]

from novaclient import auth_plugin as _cs_auth_plugin
_cs_auth_plugin.discover_auth_systems()
auth_plugin = _cs_auth_plugin.load_plugin("rackspace_uk")

cs = client.Client(os.environ["OS_USERNAME"], os.environ["OS_PASSWORD"], os.environ["OS_TENANT_NAME"], auth_url=os.environ["OS_AUTH_URL"], auth_system="rackspace", region_name="LON",  service_type="compute", auth_plugin=auth_plugin)

help_str="""
the novaclient module has been loaded and INITIATED, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account 
l=cs.servers.list()
print(l)

# example 2: show list of images 
pprint(cs.images.list())
"""

print(help_str)

Open the bpython interpreter

 
root@manage2:~# bpython -i bpython-novaclient.py

the novaclient module has been loaded and INITIATED, try typing one of the commands below to see if it works.

#example 1: whow all cloud servers under you cloud account
l=cs.servers.list()
print(l)

# example 2: show list of images
pprint(cs.images.list())
>>>

Test how it works

References

Wednesday, April 17, 2013

How to modify files in your branch on github

You created and released a new branch following the RERO development philosophy paradigm. Later on you discover that there are some bugs that you would like to fix in your branch and publish.

Problem

How to fix a file in a branch.

Solution

We need to checkout the branch from repository.
Modify the file and fix the issue.
Commit the changes back to our branch.

An example session is showed below.

 
git checkout challenge2
#now you modify the file, once done
git commit -m 'fixed delete_images func' -a
git push origin

# after modification we want to return in default branch
git checkout master
git branch

Challenge 2 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Challenge 2 script

Below is the output and the results for the script #2 from the api-challenge.
The script first create a single new cloud server with a name web0.
Next it waits for the cloud server to be created. This behavior is similar to challenge #1
Once the cloud is built it create a backup under the name web0-img, by cloning the live cloud server.
Once the image is saved it uses it to spin up a new cloud server with a name web0_m.dd.hh.mm.ss.
Once the new cloud image is built it shows the summary of the created objects.
Before the end of the work the created objects (2 cloud server, 1 image) are deleted.

 
$ python.exe challenge2.py
[00:41:59] Building 1 cloud server(s).
[00:41:59] building web0 cloud server
[00:42:00] Waiting for the servers to be built ...
.
.
.
.
.
.
.
.
.
.
[00:47:36] Preparing to clone 1 server(s) ...
[00:47:36] Creating cloud image file: 'web0-img' from web0 cloud server
[00:47:38] Waiting for the clone images to be created ...
.
.
.
[00:49:41] Building 1 new cloud server(s) from taken images.
[00:49:41] building new web0_4.17.1.49.41 cloud server from image web0-img
[00:49:42] Waiting for the servers to be built from the cloned images ...
.
.
.
.
.
.
[00:53:16] Created cloud server details:
Server # 0:  ID  fd9b996c-d034-4aca-9c73-2ef11b31707d name             web0 IP     162.13.2.151 password Gz7CjrjNtNU7
[00:53:16] Created images details:
Image  # 0:  ID  a0c1b05a-3ab0-414e-8d3c-663f58f5ab94 name         web0-img from server fd9b996c-d034-4aca-9c73-2ef11b31707d
[00:53:16] Created cloned server details:
Server # 0:  ID  de18d5f6-bb97-4e5f-a315-53698edffd0e name web0_4.17.1.49.41 IP     162.13.2.231 password FDoC8mMHtppo

References

https://github.com/rtomaszewski/api-challenge/tree/challenge2

Monday, April 8, 2013

How to create a branch on github to freeze your code in a particular version

The nature of developing is that the code is in constant change. These could be new features, bug fixes, code refactoring or some modifications to polishing the code.

Sooner or later you are going to realize that on some occasions you need to have access and a reference to the code you tested or officially published. This is a big topic and I don't want to go into deep details about it. If you are interested to know more try to search on the web for Software Release Management topic.

As a simple example please take a look at my previous blog Challenge 1 script. I tested my code and documented the expected results. Some days after as I worked on the other scripts I made changes to my original code as well. For anyone who would like to take a look what I was describing previous it would be very difficult. Looking the latest code from github is not what you really want.

Problem

How to create a branch on github to release a particular version of your code.

Solution

 
$ git branch
* master
$ git checkout 47677b8c1cf79e6517b2caf7b32aca59b4d38795
HEAD is now at 47677b8... prints ipv4 ip

$ git branch
* (no branch)
  master

$ git branch challange1
$ git branch
* (no branch)
  challange1
  master

$ git checkout challange1
Switched to branch 'challange1'

$ git branch
* challange1
  master

$ git push origin challange1
Warning: Permanently added 'github.com,204.232.175.90' (RSA) to the list of known hosts.
Counting objects: 33, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (23/23), done.
Writing objects: 100% (33/33), 6.10 KiB, done.
Total 33 (delta 13), reused 15 (delta 4)
To git@github.com:rtomaszewski/api-challenge.git
 * [new branch]      challange1 - challange1

After the branch is created you can always download the code for it: https://github.com/rtomaszewski/api-challenge/tree/challange1

References

Sunday, April 7, 2013

Challenge 1 script

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Update:
Added the direct link to the branch with tested code.

Challenge 1 script

Below is an output from the script showing the results.
The script first creates 3 cloud servers with a name webX.
Next waits until they are built.
Once they are build we can find what public IP they got assigned.
And lastly we print this on stdout.

 
python challange1.py
[16:49:53] Building 3 cloud server(s).
[16:49:53] building web0 cloud server
[16:49:53] building web1 cloud server
[16:49:54] building web2 cloud server
[16:49:54] Waiting for the servers to be built ...
.
.
.
.
.
.
.
[16:54:00] Cloud server details:
Server # 0:  ID  e5d8b1bc-b85f-462e-b4bc-00f1432a89d0 IP   95.138.173.125 password JH6kFgUpBpzW
Server # 1:  ID  5d22f6f6-4b43-45ed-a40d-2f7076658168 IP    95.138.173.13 password Q5Ho8YJTqBQv
Server # 2:  ID  6d21f854-ad96-4735-8f94-6c9e40d74a37 IP    95.138.173.56 password mZJ9psBV5Crz
[Finished in 263.0s]

References

The documented results of the script can be found under this branch at github https://github.com/rtomaszewski/api-challenge/tree/challange1
The lates code and more examples are availabe here https://github.com/rtomaszewski/api-challenge

Howto add a file to api-challenge repository

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Adding my first file to the api-challenge repo

Before we start coding our first program let's checkout the api-challenge repository we created.

 
cd C:\Users\rado4752\Desktop\docs\development\_mycoding\api-challenge
git clone git@github.com:rtomaszewski/api-challenge.git

As soon as you create a file you are working on you can add it to track changes.

 
git add challange1.py
git commit -m 'init'
git push origin master

After you have completed some of your work and would like to upload a new version of the file to the master repository.

 
git commit -m 'changed output' .\challange1.py
git pull origin master
git push origin master

References

Wednesday, April 3, 2013

How to install Rackspace Python SDK for cloud

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

The Python SDK is available here. It is very simple to install it in any modern Linux distribution.

 
aptitude install python-pip
pip install pyrax

In Windows we need to do couple things more.

Download and install the distribute Python package

Powershell as Bash substitute on Windows

 
PS C:\Users\radoslaw> curl -O http://python-distribute.org/distribute_setup.py
PS C:\Users\radoslaw> python distribute_setup.py

Download and install the pip Python package

 
PS C:\Users\radoslaw> curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
PS C:\Users\radoslaw> python get-pip.py

Check that you can execute pip from command line

blog

 
PS C:\Users\radoslaw> pip

Usage:
  pip  [options]
...

Lastly install pyrax SDK module and test it

 
PS C:\Users\radoslaw> python
Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrax
>>> pyrax
<module 'pyrax' from 'c:\Python27\lib\site-packages\pyrax\__init__.pyc'>

References

Sunday, March 24, 2013

Rackspace api-challenge summary

Below is a listing of some of my steps I went through when I participated in Rackspace API Challenges: By Support, for Support contest. The final github repository with the code is here https://github.com/rtomaszewski/api-challenge

Create initial repository on github
In the OS of your choice install your favorite editor like Sublime (Installation of Sublime Text editor on Ubuntu 12.10) or other
Install Python SDK for Rackspace cloud: How to install Rackspace Python SDK for cloud
Authentication issue with London cloud account when using Pyrax module
Howto add a file to api-challenge repository on github
Challenge 1 script
As my development efforts go on I wanted make sure I can always get back to the code I was initially released as challenge1.py script: How to create a branch on github to freeze your code in a particular version
Challenge 2 script
How to modify files in your branch on github
bpython supports virtualenv
How to develop and debug programs that use pyrax or python-novaclient libraries
Challenge 3 script
How to paste into Python interpreter a code snippet for testing
Challenge 4 script
Challenge 5 script
How to merge changes from a custom branch into master branch on github
Challenge 6 script
Challenge 7 script
Challenge 8 script
Challenge 9 script
How to use pyrax and ipython together to test and learn the library api
Challenge 10 script
Challenge 11 script

Create initial repository on github

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

This is in regards to the Rackspace API-challenge. More info can be found at Rackspace api-challenge summary.

All my steps to create my repository are listed below.

Install Git Shell for windows

mkdir api-challenge
git init

Create next a README.md file

git commit
git add README.md
git commit -m 'init'
git remote add origin git@github.com:rtomaszewski/api-challenge.git

git push -u origin master
Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of know
n hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Troubleshooting base on https://help.github.com/articles/error-permission-denied-publickey

 
ssh -vT git@github.com
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
....
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /c/Users/rado4752/.ssh/github_rsa
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

ssh-add -l
The agent has no identities.

I had to add and register my public ssh key github_rsa on github. After adding it using the webgui https://github.com/settings/ssh this test went fine.

 
ssh -vT git@github.com
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Applying options for github.com
debug1: Connecting to github.com [207.97.227.239] port 22.
debug1: Connection established.
debug1: identity file /c/Users/rado4752/.ssh/github_rsa type 1
...
Hi rtomaszewski! You've successfully authenticated, but GitHub does not provide
shell access.
debug1: channel 0: free: client-session, nchannels 1
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.5 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 1

git remote add origin git@github.com:rtomaszewski/api-challenge
fatal: remote origin already exists.

git remote -v
origin  git@github.com:rtomaszewski/api-challenge (fetch)
origin  git@github.com:rtomaszewski/api-challenge (push)

git push -u origin master
Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of know hosts.
ERROR: Repository not found.
Fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

git remote rm origin
git remote add origin git@github.com:rtomaszewski/api-challenge.git
git remote -v
origin  git@github.com:rtomaszewski/api-challenge.git (fetch)
origin  git@github.com:rtomaszewski/api-challenge.git (push)

As there is not repository with the name api-challenge I created it under my github account.

 
git push -u origin master

After all this trouble i could navigate to my newly created repository under https://github.com/rtomaszewski/api-challenge

At the end got to http://challenge.cldsrvr.com/register and register your repo.

Search This Blog

Monday, April 29, 2013

Friday, April 26, 2013

Wednesday, April 24, 2013

Tuesday, April 23, 2013

Monday, April 22, 2013

Saturday, April 20, 2013

Wednesday, April 17, 2013

Monday, April 8, 2013

Sunday, April 7, 2013

Wednesday, April 3, 2013

Sunday, March 24, 2013

RSS/Atom feed

Popular posts last week

Last posts

About Me

Linux enthusiast

My links

Friendly sites

Visitor locations

Total Pageviews

Blog Archive

Labels