How to do URL based load balancing on F5

There are many load balancers out there. Some of them offer a great flexibility to control the traffic by allowing a user to upload a custom script that implement the load balancing algorithm to solve a particular problem.

Problem

How to do HTTP URL based load balancing on F5.

Solution and demonstration

This is an iRule script that inspects the HTTP GET URL string to decided where to load balance it: https://github.com/rtomaszewski/f5/blob/master/lb-based-on-url.tcl.

Create default pool

Create VIP

Create custom pools

Testing

To verify that our iRule is working properly we can enable debugging by changing the iRule variable DEBUG to 1.

Next we can simulate traffic

curl -v http://vip/
curl -v http://vip/url1
curl -v http://vip/url2
curl -v http://vip/url3

And watch the logs on the lb.

tail -f /var/log/ltm

Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80
Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url1
Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url1
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url2
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url2
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url3
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url3

Reference

https://devcentral.f5.com/wiki/iRules.HomePage.ashx

OpenShift PaaS platform from Redhat

We've looked before at AppFog PaaS platform that is leveraging CloudFoundry and Openstack. Today we will investigate the Redhat PaaS alternative, namely OpenShift.

What is OpenShift

OpenShift is a Platform as a Service solution according the the cloud systems taxonomy. It offers a solution, a software development platform that facilitates easy and rapid application development in the cloud.

It can provide a preintegrated software stack environment and can leverage IaaS providers for the virtual infrastructure management (How to Deploy OpenShift Enterprise on Red Hat OpenStack).

A comprehensive presentation that provide further details can be found here: http://rhsummit.files.wordpress.com/2013/06/noceda_t_0120_consumepaasinthecloudwithopenshift.pdf

Demo

• Sign in for free account on https://openshift.redhat.com
• Login and from "My App" tab create new application by selecting "Add Application ..." button.
• I've used as simple Python app based on CherryPy framework. 
• Once your gear (a Openshift name for your app container) is created you can take a look at the hello world code 
• To modify the source code clone the git repo. You will need to first register your public key.
• Generate a new ssh key pair.

$  ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/rado/.ssh/id_rsa): /tmp/tmp.key

$ ssh-agent | tee -a agent.sh
SSH_AUTH_SOCK=/tmp/ssh-pMmGGPP15635/agent.15635; export SSH_AUTH_SOCK;
SSH_AGENT_PID=15636; export SSH_AGENT_PID;
echo Agent pid 15636;

$source agent.sh
$ssh-add tmp.key
Identity added: tmp.key (tmp.key)

• Clone your source repo to modify your app source code.

$ git clone ssh://52c499a1e0b8cdda0f000016@rado-rado1stapp.rhcloud.com/~/git/rado.git/
Cloning into 'rado'...
remote: Counting objects: 62, done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 62 (delta 16), reused 62 (delta 16)
Receiving objects: 100% (62/62), 18.93 KiB, done.
Resolving deltas: 100% (16/16), done.

• Modify the code to personalize it.

$cd rado/

$ find  | grep -v git
./.openshift
./.openshift/markers
./.openshift/action_hooks
./.openshift/action_hooks/README.md
./.openshift/cron
./.openshift/cron/minutely
./.openshift/cron/monthly
./.openshift/cron/weekly
./.openshift/cron/weekly/chrono.dat
./.openshift/cron/weekly/chronograph
./.openshift/cron/weekly/jobs.deny
./.openshift/cron/weekly/README
./.openshift/cron/weekly/jobs.allow
./.openshift/cron/README.cron
./.openshift/cron/daily
./.openshift/cron/hourly
./README.md
./LICENSE
./data
./libs
./wsgi
./wsgi/static
./wsgi/static/README
./wsgi/application
./app.py.disabled
./setup.py

$vim ./wsgi/application
$ cat wsgi/application
import sys
sys.stdout = sys.stderr

import atexit
import threading
import cherrypy

cherrypy.config.update({'environment': 'embedded'})

if cherrypy.__version__.startswith('3.0') and cherrypy.engine.state == 0:
    cherrypy.engine.start(blocking=False)
    atexit.register(cherrypy.engine.stop)

class Root(object):
    def index(self):
        return 'Hello from rado 1st app on openshift ;)!'
    index.exposed = True

application = cherrypy.Application(Root(), script_name=None, config=None)

• Once you are happy with t he code it is time to push changes back to git and redeploy it inside OpenShift.

$ git commit ./wsgi/application
[master 0eccd4d] init
 1 file changed, 1 insertion(+), 1 deletion(-)

$ git push origin master
Counting objects: 7, done.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 393 bytes, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Stopping PYTHON cart
remote: [Wed Jan 01 17:59:08 2014] [warn] PassEnv variable SHELL was undefined
remote: [Wed Jan 01 17:59:08 2014] [warn] PassEnv variable USER was undefined
remote: [Wed Jan 01 17:59:08 2014] [warn] PassEnv variable LOGNAME was undefined
remote: Waiting for stop to finish
remote: Building git ref 'master', commit 0eccd4d
remote: running develop
remote: running egg_info
remote: creating Example_CherryPy.egg-info
remote: writing requirements to Example_CherryPy.egg-info/requires.txt
remote: writing Example_CherryPy.egg-info/PKG-INFO
remote: writing top-level names to Example_CherryPy.egg-info/top_level.txt
remote: writing dependency_links to Example_CherryPy.egg-info/dependency_links.txt
remote: writing requirements to Example_CherryPy.egg-info/requires.txt
remote: writing Example_CherryPy.egg-info/PKG-INFO
remote: writing top-level names to Example_CherryPy.egg-info/top_level.txt
remote: writing dependency_links to Example_CherryPy.egg-info/dependency_links.txt
remote: writing manifest file 'Example_CherryPy.egg-info/SOURCES.txt'
remote: reading manifest file 'Example_CherryPy.egg-info/SOURCES.txt'
remote: writing manifest file 'Example_CherryPy.egg-info/SOURCES.txt'
remote: running build_ext
remote: Creating /var/lib/openshift/52c499a1e0b8cdda0f000016/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/Example-CherryPy.egg-link (link to .)
remote: Example-CherryPy 1.0 is already the active version in easy-install.pth
remote:
remote: Installed /var/lib/openshift/52c499a1e0b8cdda0f000016/app-root/runtime/repo
remote: Processing dependencies for Example-CherryPy==1.0
remote: Searching for CherryPy==3.2.4
remote: Best match: CherryPy 3.2.4
remote: Processing CherryPy-3.2.4-py2.7.egg
remote: CherryPy 3.2.4 is already the active version in easy-install.pth
remote: Installing cherryd script to /var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin
remote:
remote: Using /var/lib/openshift/52c499a1e0b8cdda0f000016/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/CherryPy-3.2.4-py2.7.egg
remote: Finished processing dependencies for Example-CherryPy==1.0
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Preparing build for deployment
remote: Deployment id is f17888e7
remote: Activating deployment
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Starting PYTHON cart
remote: Result: success
remote: Activation status: success
remote: Deployment completed with status: success

• Final test that all worked fine ;)

$ curl -v -s http://rado-rado1stapp.rhcloud.com ;echo
* About to connect() to rado-rado1stapp.rhcloud.com port 80 (#0)
*   Trying 54.221.76.112... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: rado-rado1stapp.rhcloud.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 01 Jan 2014 23:33:20 GMT
< Server: Apache/2.2.22 (Red Hat Enterprise Web Server)
< Content-Length: 40
< Content-Type: text/html;charset=utf-8
< Vary: Accept-Encoding
<
* Connection #0 to host rado-rado1stapp.rhcloud.com left intact
* Closing connection #0

Hello from rado 1st app on openshift ;)!

Simple MongoDB tutorial

MongoDB has become one of the popular NoSQL db engines. It is considered as a NoSQL db and uses some concepts and names that are different from these used in relational database systems.

Alongside MongoDB there are many others NoSQL dbs. An short summary of available db options and theirs features can be found here.

Below are some of my notes how to start with MongoDB.

How to install MongoDB

• Create a cloud instance
• Download the software into your cloud  http://www.mongodb.org/downloads
• Install and start your mongod server process http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

How to create your first table in the NoSQL data base

It is important to note that data bases will be created dynamically after you insert a first data into a collection (e.i insert a row into your table in the db - collection is a name for table in mongo).

http://docs.mongodb.org/manual/reference/sql-comparison/
http://docs.mongodb.org/manual/tutorial/getting-started/
http://www.mkyong.com/mongodb/how-to-create-database-or-collection-in-mongodb/

How to insert data into table

http://docs.mongodb.org/manual/tutorial/getting-started/

# start the shell interpreter
$ mongo
> show dbs
help    (empty)
local   0.078125GB
test    (empty)

> use rado
switched to db rado

> j = { name : "hello  word" }
{ "name" : "hello  word" }
> k = { x : 3 }
{ "x" : 3 }

> db.mycollection1.insert(j)
> db.mycollection1.insert(k)

> show dbs
help    (empty)
local   0.078125GB
rado    0.203125GB  <<<
test    (empty)

> use rado
> show collections
mycollection1  <<<<< 
system.indexes

> db.mycollection1.find()
{ "_id" : ObjectId("52767264795748b715336b87"), "x" : 3 }
{ "_id" : ObjectId("5276726a795748b715336b88"), "name" : "hello  word" }

Mongo shell commands 

http://docs.mongodb.org/manual/reference/method/

Install python API library for MongoDB

• Install the library 

http://docs.mongodb.org/manual/applications/drivers/
http://docs.mongodb.org/ecosystem/drivers/python/

apt-get install python-pymongo

• Customize the bash environment and install ipython 

http://rtomaszewski.blogspot.co.uk/2013/10/home-directory-and-dotfiles-management.html

 apt-get install ipython

Your first Python script that connects to MongoDB 

http://api.mongodb.org/python/current/tutorial.html
http://docs.mongodb.org/manual/reference/method/db.collection.insert/

Example Python code to insert data to our Mongo collection:

 
# rado-hello-world.py

import pymongo
from pymongo import MongoClien

client = MongoClient()
db = client['rado']

db.collection_names()
[u'system.indexes', u'mycollection1']

collection = db['mycollection1']

for doc in collection.find():
   ....:     print doc
   ....:
{u'x': 3.0, u'_id': ObjectId('52767264795748b715336b87')}
{u'_id': ObjectId('5276726a795748b715336b88'), u'name': u'hello  word'}

l = { "from" : "python", "mytext" : "hello from python" }
collection.insert(l)
ObjectId('52767d071d011c22eb1a7de5')

for doc in collection.find():
    print doc
   ....:
{u'x': 3.0, u'_id': ObjectId('52767264795748b715336b87')}
{u'_id': ObjectId('5276726a795748b715336b88'), u'name': u'hello  word'}
{u'_id': ObjectId('52767d071d011c22eb1a7de5'), u'from': u'python', u'mytext': u'hello from python'}

Further info about write operations 

Once of the big differences when working with NoSQL is that they are an implement of CRUD instead of ACID operation.

http://rsmith.co/opinion/technology/2012/11/05/mongodb-gotchas-and-how-to-avoid-them/
http://docs.mongodb.org/manual/core/write-operations/
http://docs.mongodb.org/manual/core/write-concern/
http://api.mongodb.org/python/current/api/pymongo/mongo_client.html#module-pymongo.mongo_client

We could say than that the performance increase is at the cost of security and integrity. If you don't want to lost any data and see errors as you type the commands you can use the options below when connecting to mongo DB. Please keep in mind that this trade off will have some performance implication in busy production systems although

 
# default 
>>> client = MongoClient()
>>> client2 = MongoClient( j=True, fsync=True )

>>> client2.write_concern
 {'fsync': True, 'j': True}

>>> client.write_concern
 {}

Nova release notes and blueprints

To navigate between various systems and versions in Openstack you need understand the concept of blueprints and how they link to the source code.

For every project that is developed under the umbrella of Openstack there is a list of features people are working on. We call them blueprints and they are tracked in a written form on launchpad.

Release notes and blueprints for Openstack Nova project

All Nova blueprints.

Blueprints sorted by the Nova release.

Essex: Release Note, Blueprints
Folsom: Release Note, Blueprints
Grizzly: Release Note, Blueprints

Searching and matching

This isn't a simple task when you want to track down in reverse order what was implemented.
One way of doing this is to search for the blueprint on the blueprint list or in the release note and then browse through the Gerrit code reviews.

In the example below you can see the blueprint name and a link (see the yellow text). Unfortunately the name used in Gerrit and the links are not always pointing directly to each other.

The Project and branch (red color) are not links to the code with the changes but rather indication of the name for project and branch at the time this review was going on.

If you want to download or checkout the code to play with it or test you need to scroll down the the 'Download' section. Alternatively you can see the diff and changes in browser.

https://review.openstack.org/#/c/3035/

Change Iceb27609: blueprint host-aggregates

Change-Id: Iceb27609dc53bf4305c02d7cbc436fba4c4a7256 Owner Armando Migliaccio Project openstack/nova Branch master Topic bp/host-aggregates Uploaded Jan 13, 2012 5:05 PM Updated Jan 17, 2012 9:37 PM Status Merged Permalink

blueprint host-aggregates This is the first of a series of commits that add the host-aggregates capability, as described on the blueprint page. This commit, more precisely, introduces changes to the Nova model: model classes related to aggregates have been added, as well as DB API methods to interact with the model; a sqlalchemy migration script plus a bunch of tests are also part of this changeset. Commits that will follow are going to add: - Extensions to OSAPI Admin, and related python_novaclient mappings - Implementation of the XenAPI virt layer - Integration of OSAPI and virt layer, via the compute_api - smoketests - openstack-manuals documentation These commits will be pushed for review not necessarily in this exact order. Change-Id: Iceb27609dc53bf4305c02d7cbc436fba4c4a7256

Example 'Download' section.

Download

checkoutpullcherry-pickpatch Anonymous HTTP git fetch https://review.openstack.org/openstack/nova refs/changes/35/3035/8 && git checkout FETCH_HEAD

Authentication issue with London cloud account when using Pyrax module

The Python pyrax module is a Rackspace library that integrates various python modules like  python-novaclient to allow a user to use Openstack API with Rackspace cloud. An introduction how to start with it can be found here: pyrax – Python Bindings for the Rackspace Cloud

A simple script that lists currently existed cloud server is below. For more examples and API documentation please refer to the github directories:

https://github.com/rackspace/pyrax/tree/master/docs
https://github.com/rackspace/pyrax/tree/master/samples



Problem

By default an execution of the script generate this error:

 
Traceback (most recent call last):
  File "C:\Users\radoslaw\workspace\api-challenge\challange1.py", line 6, in <module>
    pyrax.set_credential_file(conf)
  File "c:\Python27\lib\site-packages\pyrax\__init__.py", line 215, in set_credential_file
    authenticate=authenticate)
  File "c:\Python27\lib\site-packages\pyrax\rax_identity.py", line 108, in set_credential_file
    self.authenticate()
  File "c:\Python27\lib\site-packages\pyrax\rax_identity.py", line 138, in authenticate
    raise exc.AuthenticationFailed("Incorrect/unauthorized "
pyrax.exceptions.AuthenticationFailed: Incorrect/unauthorized credentials received

By default the function pyrax.set_credential_file(conf)  assumes your cloud account belongs to the USA. For it to work you need to specify the LONDON region like in the commented code above.

pyrax.set_credential_file(conf, "LON")

Good developer traits

I was searching for a way to use regex when searching in Chrome (found here Regular Expression Search Bookmarklet). Accidentally found this interesting blog entry and I though to repost it here.

How I read it is the fact that what really matters is to be good in what you are doing and specializing. The soft skill are nice to have but are of secondary requirements.

Taken from What Really Makes a Good Programmer blog:

Openstack Gerrit code review process details

Are you looking to contribute code to one of the Openstack projects (example include Nova, Quantum or Glance and many more). If the answer is yes, you definitely need to get familiar with the code review that Openstack enforces. A good overview of what this is can be found here Code review process in Openstack uses Zuul.

On a technical site the process has been implemented with a help of Gerrit system. From end user perspective you get access to a nice and good looking page in a browser that helps you to review, comment and approve code changes that should be committed to master repository. A quick and good introduction of what Gerrit do this and how it helps can be found here: Gerrit Code Review - A Quick Introduction.

Example how this interface looks like:

Code Review: https://review.openstack.org

When you select a link you dig into more details: (example) https://review.openstack.org/#/c/23878/

In this screen we can find:

• linked Bug id if exists (https://code.launchpad.net/bugs/1131759) 
• linked Blueprints if exists
• reviewer list
• one or more patches with proposed code changes
• comments and suggestions before the code can be accepted in the mainstream repository

But the best part of Gerrit is its capability to pull the original file and create a diff to visual the changes. An example from the review #23878 is seen below.

https://review.openstack.org/#/c/23878/4/heat/tests/test_api_openstack_v1.py

Text developer editor with Python API

I often need to work between Linux and Windows systems. Under every OS I have my favorite tools I like that help me to get the job done. But there has always been one tool that I wasn't very happy with: a good text editor.

Problem

What is a good cross platform editor with development features that is written and integrated with Python.

Analisis and discussion 

When I code I always like to know the editor so I can quickly and comfortably navigate in a single or multiple files at the same time. I've found recently one that I tend to use more: Sublime. The other one I was using for a long while was Notepad++ but it was only limited to Windows.

Why Sublime works for me:

• You can use it in Linux and Windows
• It has an extensive number of plugins and a packet manager Package Control
• It has a Python API that you can use for plugin development
• Many opensource plugins are available at github alrady (example https://github.com/facelessuser/BracketHighlighter)
• Examples Python API tutorial
• Inside view what is happening under the hood and easy debugging with Control+~ Python console

My simple config

You can view and change all global settings under Menu - Preferences - Settings - Default but a better way is to create a local customized user preferences file.

To modify user settings open the following file under Menu - Preferences - Settings - User and copy or modify these options:

 
"fade_fold_buttons"        : false,
"highlight_line"           : true,
"auto_complete_size_limit" : 44194304,
"tree_animation_enabled"   : true,

My packages

Below are some of my packages I'm using

• https://github.com/SublimeText/Tag
• http://wbond.net/sublime_packages/package_control
• https://github.com/facelessuser/BracketHighlighter
• https://github.com/titoBouzout/SideBarEnhancements
• http://wbond.net/sublime_packages/alignment

How to highlight a whole line in Sublime like in Notepad++ 

In notepad++ when editing your can enable whole like to be highlighted (here are some example screenshots). To achieve the same effect in Sublime you need to enable the highlight_line: true.

References and documentation

http://www.sublimetext.com/2
http://docs.sublimetext.info/en/latest/index.html

https://tutsplus.com/course/improve-workflow-in-sublime-text-2/

How to document python code

There are couple of way how you can document a python code. In this article I'm not trying to compare them but rather to show how a final documentation may look like.

Problem

How many tools can we use to generate documentation from python source.
How a final documentation look like after generation from a source code.

Analysis

When searching on Google we quickly find that the most popular are (there are more that I'm not listing here):

• epydoc
• sphinx 
• doxygen

Example

These are examples how the documentation looks like.

Epydoc - It has as stile of a classical javadoc documentation introduced by Sun when Java was released.

Sphinx - the layout is very different from Epydoc. It seems to be very likable by the python project itself. All the doc for it is generated using it.

Demonstration

Epydoc installation

# python --version
Python 2.7.3
# aptitude show python-epydoc
# aptitude install  python-epydoc
# aptitude install apache2

The source code

Generation of epydoc documentation

# epydoc --verbose --verbose  example_epydoc.py
Building documentation
[  0%] example_epydoc (example_epydoc.py)
Merging parsed & introspected information
[  0%] example_epydoc
Linking imported variables
[  0%] example_epydoc
[ 12%] example_epydoc.MyClass
Indexing documentation
[  0%] example_epydoc
Checking for overridden methods
[ 12%] example_epydoc.MyClass
Parsing docstrings
[  0%] example_epydoc
[ 12%] example_epydoc.MyClass
Inheriting documentation
[ 12%] example_epydoc.MyClass
Sorting & Grouping
[  0%] example_epydoc
[ 12%] example_epydoc.MyClass
Writing HTML docs to 'html'
[  4%] epydoc.css
[  9%] epydoc.js
[ 13%] identifier-index.html
[ 45%] module-tree.html
[ 50%] class-tree.html
[ 54%] help.html
[ 59%] frames.html
[ 63%] toc.html
[ 68%] toc-everything.html
[ 72%] toc-example_epydoc-module.html
[ 77%] example_epydoc-module.html
[ 81%] example_epydoc.MyClass-class.html
[ 86%] example_epydoc-pysrc.html
[ 90%] redirect.html
[ 95%] api-objects.txt
[100%] index.html

Timing summary:
  Building documentation.............     0.2s |=================================================
  Merging parsed & introspected i....     0.0s |
  Linking imported variables.........     0.0s |
  Indexing documentation.............     0.0s |
  Checking for overridden methods....     0.0s |
  Parsing docstrings.................     0.0s |=
  Inheriting documentation...........     0.0s |
  Sorting & Grouping.................     0.0s |
  Writing HTML docs to 'html'........     0.0s |=======

Showing the doc

You have to configure the Apache and points it to the generated 'html' directory. Once the page is loaded in the browser the documentation looks like this:

References

1. http://stackoverflow.com/questions/4126421/how-to-document-python-code-epydoc-doxygen-sphinx
2. http://stackoverflow.com/questions/1125970/python-documentation-generator
3. http://stackoverflow.com/questions/5334531/python-documentation-standard-for-docstring
4. http://epydoc.sourceforge.net/
5. http://sphinx-doc.org/