Introduction into tunneling protocols when deploying cloud network for your cloud infrastructure

Cloud network is a hot topic for cloud providers and hosting companies. In basic the concept should enable and allow tenants to create, manage and destroy network typologies on demand for the cloud servers by using cloud open API. That said, we want to allow a tenant to create an isolated virtual layer 2 network with its own IP subnet.

Before going into further details we have to realize that the problem isn't trivial to solve. The difficulty comes from a fact that the existing network that interconnects hypervizors hosts is not very flexible and adaptable for changes. That physical network architecture was build and tune to  allow to handle all traffic from all cloud VM across you cloud deployment. This represent its strength and limitation as it is not flexible enough when it comes to configure and create many isolated virtual layer 2 or layer 3 networks per single tenant. This is exactly the problem that the cloud network promises to resolve.

At the moment there isn't a single standard how to implement a cloud network. Instead, we have 3 different protocols that were proposed: VXLAN, NVGRE, STT [1].

All these protocols relay on the fact that the hypervisor hosts are interconnected  The implementation of the additional features is done by using tunneling mechanisms. All of them implement a L2 in L3 tunnels by using TCP, UDP or IP datagrams.

A short introduction and more explanation how this works can on the screenshots below that were taken from this video:  Video: Cloud Tunnels @ Cloud Mafia ( slies can be found here http://ifup.org/slides/cloud-tunnels/ )

References

1. VXLAN
2. NVGRE
3. STT
   

Installing Rackspace Private Cloud (Alamo) on a single virtual server using VMware Workstation

If you are looking for info about installation on a physical server you may like this article instead: How to install Rackspace Private Cloud (Alamo) on a single physical server

VMware Workstation

The Openstack installation within a virtual server is almost identical like on a physical one. The main difference is that we have to create a virtual machine that emulated/simulate the CPU extension needed for the nested hypervisor [1]

Practically that means that  you have to enable this box in your VM config.

If you prefere editing of a config file this is the variable that you have to set in *.vmx file

 
.encoding = "windows-1252"
config.version = "8"
virtualHW.version = "8"
numvcpus = "4"
vcpu.hotadd = "TRUE"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "4096"
mem.hotadd = "TRUE"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "Ubuntu 64-bit.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "C:\Users\radoslaw\Downloads\alamo-v2.0.0.iso"
ide1:0.deviceType = "cdrom-image"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.autodetect = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.addressType = "static"
ethernet0.address = "00:50:56:25:71:FB"
usb.present = "TRUE"
ehci.present = "TRUE"
sound.present = "TRUE"
sound.fileName = "-1"
sound.autodetect = "TRUE"
serial0.present = "TRUE"
serial0.fileType = "thinprint"
serial1.present = "TRUE"
serial1.fileType = "file"
serial1.fileName = "test.py"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
usb.vbluetooth.startConnected = "TRUE"
displayName = "Ubuntu 64-bit"
guestOS = "ubuntu-64"
nvram = "Ubuntu 64-bit.nvram"
virtualHW.productCompatibility = "hosted"
vhv.enable = "TRUE"
powerType.powerOff = "hard"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "hard"
extendedConfigFile = "Ubuntu 64-bit.vmxf"
vmci0.id = "-1561829832"
uuid.location = "56 4d 22 7e ee b5 2b 19-3e c1 5e 76 a2 e8 5e 38"
uuid.bios = "56 4d 22 7e ee b5 2b 19-3e c1 5e 76 a2 e8 5e 38"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
usb.pciSlotNumber = "32"
ethernet0.pciSlotNumber = "33"
sound.pciSlotNumber = "34"
ehci.pciSlotNumber = "35"
vmci0.pciSlotNumber = "36"
usb:1.present = "TRUE"
vmotion.checkpointFBSize = "37748736"
usb:1.speed = "2"
usb:1.deviceType = "hub"
usb:1.port = "1"
usb:1.parent = "-1"
tools.remindInstall = "TRUE"
usb:0.present = "TRUE"
usb:0.deviceType = "hid"
usb:0.port = "0"
usb:0.parent = "-1"

References

1. Nested virtualization
   http://www.ibm.com/developerworks/cloud/library/cl-nestedvirtualization/ http://www.veeam.com/blog/nesting-hyper-v-with-vmware-workstation-8-and-esxi-5.html
   
   You can't use Oracle Virtual Box tool as it doesn't support nested virtualisation. https://www.virtualbox.org/ticket/4032


2. http://www.rackspace.com/knowledge_center/article/installing-rackspace-private-cloud-vmware-fusion

How to install Rackspace Private Cloud (Alamo) on a single physical server

Rackspace has released a new version of its private cloud offerings. The new private cloud is based on the lasted Folsom Openstack release and includes many of the latest features.

ISO downloading 

The software can be downloaded fro http://www.rackspace.com/cloud/private/openstack_software/ after you fill in a simple form. Once filled you receive an email with a direct link to download the ISO image (alamo-v2.0.0.iso). The other important link is Getting Started Guide document.

Supported features

In the guide we can find more details about supported and unsupported features.

Supported OpenStack Features:

• Rackspace supports integration with the other components of OpenStack, as well as features such as floating IP address management, security groups, availability zones, and the python-novaclient command line client
• Single and dual NIC configurations
• NFS and ISCSI file storage as backing stores for VM storage
• VNC Proxy
• KVM hypervisor
• Nova Multi Scheduler instead of Filter Scheduler
• Keystone integrated authentication
• Glance integrated image service
• Horizon dashboard
• Cinder block storage service
• Swift object storage service
• Linux and Windows guests to the extent to which they accept handoff from KVM and boot
• Single metadata server running on each device
• Cloud management through OpenStack APIs
• Rackspace also supports the use of Rackspace Cloud Files as a backend for OpenStack
• Image Storage. For information about OpenStack Object Storage, refer to Rackspace Private Cloud OpenStack Object Storage installation

Unsupported OpenStack Features. The following features are not supported yet.

• Nova high availability
• Nova object store
• Nova volumes
• NFS and ISCSI file storage via volumes for guest VMs
• Clustered file system solutions
• xpvnc
• Xen and other hypervisors
• Centralized metadata servers
• Quantum Software Defined Networking

Installation

To start the installer I had to first convert the ISO into a bootable USB (In the guide you find a link to this tool that can help you with this task www.pendrivelinux.com)

Once done change the boot sequence in BIOS, plug your Pendrive and restart the server.

The installation is relatively simple. The couple of questions that you may get confused are about the number of NIC and what subnet do you want to use for the cloud servers.

Below are some of my screenshots to demonstrate how it looks like (sorry for the quality)

The installation welcome screen

There will be couple of questions for IP address, gateway, netmask, DNS, user, password that you have to answer as well. Random screenshots are showed below.

During the whole installation you can switch to console 4 (ALT+4) and observe various debugging logs as the installation progresses.

At some point the phase 1 of 2 will be completed and the system reboots. Once the system come back we can see different logs showing how the chef progress as Openstack is configured.

At the end of phase 2/2 when the installation is completed you are going to see a screen with summaries about the server resources.

References

1. http://www.rackspace.com/cloud/private/openstack_software/
2. http://www.rackspace.com/knowledge_center/article/installing-rackspace-private-cloud-on-physical-hardware

Openstack popularity trends over time since 2010

It is a difficult task to measure a popularity of a product, item, thing or a software. But with a help of Internet we can use specialized search engines that track internet messages and post around the world for a particular string, key work or some for of textual identity.

An simple but powerful example of such an search engine is trend search on Google: http://www.google.com/trends .

Problem

How popular is Openstack.
Who uses Openstack.

Analysis

The below graphs and stats are generated by using this link: http://www.google.com/trends/explore#q=openstack.

Q: Is Openstack getting more and more popular?
Q: How popular is Openstack?

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)


Q:What is a country where Openstack is the most popular?
A: China

Q: Is the USA the most popular country for searches that include Openstack?
A: No


Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)


Q: Is Europe interested in Openstack?
A: Yes

Q: In what European countries is Openstack the most popular?
A: In 2012: UK, Germany, France, Spain, Sweden and Poland

Q: In what cities is Openstack the most popular?

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)

What does Software Defined Data Center means

After the industry created a Software Defined Network (SDN) [1] term it is time for a new one. A new emerging IT buzz word is a Software Defined Data Center (SDDC) [2]. It appears that only VMware is marketing this extensively at the moment.

From a technical point of view its all makes sense: compute resources are already largely virtualized, virtual storage and virtual networks are following. Looking at the last VMware acquisition of Nicira [3] the company has many if not all necessary products to build such a SDDC Data Center.

Let's see how the market will respond to it and if other vendors start looking and using this as well in the near future.

References

1. SDN
http://rtomaszewski.blogspot.co.uk/2012/10/software-defined-network-sdn-as-example.html
http://rtomaszewski.blogspot.co.uk/2012/10/google-does-use-sdn-in-its-data-centers.html
http://rtomaszewski.blogspot.co.uk/2012/09/emerging-of-virtual-network-aka-quantum.html


2. SDDC
http://www.networkcomputing.com/data-center/the-software-defined-data-center-dissect/240006848
http://www.vmware.com/solutions/datacenter/software-defined-datacenter/index.html
http://blogs.vmware.com/console/2012/08/the-software-defined-datacenter-meets-vmworld.html



3. Openstack, Nicira and VMware
http://www.infoworld.com/t/data-center/what-the-software-defined-data-center-really-means-199930

What your Dev, Engineering and sales teams can achieve by using Openstack cloud and Quantum network

Nicira NVP controller can be used together with Quantum as a plugin to enable extend virtual networking capabilities in Openstack. With NVP help the providers, operators and users can start building Software Defined Network (SDN) networks to interconnect the virtual machines.

It was predicted and by many it is seen as a natural evolution and a step forward for Openstack and cloud network in general.

We are no longer tight to static, unchangeable set of switches and routers when it comes to building and designing networks. The more flexible networking allow us to build more agile cloud infrastructures that will allow us to meet changing demands of rapidly evolving applications.

As a real business example of cloud flexibility this is a video of how Nicira is using Quantum to solve its internal infrastructure challenges for the Dev, Engineering and Sales teams: Running Quantum on Quantum @ Nicira's Multi-tenant OpenStack.

For these who don't like videos I have copied some of the key point from the presentation:

• In a physical word it is difficult to achieve full automation and implement provisioning process that can build new configuration in a few minutes only 
• This is a challenge for the operational team to deliver results in a rapidly changing environment and keep the some quality of work when your company grows fast
• There is always a risk of a human error that can potentially bring your production network down
• Most network are designed to meet only the requirements that are known initially; It is difficult to plan ahead and new changes can be impossible to implement later on
• Developers need to have a flexibility to test without a risk of bringing a company network down
• You can easily provision new isolated environments for new projects or new hires
• For better resource utilization you want to have an automation in place that allows you to tier down and spin up whole environments (cloud servers + networking) as easy as possible during a day
• Not only a compute resources have to be easy to provision but as well as interconnecting and isolating them to achieve infrastructure resources agility; This is important especially for tier application typologies where we have web, db and application servers. 
• Quickly building POC for new projects and customers demonstration
• Flexible network infrastructure allows collaboration between different development teams; breaking the network isolation boundaries in a statically built network may be impossible
• Within minutes or second you can create and build new networks as well as to decommission them
• You can add new resources and boost capacity when you need it; More flexible handling of cloud bursting; You don't have to oversubscribe resource only to meet the pick application requirements

And at the end copies of the interesting slides from the above presentation.

How to extract captions from a youtube video

There are videos on youtube that are 30min and longer. You've watched a video and later on you would like to find only a single moment that is interesting for you. With a help of subtitles and key word searching we could easily do it.

Problem

How to download and search in Google text captions file from a youtube video.

Solution

There are couple of solution that can be found on Google in the referece section. All the automatic ways didn't work for me. Below is a manual method that I used to extract the subtitles (caption) file from a Google video.

• Open a video page in Chrome browser. 
• Enable debuging and reqeuest trucking by pressing F12 in Chrome.
• Enable caption in the video. 
• Navigate into the Network tab and find the last timedtext request (the last at the bottom)
• Right click on it and open that file in a new tab; An xml file containing subtitles with the imestamps should be opened. 

References

1. http://mvark.blogspot.in/2012/06/how-to-extract-subtitles-from-youtube.html
2. http://stackoverflow.com/questions/10036796/how-to-extract-subtitles-from-youtube-videos
3. http://webapps.stackexchange.com/questions/25072/how-do-i-download-subtitles-from-a-youtube-video
4. http://www.youtube.com/watch?v=drTyNDRnyxs