Search This Blog

Friday, November 30, 2012

Introduction into tunneling protocols when deploying cloud network for your cloud infrastructure

Cloud network is a hot topic for cloud providers and hosting companies. In basic the concept should enable and allow tenants to create, manage and destroy network typologies on demand for the cloud servers by using cloud open API. That said, we want to allow a tenant to create an isolated virtual layer 2 network with its own IP subnet.

Before going into further details we have to realize that the problem isn't trivial to solve. The difficulty comes from a fact that the existing network that interconnects hypervizors hosts is not very flexible and adaptable for changes. That physical network architecture was build and tune to  allow to handle all traffic from all cloud VM across you cloud deployment. This represent its strength and limitation as it is not flexible enough when it comes to configure and create many isolated virtual layer 2 or layer 3 networks per single tenant. This is exactly the problem that the cloud network promises to resolve.

At the moment there isn't a single standard how to implement a cloud network. Instead, we have 3 different protocols that were proposed: VXLAN, NVGRE, STT [1].

All these protocols relay on the fact that the hypervisor hosts are interconnected  The implementation of the additional features is done by using tunneling mechanisms. All of them implement a L2 in L3 tunnels by using TCP, UDP or IP datagrams.

A short introduction and more explanation how this works can on the screenshots below that were taken from this video:  Video: Cloud Tunnels @ Cloud Mafia ( slies can be found here http://ifup.org/slides/cloud-tunnels/ )






References
  1. VXLAN
  2. NVGRE
  3. STT

Friday, November 23, 2012

Installing Rackspace Private Cloud (Alamo) on a single virtual server using VMware Workstation

If you are looking for info about installation on a physical server you may like this article instead: How to install Rackspace Private Cloud (Alamo) on a single physical server

VMware Workstation

The Openstack installation within a virtual server is almost identical like on a physical one. The main difference is that we have to create a virtual machine that emulated/simulate the CPU extension needed for the nested hypervisor [1]

Practically that means that  you have to enable this box in your VM config.


If you prefere editing of a config file this is the variable that you have to set in *.vmx file
 
.encoding = "windows-1252"
config.version = "8"
virtualHW.version = "8"
numvcpus = "4"
vcpu.hotadd = "TRUE"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "4096"
mem.hotadd = "TRUE"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "Ubuntu 64-bit.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "C:\Users\radoslaw\Downloads\alamo-v2.0.0.iso"
ide1:0.deviceType = "cdrom-image"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.autodetect = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.addressType = "static"
ethernet0.address = "00:50:56:25:71:FB"
usb.present = "TRUE"
ehci.present = "TRUE"
sound.present = "TRUE"
sound.fileName = "-1"
sound.autodetect = "TRUE"
serial0.present = "TRUE"
serial0.fileType = "thinprint"
serial1.present = "TRUE"
serial1.fileType = "file"
serial1.fileName = "test.py"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
usb.vbluetooth.startConnected = "TRUE"
displayName = "Ubuntu 64-bit"
guestOS = "ubuntu-64"
nvram = "Ubuntu 64-bit.nvram"
virtualHW.productCompatibility = "hosted"
vhv.enable = "TRUE"
powerType.powerOff = "hard"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "hard"
extendedConfigFile = "Ubuntu 64-bit.vmxf"
vmci0.id = "-1561829832"
uuid.location = "56 4d 22 7e ee b5 2b 19-3e c1 5e 76 a2 e8 5e 38"
uuid.bios = "56 4d 22 7e ee b5 2b 19-3e c1 5e 76 a2 e8 5e 38"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
usb.pciSlotNumber = "32"
ethernet0.pciSlotNumber = "33"
sound.pciSlotNumber = "34"
ehci.pciSlotNumber = "35"
vmci0.pciSlotNumber = "36"
usb:1.present = "TRUE"
vmotion.checkpointFBSize = "37748736"
usb:1.speed = "2"
usb:1.deviceType = "hub"
usb:1.port = "1"
usb:1.parent = "-1"
tools.remindInstall = "TRUE"
usb:0.present = "TRUE"
usb:0.deviceType = "hid"
usb:0.port = "0"
usb:0.parent = "-1"

References
  1. Nested virtualization
    http://www.ibm.com/developerworks/cloud/library/cl-nestedvirtualization/ http://www.veeam.com/blog/nesting-hyper-v-with-vmware-workstation-8-and-esxi-5.html

    You can't use Oracle Virtual Box tool as it doesn't support nested virtualisation. https://www.virtualbox.org/ticket/4032

  2. http://www.rackspace.com/knowledge_center/article/installing-rackspace-private-cloud-vmware-fusion

Thursday, November 22, 2012

How to install Rackspace Private Cloud (Alamo) on a single physical server

Rackspace has released a new version of its private cloud offerings. The new private cloud is based on the lasted Folsom Openstack release and includes many of the latest features.

ISO downloading 

The software can be downloaded fro http://www.rackspace.com/cloud/private/openstack_software/ after you fill in a simple form. Once filled you receive an email with a direct link to download the ISO image (alamo-v2.0.0.iso). The other important link is Getting Started Guide document.

Supported features

In the guide we can find more details about supported and unsupported features.

Supported OpenStack Features:
  • Rackspace supports integration with the other components of OpenStack, as well as features such as floating IP address management, security groups, availability zones, and the python-novaclient command line client
  • Single and dual NIC configurations
  • NFS and ISCSI file storage as backing stores for VM storage
  • VNC Proxy
  • KVM hypervisor
  • Nova Multi Scheduler instead of Filter Scheduler
  • Keystone integrated authentication
  • Glance integrated image service
  • Horizon dashboard
  • Cinder block storage service
  • Swift object storage service
  • Linux and Windows guests to the extent to which they accept handoff from KVM and boot
  • Single metadata server running on each device
  • Cloud management through OpenStack APIs
  • Rackspace also supports the use of Rackspace Cloud Files as a backend for OpenStack
  • Image Storage. For information about OpenStack Object Storage, refer to Rackspace Private Cloud OpenStack Object Storage installation
Unsupported OpenStack Features. The following features are not supported yet.
  • Nova high availability
  • Nova object store
  • Nova volumes
  • NFS and ISCSI file storage via volumes for guest VMs
  • Clustered file system solutions
  • xpvnc
  • Xen and other hypervisors
  • Centralized metadata servers
  • Quantum Software Defined Networking
Installation

To start the installer I had to first convert the ISO into a bootable USB (In the guide you find a link to this tool that can help you with this task www.pendrivelinux.com)

Once done change the boot sequence in BIOS, plug your Pendrive and restart the server.

The installation is relatively simple. The couple of questions that you may get confused are about the number of NIC and what subnet do you want to use for the cloud servers.

Below are some of my screenshots to demonstrate how it looks like (sorry for the quality)

The installation welcome screen


There will be couple of questions for IP address, gateway, netmask, DNS, user, password that you have to answer as well. Random screenshots are showed below.




During the whole installation you can switch to console 4 (ALT+4) and observe various debugging logs as the installation progresses.


At some point the phase 1 of 2 will be completed and the system reboots. Once the system come back we can see different logs showing how the chef progress as Openstack is configured.


At the end of phase 2/2 when the installation is completed you are going to see a screen with summaries about the server resources.


References
  1. http://www.rackspace.com/cloud/private/openstack_software/
  2. http://www.rackspace.com/knowledge_center/article/installing-rackspace-private-cloud-on-physical-hardware

Monday, November 19, 2012

Openstack popularity trends over time since 2010

It is a difficult task to measure a popularity of a product, item, thing or a software. But with a help of Internet we can use specialized search engines that track internet messages and post around the world for a particular string, key work or some for of textual identity.

An simple but powerful example of such an search engine is trend search on Google: http://www.google.com/trends .

Problem

How popular is Openstack.
Who uses Openstack.

Analysis

The below graphs and stats are generated by using this link: http://www.google.com/trends/explore#q=openstack.

Q: Is Openstack getting more and more popular?
Q: How popular is Openstack?

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)


Q:
What is a country where Openstack is the most popular?
A: China

Q: Is the USA the most popular country for searches that include Openstack?
A: No


Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)


Q:
Is Europe interested in Openstack?
A: Yes

Q: In what European countries is Openstack the most popular?
A: In 2012: UK, Germany, France, Spain, Sweden and Poland

Q: In what cities is Openstack the most popular?

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)

Sunday, November 18, 2012

What does Software Defined Data Center means

After the industry created a Software Defined Network (SDN) [1] term it is time for a new one. A new emerging IT buzz word is a Software Defined Data Center (SDDC) [2]. It appears that only VMware is marketing this extensively at the moment.

From a technical point of view its all makes sense: compute resources are already largely virtualized, virtual storage and virtual networks are following. Looking at the last VMware acquisition of Nicira [3] the company has many if not all necessary products to build such a SDDC Data Center.



Let's see how the market will respond to it and if other vendors start looking and using this as well in the near future.

References
  1. SDN
  2. http://rtomaszewski.blogspot.co.uk/2012/10/software-defined-network-sdn-as-example.html
    http://rtomaszewski.blogspot.co.uk/2012/10/google-does-use-sdn-in-its-data-centers.html
    http://rtomaszewski.blogspot.co.uk/2012/09/emerging-of-virtual-network-aka-quantum.html

  3. SDDC
  4. http://www.networkcomputing.com/data-center/the-software-defined-data-center-dissect/240006848
    http://www.vmware.com/solutions/datacenter/software-defined-datacenter/index.html
    http://blogs.vmware.com/console/2012/08/the-software-defined-datacenter-meets-vmworld.html


  5. Openstack, Nicira and VMware
  6. http://www.infoworld.com/t/data-center/what-the-software-defined-data-center-really-means-199930

What your Dev, Engineering and sales teams can achieve by using Openstack cloud and Quantum network

Nicira NVP controller can be used together with Quantum as a plugin to enable extend virtual networking capabilities in Openstack. With NVP help the providers, operators and users can start building Software Defined Network (SDN) networks to interconnect the virtual machines.

It was predicted and by many it is seen as a natural evolution and a step forward for Openstack and cloud network in general.

We are no longer tight to static, unchangeable set of switches and routers when it comes to building and designing networks. The more flexible networking allow us to build more agile cloud infrastructures that will allow us to meet changing demands of rapidly evolving applications.

As a real business example of cloud flexibility this is a video of how Nicira is using Quantum to solve its internal infrastructure challenges for the Dev, Engineering and Sales teams: Running Quantum on Quantum @ Nicira's Multi-tenant OpenStack.



For these who don't like videos I have copied some of the key point from the presentation:
  • In a physical word it is difficult to achieve full automation and implement provisioning process that can build new configuration in a few minutes only 
  • This is a challenge for the operational team to deliver results in a rapidly changing environment and keep the some quality of work when your company grows fast
  • There is always a risk of a human error that can potentially bring your production network down
  • Most network are designed to meet only the requirements that are known initially; It is difficult to plan ahead and new changes can be impossible to implement later on
  • Developers need to have a flexibility to test without a risk of bringing a company network down
  • You can easily provision new isolated environments for new projects or new hires
  • For better resource utilization you want to have an automation in place that allows you to tier down and spin up whole environments (cloud servers + networking) as easy as possible during a day
  • Not only a compute resources have to be easy to provision but as well as interconnecting and isolating them to achieve infrastructure resources agility; This is important especially for tier application typologies where we have web, db and application servers. 
  • Quickly building POC for new projects and customers demonstration
  • Flexible network infrastructure allows collaboration between different development teams; breaking the network isolation boundaries in a statically built network may be impossible
  • Within minutes or second you can create and build new networks as well as to decommission them
  • You can add new resources and boost capacity when you need it; More flexible handling of cloud bursting; You don't have to oversubscribe resource only to meet the pick application requirements
And at the end copies of the interesting slides from the above presentation.






How to extract captions from a youtube video

There are videos on youtube that are 30min and longer. You've watched a video and later on you would like to find only a single moment that is interesting for you. With a help of subtitles and key word searching we could easily do it.

Problem

How to download and search in Google text captions file from a youtube video.

Solution

There are couple of solution that can be found on Google in the referece section. All the automatic ways didn't work for me. Below is a manual method that I used to extract the subtitles (caption) file from a Google video.

  • Open a video page in Chrome browser. 
  • Enable debuging and reqeuest trucking by pressing F12 in Chrome.
  • Enable caption in the video. 
  • Navigate into the Network tab and find the last timedtext request (the last at the bottom)
  • Right click on it and open that file in a new tab; An xml file containing subtitles with the imestamps should be opened. 

References
  1. http://mvark.blogspot.in/2012/06/how-to-extract-subtitles-from-youtube.html
  2. http://stackoverflow.com/questions/10036796/how-to-extract-subtitles-from-youtube-videos
  3. http://webapps.stackexchange.com/questions/25072/how-do-i-download-subtitles-from-a-youtube-video
  4. http://www.youtube.com/watch?v=drTyNDRnyxs

Thursday, November 15, 2012

What network topologies can I build with Openstack Quantum server

The Folsom Openstack release brings many enhancements to networking stack in the Openstack version of the cloud. The features have been encapsulated with in a new core project called Quantum. But even looking at the documentation it can be very difficult at the beginning to understand what are the differences to old nova-network solution and how an example Quantum virtual network can be configured.

Fortunately there is an easy to follow video introduction to Quantum from last Summit [1]. The slides to it can be found here as well [2].

Below are example slides from the presentation of how a virtual network can be build and how it can look like in cloud powered by Openstack, examples: single flat network, multiple flat networks, mixed flat plus private networks, single provider network, multiple per tenant private networks plus single provider network.






References

  1. OpenStack Networking (Quantum) Project Update 
  2. http://www.slideshare.net/danwent/quantum-grizzly-summit 
  3. More slides about Quantum from Dan Wendlandt at slideshare

Wednesday, November 14, 2012

How many Linux distributions provide officially Openstack support

In the same time as Openstack gets more mature the number of Python code line increases and the software become more complex. The usual tasks like installation, support and operational maintenance start to get more complicated and require more advance skills.

Seeing the opportunities on the market companies like Rackspace, Red Hat and Ubuntu rolling out new processes for sales and support to address these needs. As a prove of evidence below is couple of screenshots from Openstack Ubuntu presentation The promise of the Open Cloud with further links in references section.



References

http://www.rackspace.com/cloud/openstack/ 
http://www.redhat.com/about/news/press-archive/2012/8/red-hat-announces-preview-version-of-enterprise-ready-openstack-distribution
http://www.ubuntu.com/cloud/private-cloud/openstack 
Videos from 2012 Openstack Summit in San Diego

Videos from 2012 Openstack Summit in San Diego

Here you can find videos for these of us who couldn't participate in the 2012 Openstack Summit in San Diego:

At Openstack home page: Openstack summit sessions/
At Openstack chanell  on youtube: OpenStackFoundation

Tuesday, November 13, 2012

What happens to FTPS data channel if client closes control connection

There are couple of extensions added to standard FTP protocol to make it secure. It is important as in the default FTP configuration the control as well as the data channel use clear text to exchange commands or transmit data.

Problem

We assume that we were able to established a successful FTPS base session between a client and server. The client started a new data session to download a large file from the server or is uploading a file using the passive mode.

What happens to a file transfer if the control session is terminated by the client.

Troubleshooting

To verify the scenario we are going to setup a simple test scenario like in Does IPv4 based FTPS server supports EPSV FTP protocol extension blog [1].

As the curl client by default is not closing the control connections (what is a correct behavior that we will discuss at the end of this blog) we are going to use an active method to close an established tcp session described here How to forcibly kill an established TCP connection in Linux  [2].

Test #1: client download a large file

Client logs

Logs when the control connection is being closed and reseted

root@clinet:~# netstat -tulpan | grep curl
tcp        0      0 5.79.21.166:45707       5.79.17.48:8000         ESTABLISHED 5546/curl
tcp    64210      0 5.79.21.166:43796       5.79.17.48:8011         ESTABLISHED 5546/curl

root@clinet:~# ./killcx.pl 5.79.17.48:8011
killcx v1.0.3 - (c)2009-2011 Jerome Bruandet - http://killcx.sourceforge.net/

[PARENT] checking connection with [5.79.17.48:8011]
[PARENT] found connection with [5.79.21.166:43796] (ESTABLISHED)
[PARENT] forking child
[CHILD]  interface not defined, will use [eth0]
[CHILD]  setting up filter to sniff ACK on [eth0] for 5 seconds
[CHILD]  hooked ACK from [5.79.21.166:43796]
[CHILD]  found AckNum [1229126485] and SeqNum [3095306962]
[CHILD]  sending spoofed RST to [5.79.21.166:43796] with SeqNum [1229126485]
[CHILD]  sending RST to remote host as well with SeqNum [3095306962]
[CHILD]  all done, sending USR1 signal to parent [5781] and exiting
[PARENT] received child signal, checking results...
         => success : connection has been closed !

These are the client logs from the start of downloading until the control session is closed.

root@client:~# curl -v --limit-rate 10K -o file.txt -u rado:pass -k --ftp-ssl ftp://5.79.17.48:8000/c2900-universalk9-mz.SPA.152-1.T.bin
* About to connect() to 5.79.17.48 port 8000 (#0)
*   Trying 5.79.17.48...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
< 220-FileZilla Server version 0.9.41 beta
< 220-written by Tim Kosse (Tim.Kosse@gmx.de)
< 220 Please visit http://sourceforge.net/projects/filezilla/
> AUTH SSL
< 234 Using authentication type SSL
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> USER rado
< 331 Password required for rado
> PASS pass
< 230 Logged on
> PBSZ 0
< 200 PBSZ=0
> PROT P
< 200 Protection level set to P
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||8011|)
*   Trying 5.79.17.48... connected
* Connecting to 5.79.17.48 (5.79.17.48) port 8011
> TYPE I
< 200 Type set to I
> SIZE c2900-universalk9-mz.SPA.152-1.T.bin
< 213 77200652
> RETR c2900-universalk9-mz.SPA.152-1.T.bin
< 150 Connection accepted
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL re-using session ID
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
* Maxdownload = -1
* Getting file with size: 77200652
{ [data not shown]
  0 73.6M    0  616k    0     0  10095      0  2:07:27  0:01:02  2:06:25  9753* SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
  0 73.6M    0  620k    0     0  10160      0  2:06:38  0:01:02  2:05:36 11170
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

Server logs

As the file download started this is logged on the server.


After the client control connections is terminated the server logs '426 Connection closed' tranfer aborted' log message.


After about 3-5 seconds after the connections clears from the server logs.


Test #2: client upload a large file

Client logs

The client logs when control channel is terminated.

root@client:~# netstat -tulpan | grep curl
tcp        0      0 5.79.21.166:43489       5.79.17.48:8016         ESTABLISHED 13177/curl
tcp        0      0 5.79.21.166:45717       5.79.17.48:8000         ESTABLISHED 13177/curl

root@client:~# ./killcx.pl  5.79.17.48:8016 
killcx v1.0.3 - (c)2009-2011 Jerome Bruandet - http://killcx.sourceforge.net/

[PARENT] checking connection with [5.79.17.48:8016]
[PARENT] found connection with [5.79.21.166:43489] (ESTABLISHED)
[PARENT] forking child
[CHILD]  interface not defined, will use [eth0]
[CHILD]  setting up filter to sniff ACK on [eth0] for 5 seconds
[PARENT] sending spoofed SYN to [5.79.21.166:43489] with bogus SeqNum
[CHILD]  hooked ACK from [5.79.21.166:43489]
[CHILD]  found AckNum [781536832] and SeqNum [2094006657]
[CHILD]  sending spoofed RST to [5.79.21.166:43489] with SeqNum [781536832]
[CHILD]  sending RST to remote host as well with SeqNum [2094006657]
[CHILD]  all done, sending USR1 signal to parent [13547] and exiting
[PARENT] received child signal, checking results...
         => success : connection has been closed !

Curl logs when the upload starts and the control channel is terminated.

root@client:~# curl -v --limit-rate 10K -T file.txt -u rado:pass -k --ftp-ssl ftp://5.79.17.48:8000/
* About to connect() to 5.79.17.48 port 8000 (#0)
*   Trying 5.79.17.48...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
< 220-FileZilla Server version 0.9.41 beta
< 220-written by Tim Kosse (Tim.Kosse@gmx.de)
< 220 Please visit http://sourceforge.net/projects/filezilla/
> AUTH SSL
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< 234 Using authentication type SSL
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> USER rado
< 331 Password required for rado
> PASS pass
< 230 Logged on
> PBSZ 0
< 200 PBSZ=0
> PROT P
< 200 Protection level set to P
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||8016|)
*   Trying 5.79.17.48... connected
* Connecting to 5.79.17.48 (5.79.17.48) port 8016
> TYPE I
< 200 Type set to I
> STOR file.txt
< 150 Connection accepted
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL re-using session ID
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
} [data not shown]
  0 73.6M    0     0    0  688k      0  10122  2:07:07  0:01:09  2:05:58  9814* SSL_write() returned SYSCALL, errno = 10422:51:35
  0 73.6M    0     0    0  688k      0  10122  2:07:07  0:01:09  2:05:58  8177
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
curl: (55) SSL_write() returned SYSCALL, errno = 104

Server logs

When the upload starts and 1-3 seconds after the control channel is closed.




Results discussion

We can see that every time the client closes the TCP session used to host the control channel bad things happen to the upload or download process.

This is expected behavior and is documented in the relevant RFC documents:


http://tools.ietf.org/html/rfc4217
7. Data Connection Behaviour

http://tools.ietf.org/html/rfc959
3.2.  ESTABLISHING DATA CONNECTIONS

The server MUST close the data connection under the following conditions:

         1. The server has completed sending data in a transfer mode
            that requires a close to indicate EOF.

         2. The server receives an ABORT command from the user.

         3. The port specification is changed by a command from the
            user.

         4. The control connection is closed legally or otherwise.

         5. An irrecoverable error condition occurs.


References
  1. http://rtomaszewski.blogspot.co.uk/2012/11/does-ipv4-based-ftps-server-supports.html
  2. http://rtomaszewski.blogspot.co.uk/2012/11/how-to-forcibly-kill-established-tcp.html

Monday, November 12, 2012

Does IPv4 based FTPS server supports EPSV FTP protocol extension

FTP Extension description

The EPSV stands for Extended Passive Mode and is defined in RFC 2428 [1].

According to the RFC specification it is used for:

This paper specifies extensions to FTP that will allow the protocol to work over IPv4 and IPv6. 
...
The EPRT command allows for the specification of an extended address for the data connection. 
... 
The following are sample EPRT commands: 
   EPRT |1|132.235.1.2|6275| 
   EPRT |2|1080::8:800:200C:417A|5282|

In the RFC I couldn't find any word about default values or how the server should behave if the client doesn't provide any additional arguments and used the command in this simple way:

EPSV

Test configuration

To verify the ftp extension I build a simple test scenario using Rackspace cloud:

  • Windows 2008 cloud server running FTPS server; I used FileZilla Server [2]
  • Ubuntu 12.04 LTS Linux base system acting as a client; we used curl tool to simulate FTPS requests
Setting up of the clouds, ftps server and client are relatively simple so we are not going to describe these here. After FileZilla Server was installed I enabled the FTPS and a little bit customized the standard configuration. The screenshots below show the relevant settings.





Client connection

Below are client logs when we try to download a file from ftps server.

root@client:~# curl -v -o tmp -u user:pass -k --ftp-ssl ftp://<server_ip>:8000/file.txt
* About to connect() to 5.79.17.48 port 8000 (#0)
< 220-FileZilla Server version 0.9.41 beta
< 220-written by Tim Kosse (Tim.Kosse@gmx.de)
< 220 Please visit http://sourceforge.net/projects/filezilla/
> AUTH SSL
< 234 Using authentication type SSL
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> USER user
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< 331 Password required for user
> PASS pass
< 230 Logged on
> PBSZ 0
< 200 PBSZ=0
> PROT P
< 200 Protection level set to P
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||8007|)
*   Trying 5.79.17.48... connected
* Connecting to 5.79.17.48 (5.79.17.48) port 8007
> TYPE I
< 200 Type set to I
> SIZE file.txt
< 213 77200652
> RETR file.txt
< 150 Connection accepted
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL re-using session ID
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
* Maxdownload = -1
* Getting file with size: 77200652
{ [data not shown]


FileZilla server connection logs

As the client connects and start the session these are the logs we can observe on the serve.

Creating listen socket on port 8000...
Creating listen socket on port 990...
Server online
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> Connected, sending welcome message...
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 220-FileZilla Server version 0.9.41 beta
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 220-written by Tim Kosse (Tim.Kosse@gmx.de)
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 220 Please visit http://sourceforge.net/projects/filezilla/
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> AUTH SSL
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 234 Using authentication type SSL
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> SSL connection established
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> USER user
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 331 Password required for user
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> PASS ********
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 230 Logged on
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> PBSZ 0
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 200 PBSZ=0
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> PROT P
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 200 Protection level set to P
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> PWD
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 257 "/" is current directory.
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> EPSV
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 229 Entering Extended Passive Mode (|||8007|)
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> TYPE I
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 200 Type set to I
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> SIZE c2900-universalk9-mz.SPA.152-1.T.bin
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 213 77200652
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> RETR c2900-universalk9-mz.SPA.152-1.T.bin
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 150 Connection accepted
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> SSL connection for data connection established
(000022)11/12/2012 22:34:33 PM - user (5.79.21.166)> 426 Connection closed; transfer aborted.

Summary

We can see that the EPSV extension can be used even on a server that has only IPv4 addresses. It is not a surprise as the RFC clearly defines that both protocols are supported (IPv6 and IPv4).

What is interesting is the server that once receives the EPSV command that is sent by the client using IPv4 it assumes this is the default protocol and defaults itself to IPv4 address.

References
  1. http://tools.ietf.org/html/rfc2428
  2. http://filezilla-project.org/

Understanding network architecture in Openstack

With the new Folsom Openstack release we have now access to new networking architectures called Quantum. Even more, network is no longer only an internal part of nova but rather become a new core component [1].

It is going to take some time before Quantum become the main single option for network configuration. Many small to medium deployments will probably never require such a complicated mechanism to manage network creation, ports assignments or IPs allocation.

Problem
  • How many network configuration options do we have in Folsom.
  • How does the old nova-network work.

Solution

In Folsom we have a choice and we can use either nova-network [1] or quantum [3] service for our  cloud infrastructure.

Depending on the complexity and flexibility we can use the the following configurations in nova-network [2]:


Configuring Flat Networking
Configuring Flat DHCP Networking
Configuring VLAN Networking


This picture shows an example interaction between 2 VMs. More in depth description of how the network flows look like and what role plays the hypervisor in routing or restricting the traffic can be found in [4].


References
  1. http://docs.openstack.org/trunk/openstack-compute/admin/content/networking-options.html
  2. http://docs.openstack.org/trunk/openstack-compute/admin/content/ch_networking.html
  3. http://wiki.openstack.org/Quantum
  4. Mirrantis 
  5. http://www.mirantis.com/blog/openstack-networking-flatmanager-and-flatdhcpmanager/
    http://www.mirantis.com/blog/openstack-networking-single-host-flatdhcpmanager/"
    http://www.mirantis.com/blog/vlanmanager-network-flow-analysis/ 
    http://www.mirantis.com/blog/openstack-networking-vlanmanager/
  6. IBM OpenStack

Event flow when a cloud instance is provisioned in Openstack

Many people think that cloud is alike virtualizastion or other way around. Both technologies have a lot of things in common and with a sophisticated virtualization manager or orchestration tool (example can be a VMware vCenter or vCloud) we get a look and feel that cloud promise but in essence cloud is not the same like virtualization.

In this post instead to polemize who is right or not I would like to rather take a look at the events that are happening when we provision a cloud server, either in cloud or VMware. The inside view is only for the Openstack although.

Problem

What is happening in the Openstack cluster when a cloud server is provisioned and booted.

Solution

Before a VM can be marked as booted and fully provisioned there are number or messages and requests exchanged between the core openstack components[4].

This slides show event after event what is happening: Openstack cloud request flow [2].
This diagram [3] below is an additional visualization of interaction between the components.


References
  1. Python Code level analyse
    http://www.laurentluce.com/posts/openstack-nova-internals-of-instance-launching/
  2. http://www.slideshare.net/mirantis/openstack-cloud-request-flow
  3. Slides on Technical Architecture of Quantum (9/12) from the http://wiki.openstack.org/Quantum
  4. http://www.openstack.org/software

Sunday, November 11, 2012

High level Openstack software architecture

This is an overwhelming task of tracking code changes for Openstack projects as well as to maintain an up to date view of current software architecture model. As the number of core Openstack projects increase it drives as well as the overall complexity and communication patters between the them.

Below is a nice diagram that through visualization helps to represent the complexities and simplify the communications view between all components in Folscom Openstack release.




Sources
  1. http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/
  2. http://ken.pepple.info/openstack/2012/02/21/revisit-openstack-architecture-diablo/

New 2012 Gartner Magic Quadrant for Cloud Infrastructure as a Service (Iaas)

There is a new report from Gartner describing the IaaS main key player who drive the IaaS adoption and evolution as well as comparing the strengths and weakness they have.

The 2012 version of the document can be found here:

Magic Quadrant for Cloud Infrastructure as a Service
18 October 2012 ID:G00237002
Analyst(s): Lydia Leong, Douglas Toombs, Bob Gill, Gregor Petri, Tiny Haynes

With a little bit of history from the old Gartner's reports we can see as well as how the market has changed:





Friday, November 9, 2012

How to forcibly kill an established TCP connection in Linux

During one of the repros I needed to find a way to kill an already established TCP session but without killing the process that opened it.

Problem

How to kill an ESTABLISHED TCP connection in Linux.

Solution #1: passive mechanism

We can use a little tool called tcpkill that you can find desription and example usage here link 

tcpkill -i eth0 { expression }

When started it first listens on a wire for any traffic matching your expression filter (that is compatible with tcpdump expressions). This is passive mechanism to learn about the TCP ACK and SEQ numbers. Once it knows the numbers it spoofs a TCP segments and try to reset the TCP session on both sides.

If the TCP session is idle or in generic there is no data being exchanged than this passive mechanism unfortunately is not going to work.

Solution #2: active mechanism

This killcx tool is taking a different approach and tries to generate traffic on the wire to discover what the ACK and SEQ numbers are. Once it forces the peer to replay to its spoofed traffic it kills the TCP session immediately after.

killcx.pl dest_ip:dest_port 

Simple demonstration
  1. tcpkill (passive)
On 2 different hosts we simulate a daemon that actively listens on incoming traffic and a client that initiate a connection to a server. Once a TCP session is created the client sent single byte. The commands show the outputs.

root@server:~# nc -v -l 7777                                                                                       
Connection from 164.177.146.87 port 7777 [tcp/*] accepted
a

root@client:~# nc -D 5.79.21.166 7777
a
b

Below are tcpdumps showing TCP handshaking and 1 byte exchange on both hosts.

root@server:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:15:59.049739 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [S], seq 2946797440, win 14600, options [mss 1460,sackOK,TS val 726987371 ecr 0,nop,wscale 2], length 0
23:15:59.049803 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [S.], seq 2303834803, ack 2946797441, win 14480, options [mss 1460,sackOK,TS val 350774705 ecr 726987371,nop,wscale 3], length 0
23:15:59.051394 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 726987375 ecr 350774705], length 0
23:16:00.815434 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 726987816 ecr 350774705], length 2
23:16:00.815493 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 3, win 1810, options [nop,nop,TS val 350775146 ecr 726987816], length 0

root@client:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:20:38.159839 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [S], seq 2946797440, win 14600, options [mss 1460,sackOK,TS val 726987371 ecr 0,nop,wscale 2], length 0
23:20:38.174837 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [S.], seq 2303834803, ack 2946797441, win 14480, options [mss 1460,sackOK,TS val 350774705 ecr 726987371,nop,wscale 3], length 0
23:20:38.174880 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 726987375 ecr 350774705], length 0
23:20:39.938895 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 726987816 ecr 350774705], length 2
23:20:39.939313 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 3, win 1810, options [nop,nop,TS val 350775146 ecr 726987816], length 0 

At this point from another terminal on the server we can verify that the connection exists and start the tcpkill to try to kill it.

root@server:~# netstat -tulpan | grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      1338/nc
tcp        0      0 5.79.21.166:7777        164.177.146.87:50185    ESTABLISHED 1338/nc
root@server:~# tcpkill -i eth0 -9 port 50185

At this stage nothing is going to happen because neither the client or server sending any data. The tcpkill simply hangs and waits. The open TCP session continue to exists. But as soon as we sent another byte from the client to the server these is being logged on the tcpdump sessions  for the tcpkill, client and server.

tcpkill

root@server:~# tcpkill -i eth0 -9 port 50185
tcpkill: listening on eth0 [port 50185]
164.177.146.87:50185 > 5.79.21.166:7777: R 2303834804:2303834804(0) win 0             
164.177.146.87:50185 > 5.79.21.166:7777: R 2303838454:2303838454(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303845754:2303845754(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303856704:2303856704(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303871304:2303871304(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303889554:2303889554(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303911454:2303911454(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303937004:2303937004(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303966204:2303966204(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946797445:2946797445(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946799255:2946799255(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946802875:2946802875(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946808305:2946808305(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946815545:2946815545(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946824595:2946824595(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946835455:2946835455(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946848125:2946848125(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946862605:2946862605(0) win 0

client

root@client:~# tcpdump -i any -nn port 7777
...
23:21:03.938535 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 3:5, ack 1, win 3650, options [nop,nop,TS val 726993816 ecr 350775146], length 2
23:21:03.943047 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 5, win 1810, options [nop,nop,TS val 350781146 ecr 726993816], length 0
23:21:03.943070 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303834804, win 0, length 0
23:21:03.943082 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303838454, win 0, length 0
23:21:03.943086 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303845754, win 0, length 0
23:21:03.943090 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303856704, win 0, length 0
23:21:03.943093 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303871304, win 0, length 0
23:21:03.943096 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303889554, win 0, length 0
23:21:03.943099 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303911454, win 0, length 0
23:21:03.943102 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303937004, win 0, length 0 
23:21:03.943106 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303966204, win 0, length 0

Server

root@server:~# tcpdump -i any -nn port 7777
...
23:16:24.815402 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 3:5, ack 1, win 3650, options [nop,nop,TS val 726993816 ecr 350775146], length 2
23:16:24.815449 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 5, win 1810, options [nop,nop,TS val 350781146 ecr 726993816], length 0
23:16:24.816040 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303834804, win 0, length 0
23:16:24.816080 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303838454, win 0, length 0
23:16:24.816103 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303845754, win 0, length 0
23:16:24.816130 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303856704, win 0, length 0
23:16:24.816153 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303871304, win 0, length 0
23:16:24.816177 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303889554, win 0, length 0
23:16:24.816199 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303911454, win 0, length 0
23:16:24.816225 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303937004, win 0, length 0
23:16:24.816245 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303966204, win 0, length 0
23:16:24.816280 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946797445, win 0, length 0
23:16:24.816314 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946799255, win 0, length 0
23:16:24.816339 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946802875, win 0, length 0
23:16:24.816362 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946808305, win 0, length 0
23:16:24.816384 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946815545, win 0, length 0
23:16:24.816407 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946824595, win 0, length 0
23:16:24.816429 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946835455, win 0, length 0
23:16:24.816451 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946848125, win 0, length 0
23:16:24.816474 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946862605, win 0, length 0


We can see that as soon as the new byte generate 2 TCP segment the tcpkill learns about the SEQ and ACK numbers and generate its RST packet to desynchronize and kill the TCP session.
  1. killcx (active)
We start the 2 client and server sessions in a similar way. These are the command output showing what is happening on the wire this time.

server

root@server:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:46:06.228566 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 11257294, win 14600, options [mss 1460,sackOK,TS val 727439173 ecr 0,nop,wscale 2], length 0
23:46:06.228625 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [S.], seq 1319656020, ack 11257295, win 14480, options [mss 1460,sackOK,TS val 351226499 ecr 727439173,nop,wscale 3], length 0
23:46:06.233328 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 727439174 ecr 351226499], length 0
23:46:07.960259 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 727439605 ecr 351226499], length 2
23:46:07.960322 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351226932 ecr 727439605], length 0
# at this moment the TCP is estabilished and the server received one byte

# we see this as soon as the killcx is started
23:46:32.411167 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 10, win 65535, length 0
23:46:32.411194 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351233045 ecr 727439605,nop,nop,sack 1 {4283710012:4283710013}], length 0
23:46:32.413315 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [R], seq 11257297, win 65535, length 0
23:46:32.414945 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [R], seq 1319656021, win 65535, length 0

client

root@client:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:50:45.365610 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 11257294, win 14600, options [mss 1460,sackOK,TS val 727439173 ecr 0,nop,wscale 2], length 0
23:50:45.370015 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [S.], seq 1319656020, ack 11257295, win 14480, options [mss 1460,sackOK,TS val 351226499 ecr 727439173,nop,wscale 3], length 0
23:50:45.370059 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 727439174 ecr 351226499], length 0
23:50:47.097391 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 727439605 ecr 351226499], length 2
23:50:47.098888 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 35122693221:20:14439605], length 0
# at this moment the TCP is estabilished and the client sent one byte

# we see this as soon as the killcx is started
23:51:11.550122 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351233045 ecr 727439605,nop,nop,sack 1 {4283710012:4283710013}], length 0
23:51:11.552854 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [R], seq 1319656021, win 65535, length 0

killcx

root@server:~# netstat -tulpan | grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      10568/nc
tcp        0      0 5.79.21.166:7777        164.177.146.87:50189    ESTABLISHED 10568/nc

root@manage2:~# ./killcx.pl 164.177.146.87:50189
killcx v1.0.3 - (c)2009-2011 Jerome Bruandet - http://killcx.sourceforge.net/

[PARENT] checking connection with [164.177.146.87:50189]
[PARENT] found connection with [5.79.21.166:7777] (ESTABLISHED)
[PARENT] forking child
[CHILD]  interface not defined, will use [eth0]
[CHILD]  setting up filter to sniff ACK on [eth0] for 5 seconds
[PARENT] sending spoofed SYN to [5.79.21.166:7777] with bogus SeqNum
[CHILD]  hooked ACK from [5.79.21.166:7777]
[CHILD]  found AckNum [11257297] and SeqNum [1319656021]
[CHILD]  sending spoofed RST to [5.79.21.166:7777] with SeqNum [11257297]
[CHILD]  sending RST to remote host as well with SeqNum [1319656021]
[CHILD]  all done, sending USR1 signal to parent [10726] and exiting
[PARENT] received child signal, checking results...
         => success : connection has been closed !

As before the TCP session is established and idle waiting. As soon as we start the killcx it successfully spoofs a SYN packet on behalf of the client and sent it to the server. The server than replays with a valid TCP packet revealing the ACK and SEQ numbers. As soon as this this is on a wire the killcx sniffs this up and sent RST to kill the active session.

References
  1. http://killcx.sourceforge.net/
  2. http://www.cyberciti.biz/howto/question/linux/kill-tcp-connection-using-linux-netstat.php

  3. Further reading
  4. http://serverfault.com/questions/179702/is-it-possible-to-close-a-socket-with-a-shell-command
  5. http://superuser.com/questions/127863/manually-closing-a-port-from-commandline

Wednesday, November 7, 2012

What IP addresses are initialy assigned on a new cloud server

I have spun up a standard tiny/small cloud server using the Amazon Cloud Console, Rackspace Cloud Control Panel and the HP Management Console [1]. We can see that for every vendors the provisioned cloud servers have different network settings.

Problem

What is a standard IP and routing configuration on a freshly built cloud server.

Solution
  1. Rackspace

    Example network config for Rackspace NextGen cloud server

    root@manage2:~# ip a
    1: lo: LOOPBACK,UP,LOWER_UP mtu 16436 qdisc noqueue state UNKNOWN
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether bc:76:4e:08:07:f4 brd ff:ff:ff:ff:ff:ff
        inet 5.79.xx.yy/24 brd 5.79.xx.yy scope global eth0
        inet6 2a00:1a48:7805:111:8cfc:xxx:yyy:zzz/64 scope global
           valid_lft forever preferred_lft forever
        inet6 fe80::be76:aaa:bbb:ccc/64 scope link
           valid_lft forever preferred_lft forever
    3: eth1:  BROADCAST,MULTICAST,UP,LOWER_UP  mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether bc:76:4e:08:04:39 brd ff:ff:ff:ff:ff:ff
        inet 10.178.195.rrr/18 brd 10.178.255.255 scope global eth1
        inet6 fe80::be76:4eff:rrrr:sss/64 scope link
           valid_lft forever preferred_lft forever
    
    root@manage2:~# ip route
    default via 5.79.21.1 dev eth0  metric 100
    5.79.21.0/24 dev eth0  proto kernel  scope link  src 5.79.aa.bbb
    10.176.0.0/12 via 10.178.192.1 dev eth1
    10.178.192.0/18 dev eth1  proto kernel  scope link  src 10.178.195.191
    


  2. Amazon

  3. Example network config for Amazon EC2 cloud server

    ubuntu@ip-10-203-43-168:~$ ip a
    1: lo:  LOOPBACK,UP,LOWER_UP  mtu 16436 qdisc noqueue state UNKNOWN
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0:  BROADCAST,MULTICAST,UP,LOWER_UP  mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether 12:31:3b:09:28:5a brd ff:ff:ff:ff:ff:ff
        inet 10.203.43.168/23 brd 10.203.43.255 scope global eth0
        inet6 fe80::1031:3bff:fe09:285a/64 scope link
           valid_lft forever preferred_lft forever
    
    ubuntu@ip-10-203-43-168:~$ ip route
    default via 10.203.42.1 dev eth0  metric 100
    10.203.42.0/23 dev eth0  proto kernel  scope link  src 10.203.43.168
    


  4. HP Cloud
Example network config for HP Openstack cloud server

[root@server-1352331143-az-3-region-a-geo-1 ~]# ip a
1: lo:  LOOPBACK,UP,LOWER_UP  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  BROADCAST,MULTICAST,UP,LOWER_UP  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 02:16:3e:70:b7:9e brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.219/15 brd 10.3.255.255 scope global eth0
    inet6 fe80::16:3eff:aaaa:bbbb/64 scope link
       valid_lft forever preferred_lft forever
3: sit0:  NOARP  mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

[root@server-1352331143-az-3-region-a-geo-1 ~]# ip route
169.254.0.0/16 dev eth0  scope link
10.2.0.0/15 dev eth0  proto kernel  scope link  src 10.2.0.219
default via 10.2.0.1 dev eth0



References
    http://rtomaszewski.blogspot.co.uk/2012/08/links-to-cloud-provider-web-management.html http://aws.amazon.com/free/

Easy of use and learning curve for Rackspace or Amazon Cloud

Even though Rackspace and Amazon provide similar tools and products they both are built on different principles and have a different agenda and mission.

As expected this is going to shape the company image, product and services offerings. As there are countless metrics and ways to compare I would like to take a look at the user experience and evaluate a easy of use only.

You can learn and experience this yourself when setting up a simple test infrastructural build of Apache Web Server, MySql Database and Tomcat Applicatin Server. Alternatively we can learn from other and these 2 blog posts representing a very interesting summary:

Amazon and Rackspace: A Comparison, Part 2
Amazon and Rackspace: A Comparison, Part 1

A short summary:
Rackspace cloud is easier to use when interacting with cloud for the first time. The emphasis is on simplicity rather then elasticity and richness of features.

Amazon is much more mature and feature full. Unfortunately this leads to a longer learning curve and requires more knowledge when building your own infrastructure.

Friday, November 2, 2012

After server virtualization there is a time for network to be virtualized

Virtualization become today de facto a standard in almost every company. What about 10 years ago was a revolution in computing it has become a mature product for everyone. But when we look at the history how it was evolving we can see one component to remain unmodified: the network, one of the very few unvirtualized technology bastions.

But today the market is changing. To understand the changes and what all this means I recommend to read at least these 2 blogs:

VMWARE BUYS NICIRA: A HYPERVISOR VENDOR WOKE UP
VMware’s Acquisition of Nicira – VMware confirming the hypervisor is dead

The changes the blogs describe are already happening. As a practical example is a Rackspace Cloud Network product, a hybrid network that was created to leverage the potential in software defined network that data center provider can benefit from.


Further reading & references
  1. http://www.chriscolotti.us/vmware/nicira-nvp/nicira-nvp-virtualized-networking-primer/
  2. http://www.chriscolotti.us/vmware/nicira-nvp/the-nicira-nvp-component-architecture/
  3. http://nicira.com/en/frequently-asked-questions 
  4. http://www.rackspace.com/cloud/

Thursday, November 1, 2012

Code review process in Openstack uses Zuul

For a modern software development process to be effective as well as developer friendly it requires a usage of automation systems like:

  • Continuous integration
  • Build automation
  • Automated unit and integration testing
  • Rreview process
  • (distributed) Version control system
In Openstack the process has been implemented in the following way.


Further info about the components and how this works together you can watch this video as well[1].


References
  1. http://www.youtube.com/watch?v=e2H4dfJTx68&feature=em-subs_digest-newavtr
  2. https://github.com/openstack-ci/zuul/
  3. http://www.slideshare.net/lzyeval/assign-commit-and-review