The Systems Engineer organized chaos: November 2012

Friday, November 30, 2012

Introduction into tunneling protocols when deploying cloud network for your cloud infrastructure

Cloud network is a hot topic for cloud providers and hosting companies. In basic the concept should enable and allow tenants to create, manage and destroy network typologies on demand for the cloud servers by using cloud open API. That said, we want to allow a tenant to create an isolated virtual layer 2 network with its own IP subnet.

Before going into further details we have to realize that the problem isn't trivial to solve. The difficulty comes from a fact that the existing network that interconnects hypervizors hosts is not very flexible and adaptable for changes. That physical network architecture was build and tune to allow to handle all traffic from all cloud VM across you cloud deployment. This represent its strength and limitation as it is not flexible enough when it comes to configure and create many isolated virtual layer 2 or layer 3 networks per single tenant. This is exactly the problem that the cloud network promises to resolve.

At the moment there isn't a single standard how to implement a cloud network. Instead, we have 3 different protocols that were proposed: VXLAN, NVGRE, STT [1].

All these protocols relay on the fact that the hypervisor hosts are interconnected The implementation of the additional features is done by using tunneling mechanisms. All of them implement a L2 in L3 tunnels by using TCP, UDP or IP datagrams.

A short introduction and more explanation how this works can on the screenshots below that were taken from this video: Video: Cloud Tunnels @ Cloud Mafia ( slies can be found here http://ifup.org/slides/cloud-tunnels/ )

References

Friday, November 23, 2012

Installing Rackspace Private Cloud (Alamo) on a single virtual server using VMware Workstation

If you are looking for info about installation on a physical server you may like this article instead: How to install Rackspace Private Cloud (Alamo) on a single physical server

VMware Workstation

The Openstack installation within a virtual server is almost identical like on a physical one. The main difference is that we have to create a virtual machine that emulated/simulate the CPU extension needed for the nested hypervisor [1]

Practically that means that you have to enable this box in your VM config.

If you prefere editing of a config file this is the variable that you have to set in *.vmx file

 
.encoding = "windows-1252"
config.version = "8"
virtualHW.version = "8"
numvcpus = "4"
vcpu.hotadd = "TRUE"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "4096"
mem.hotadd = "TRUE"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "Ubuntu 64-bit.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "C:\Users\radoslaw\Downloads\alamo-v2.0.0.iso"
ide1:0.deviceType = "cdrom-image"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.autodetect = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.addressType = "static"
ethernet0.address = "00:50:56:25:71:FB"
usb.present = "TRUE"
ehci.present = "TRUE"
sound.present = "TRUE"
sound.fileName = "-1"
sound.autodetect = "TRUE"
serial0.present = "TRUE"
serial0.fileType = "thinprint"
serial1.present = "TRUE"
serial1.fileType = "file"
serial1.fileName = "test.py"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
usb.vbluetooth.startConnected = "TRUE"
displayName = "Ubuntu 64-bit"
guestOS = "ubuntu-64"
nvram = "Ubuntu 64-bit.nvram"
virtualHW.productCompatibility = "hosted"
vhv.enable = "TRUE"
powerType.powerOff = "hard"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "hard"
extendedConfigFile = "Ubuntu 64-bit.vmxf"
vmci0.id = "-1561829832"
uuid.location = "56 4d 22 7e ee b5 2b 19-3e c1 5e 76 a2 e8 5e 38"
uuid.bios = "56 4d 22 7e ee b5 2b 19-3e c1 5e 76 a2 e8 5e 38"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
usb.pciSlotNumber = "32"
ethernet0.pciSlotNumber = "33"
sound.pciSlotNumber = "34"
ehci.pciSlotNumber = "35"
vmci0.pciSlotNumber = "36"
usb:1.present = "TRUE"
vmotion.checkpointFBSize = "37748736"
usb:1.speed = "2"
usb:1.deviceType = "hub"
usb:1.port = "1"
usb:1.parent = "-1"
tools.remindInstall = "TRUE"
usb:0.present = "TRUE"
usb:0.deviceType = "hid"
usb:0.port = "0"
usb:0.parent = "-1"

References

Nested virtualization
http://www.ibm.com/developerworks/cloud/library/cl-nestedvirtualization/ http://www.veeam.com/blog/nesting-hyper-v-with-vmware-workstation-8-and-esxi-5.html

You can't use Oracle Virtual Box tool as it doesn't support nested virtualisation. https://www.virtualbox.org/ticket/4032

http://www.rackspace.com/knowledge_center/article/installing-rackspace-private-cloud-vmware-fusion

Thursday, November 22, 2012

How to install Rackspace Private Cloud (Alamo) on a single physical server

Rackspace has released a new version of its private cloud offerings. The new private cloud is based on the lasted Folsom Openstack release and includes many of the latest features.

ISO downloading

The software can be downloaded fro http://www.rackspace.com/cloud/private/openstack_software/ after you fill in a simple form. Once filled you receive an email with a direct link to download the ISO image (alamo-v2.0.0.iso). The other important link is Getting Started Guide document.

Supported features

In the guide we can find more details about supported and unsupported features.

Supported OpenStack Features:

Rackspace supports integration with the other components of OpenStack, as well as features such as floating IP address management, security groups, availability zones, and the python-novaclient command line client
Single and dual NIC configurations
NFS and ISCSI file storage as backing stores for VM storage
VNC Proxy
KVM hypervisor
Nova Multi Scheduler instead of Filter Scheduler
Keystone integrated authentication
Glance integrated image service
Horizon dashboard
Cinder block storage service
Swift object storage service
Linux and Windows guests to the extent to which they accept handoff from KVM and boot
Single metadata server running on each device
Cloud management through OpenStack APIs
Rackspace also supports the use of Rackspace Cloud Files as a backend for OpenStack
Image Storage. For information about OpenStack Object Storage, refer to Rackspace Private Cloud OpenStack Object Storage installation

Unsupported OpenStack Features. The following features are not supported yet.

Nova high availability
Nova object store
Nova volumes
NFS and ISCSI file storage via volumes for guest VMs
Clustered file system solutions
xpvnc
Xen and other hypervisors
Centralized metadata servers
Quantum Software Defined Networking

Installation

To start the installer I had to first convert the ISO into a bootable USB (In the guide you find a link to this tool that can help you with this task www.pendrivelinux.com)

Once done change the boot sequence in BIOS, plug your Pendrive and restart the server.

The installation is relatively simple. The couple of questions that you may get confused are about the number of NIC and what subnet do you want to use for the cloud servers.

Below are some of my screenshots to demonstrate how it looks like (sorry for the quality)

The installation welcome screen

There will be couple of questions for IP address, gateway, netmask, DNS, user, password that you have to answer as well. Random screenshots are showed below.

During the whole installation you can switch to console 4 (ALT+4) and observe various debugging logs as the installation progresses.

At some point the phase 1 of 2 will be completed and the system reboots. Once the system come back we can see different logs showing how the chef progress as Openstack is configured.

At the end of phase 2/2 when the installation is completed you are going to see a screen with summaries about the server resources.

References

Monday, November 19, 2012

Openstack popularity trends over time since 2010

It is a difficult task to measure a popularity of a product, item, thing or a software. But with a help of Internet we can use specialized search engines that track internet messages and post around the world for a particular string, key work or some for of textual identity.

An simple but powerful example of such an search engine is trend search on Google: http://www.google.com/trends .

Problem

How popular is Openstack.
Who uses Openstack.

Analysis

The below graphs and stats are generated by using this link: http://www.google.com/trends/explore#q=openstack.

Q: Is Openstack getting more and more popular?
Q: How popular is Openstack?

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)

Q:What is a country where Openstack is the most popular?
A: China

Q: Is the USA the most popular country for searches that include Openstack?
A: No

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)

Q: Is Europe interested in Openstack?
A: Yes

Q: In what European countries is Openstack the most popular?
A: In 2012: UK, Germany, France, Spain, Sweden and Poland

Q: In what cities is Openstack the most popular?

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)

Sunday, November 18, 2012

What does Software Defined Data Center means

After the industry created a Software Defined Network (SDN) [1] term it is time for a new one. A new emerging IT buzz word is a Software Defined Data Center (SDDC) [2]. It appears that only VMware is marketing this extensively at the moment.

From a technical point of view its all makes sense: compute resources are already largely virtualized, virtual storage and virtual networks are following. Looking at the last VMware acquisition of Nicira [3] the company has many if not all necessary products to build such a SDDC Data Center.

Let's see how the market will respond to it and if other vendors start looking and using this as well in the near future.

References

http://rtomaszewski.blogspot.co.uk/2012/10/software-defined-network-sdn-as-example.html

http://rtomaszewski.blogspot.co.uk/2012/10/google-does-use-sdn-in-its-data-centers.html

http://rtomaszewski.blogspot.co.uk/2012/09/emerging-of-virtual-network-aka-quantum.html

SDDC

http://www.networkcomputing.com/data-center/the-software-defined-data-center-dissect/240006848

http://www.vmware.com/solutions/datacenter/software-defined-datacenter/index.html

http://blogs.vmware.com/console/2012/08/the-software-defined-datacenter-meets-vmworld.html

Openstack, Nicira and VMware

http://www.infoworld.com/t/data-center/what-the-software-defined-data-center-really-means-199930

What your Dev, Engineering and sales teams can achieve by using Openstack cloud and Quantum network

Nicira NVP controller can be used together with Quantum as a plugin to enable extend virtual networking capabilities in Openstack. With NVP help the providers, operators and users can start building Software Defined Network (SDN) networks to interconnect the virtual machines.

It was predicted and by many it is seen as a natural evolution and a step forward for Openstack and cloud network in general.

We are no longer tight to static, unchangeable set of switches and routers when it comes to building and designing networks. The more flexible networking allow us to build more agile cloud infrastructures that will allow us to meet changing demands of rapidly evolving applications.

As a real business example of cloud flexibility this is a video of how Nicira is using Quantum to solve its internal infrastructure challenges for the Dev, Engineering and Sales teams: Running Quantum on Quantum @ Nicira's Multi-tenant OpenStack.

For these who don't like videos I have copied some of the key point from the presentation:

In a physical word it is difficult to achieve full automation and implement provisioning process that can build new configuration in a few minutes only
This is a challenge for the operational team to deliver results in a rapidly changing environment and keep the some quality of work when your company grows fast
There is always a risk of a human error that can potentially bring your production network down
Most network are designed to meet only the requirements that are known initially; It is difficult to plan ahead and new changes can be impossible to implement later on
Developers need to have a flexibility to test without a risk of bringing a company network down
You can easily provision new isolated environments for new projects or new hires
For better resource utilization you want to have an automation in place that allows you to tier down and spin up whole environments (cloud servers + networking) as easy as possible during a day
Not only a compute resources have to be easy to provision but as well as interconnecting and isolating them to achieve infrastructure resources agility; This is important especially for tier application typologies where we have web, db and application servers.
Quickly building POC for new projects and customers demonstration
Flexible network infrastructure allows collaboration between different development teams; breaking the network isolation boundaries in a statically built network may be impossible
Within minutes or second you can create and build new networks as well as to decommission them
You can add new resources and boost capacity when you need it; More flexible handling of cloud bursting; You don't have to oversubscribe resource only to meet the pick application requirements

And at the end copies of the interesting slides from the above presentation.

How to extract captions from a youtube video

There are videos on youtube that are 30min and longer. You've watched a video and later on you would like to find only a single moment that is interesting for you. With a help of subtitles and key word searching we could easily do it.

Problem

How to download and search in Google text captions file from a youtube video.

Solution

There are couple of solution that can be found on Google in the referece section. All the automatic ways didn't work for me. Below is a manual method that I used to extract the subtitles (caption) file from a Google video.

Open a video page in Chrome browser.
Enable debuging and reqeuest trucking by pressing F12 in Chrome.
Enable caption in the video.
Navigate into the Network tab and find the last timedtext request (the last at the bottom)
Right click on it and open that file in a new tab; An xml file containing subtitles with the imestamps should be opened.

References

Thursday, November 15, 2012

What network topologies can I build with Openstack Quantum server

The Folsom Openstack release brings many enhancements to networking stack in the Openstack version of the cloud. The features have been encapsulated with in a new core project called Quantum. But even looking at the documentation it can be very difficult at the beginning to understand what are the differences to old nova-network solution and how an example Quantum virtual network can be configured.

Fortunately there is an easy to follow video introduction to Quantum from last Summit [1]. The slides to it can be found here as well [2].

Below are example slides from the presentation of how a virtual network can be build and how it can look like in cloud powered by Openstack, examples: single flat network, multiple flat networks, mixed flat plus private networks, single provider network, multiple per tenant private networks plus single provider network.

References

OpenStack Networking (Quantum) Project Update
http://www.slideshare.net/danwent/quantum-grizzly-summit
More slides about Quantum from Dan Wendlandt at slideshare

Wednesday, November 14, 2012

How many Linux distributions provide officially Openstack support

In the same time as Openstack gets more mature the number of Python code line increases and the software become more complex. The usual tasks like installation, support and operational maintenance start to get more complicated and require more advance skills.

Seeing the opportunities on the market companies like Rackspace, Red Hat and Ubuntu rolling out new processes for sales and support to address these needs. As a prove of evidence below is couple of screenshots from Openstack Ubuntu presentation The promise of the Open Cloud with further links in references section.

References

http://www.rackspace.com/cloud/openstack/
http://www.redhat.com/about/news/press-archive/2012/8/red-hat-announces-preview-version-of-enterprise-ready-openstack-distribution
http://www.ubuntu.com/cloud/private-cloud/openstack
Videos from 2012 Openstack Summit in San Diego

Videos from 2012 Openstack Summit in San Diego

Here you can find videos for these of us who couldn't participate in the 2012 Openstack Summit in San Diego:

At Openstack home page: Openstack summit sessions/
At Openstack chanell on youtube: OpenStackFoundation

Tuesday, November 13, 2012

What happens to FTPS data channel if client closes control connection

There are couple of extensions added to standard FTP protocol to make it secure. It is important as in the default FTP configuration the control as well as the data channel use clear text to exchange commands or transmit data.

Problem

We assume that we were able to established a successful FTPS base session between a client and server. The client started a new data session to download a large file from the server or is uploading a file using the passive mode.

What happens to a file transfer if the control session is terminated by the client.

Troubleshooting

To verify the scenario we are going to setup a simple test scenario like in Does IPv4 based FTPS server supports EPSV FTP protocol extension blog [1].

As the curl client by default is not closing the control connections (what is a correct behavior that we will discuss at the end of this blog) we are going to use an active method to close an established tcp session described here How to forcibly kill an established TCP connection in Linux [2].

Test #1: client download a large file

Client logs

Logs when the control connection is being closed and reseted

root@clinet:~# netstat -tulpan | grep curl
tcp        0      0 5.79.21.166:45707       5.79.17.48:8000         ESTABLISHED 5546/curl
tcp    64210      0 5.79.21.166:43796       5.79.17.48:8011         ESTABLISHED 5546/curl

root@clinet:~# ./killcx.pl 5.79.17.48:8011
killcx v1.0.3 - (c)2009-2011 Jerome Bruandet - http://killcx.sourceforge.net/

[PARENT] checking connection with [5.79.17.48:8011]
[PARENT] found connection with [5.79.21.166:43796] (ESTABLISHED)
[PARENT] forking child
[CHILD]  interface not defined, will use [eth0]
[CHILD]  setting up filter to sniff ACK on [eth0] for 5 seconds
[CHILD]  hooked ACK from [5.79.21.166:43796]
[CHILD]  found AckNum [1229126485] and SeqNum [3095306962]
[CHILD]  sending spoofed RST to [5.79.21.166:43796] with SeqNum [1229126485]
[CHILD]  sending RST to remote host as well with SeqNum [3095306962]
[CHILD]  all done, sending USR1 signal to parent [5781] and exiting
[PARENT] received child signal, checking results...
         => success : connection has been closed !

These are the client logs from the start of downloading until the control session is closed.

root@client:~# curl -v --limit-rate 10K -o file.txt -u rado:pass -k --ftp-ssl ftp://5.79.17.48:8000/c2900-universalk9-mz.SPA.152-1.T.bin
* About to connect() to 5.79.17.48 port 8000 (#0)
*   Trying 5.79.17.48...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
< 220-FileZilla Server version 0.9.41 beta
< 220-written by Tim Kosse (Tim.Kosse@gmx.de)
< 220 Please visit http://sourceforge.net/projects/filezilla/
> AUTH SSL
< 234 Using authentication type SSL
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> USER rado
< 331 Password required for rado
> PASS pass
< 230 Logged on
> PBSZ 0
< 200 PBSZ=0
> PROT P
< 200 Protection level set to P
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||8011|)
*   Trying 5.79.17.48... connected
* Connecting to 5.79.17.48 (5.79.17.48) port 8011
> TYPE I
< 200 Type set to I
> SIZE c2900-universalk9-mz.SPA.152-1.T.bin
< 213 77200652
> RETR c2900-universalk9-mz.SPA.152-1.T.bin
< 150 Connection accepted
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL re-using session ID
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
* Maxdownload = -1
* Getting file with size: 77200652
{ [data not shown]
  0 73.6M    0  616k    0     0  10095      0  2:07:27  0:01:02  2:06:25  9753* SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
  0 73.6M    0  620k    0     0  10160      0  2:06:38  0:01:02  2:05:36 11170
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

Server logs

As the file download started this is logged on the server.

After the client control connections is terminated the server logs '426 Connection closed' tranfer aborted' log message.

After about 3-5 seconds after the connections clears from the server logs.

Test #2: client upload a large file

Client logs

The client logs when control channel is terminated.

root@client:~# netstat -tulpan | grep curl
tcp        0      0 5.79.21.166:43489       5.79.17.48:8016         ESTABLISHED 13177/curl
tcp        0      0 5.79.21.166:45717       5.79.17.48:8000         ESTABLISHED 13177/curl

root@client:~# ./killcx.pl  5.79.17.48:8016 
killcx v1.0.3 - (c)2009-2011 Jerome Bruandet - http://killcx.sourceforge.net/

[PARENT] checking connection with [5.79.17.48:8016]
[PARENT] found connection with [5.79.21.166:43489] (ESTABLISHED)
[PARENT] forking child
[CHILD]  interface not defined, will use [eth0]
[CHILD]  setting up filter to sniff ACK on [eth0] for 5 seconds
[PARENT] sending spoofed SYN to [5.79.21.166:43489] with bogus SeqNum
[CHILD]  hooked ACK from [5.79.21.166:43489]
[CHILD]  found AckNum [781536832] and SeqNum [2094006657]
[CHILD]  sending spoofed RST to [5.79.21.166:43489] with SeqNum [781536832]
[CHILD]  sending RST to remote host as well with SeqNum [2094006657]
[CHILD]  all done, sending USR1 signal to parent [13547] and exiting
[PARENT] received child signal, checking results...
         => success : connection has been closed !

Curl logs when the upload starts and the control channel is terminated.

root@client:~# curl -v --limit-rate 10K -T file.txt -u rado:pass -k --ftp-ssl ftp://5.79.17.48:8000/
* About to connect() to 5.79.17.48 port 8000 (#0)
*   Trying 5.79.17.48...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
< 220-FileZilla Server version 0.9.41 beta
< 220-written by Tim Kosse (Tim.Kosse@gmx.de)
< 220 Please visit http://sourceforge.net/projects/filezilla/
> AUTH SSL
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< 234 Using authentication type SSL
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> USER rado
< 331 Password required for rado
> PASS pass
< 230 Logged on
> PBSZ 0
< 200 PBSZ=0
> PROT P
< 200 Protection level set to P
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||8016|)
*   Trying 5.79.17.48... connected
* Connecting to 5.79.17.48 (5.79.17.48) port 8016
> TYPE I
< 200 Type set to I
> STOR file.txt
< 150 Connection accepted
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL re-using session ID
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
} [data not shown]
  0 73.6M    0     0    0  688k      0  10122  2:07:07  0:01:09  2:05:58  9814* SSL_write() returned SYSCALL, errno = 10422:51:35
  0 73.6M    0     0    0  688k      0  10122  2:07:07  0:01:09  2:05:58  8177
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
curl: (55) SSL_write() returned SYSCALL, errno = 104

Server logs

When the upload starts and 1-3 seconds after the control channel is closed.

Results discussion

We can see that every time the client closes the TCP session used to host the control channel bad things happen to the upload or download process.

This is expected behavior and is documented in the relevant RFC documents:

http://tools.ietf.org/html/rfc4217
7. Data Connection Behaviour

http://tools.ietf.org/html/rfc959
3.2. ESTABLISHING DATA CONNECTIONS

The server MUST close the data connection under the following conditions:

1. The server has completed sending data in a transfer mode
that requires a close to indicate EOF.

2. The server receives an ABORT command from the user.

3. The port specification is changed by a command from the
user.

4. The control connection is closed legally or otherwise.

5. An irrecoverable error condition occurs.

References

Monday, November 12, 2012

Does IPv4 based FTPS server supports EPSV FTP protocol extension

FTP Extension description

The EPSV stands for Extended Passive Mode and is defined in RFC 2428 [1].

According to the RFC specification it is used for:

This paper specifies extensions to FTP that will allow the protocol to work over IPv4 and IPv6.
...
The EPRT command allows for the specification of an extended address for the data connection.
...
The following are sample EPRT commands:
EPRT |1|132.235.1.2|6275|
EPRT |2|1080::8:800:200C:417A|5282|

In the RFC I couldn't find any word about default values or how the server should behave if the client doesn't provide any additional arguments and used the command in this simple way:

EPSV

Test configuration

To verify the ftp extension I build a simple test scenario using Rackspace cloud:

Windows 2008 cloud server running FTPS server; I used FileZilla Server [2]
Ubuntu 12.04 LTS Linux base system acting as a client; we used curl tool to simulate FTPS requests

Setting up of the clouds, ftps server and client are relatively simple so we are not going to describe these here. After FileZilla Server was installed I enabled the FTPS and a little bit customized the standard configuration. The screenshots below show the relevant settings.

Client connection

Below are client logs when we try to download a file from ftps server.

root@client:~# curl -v -o tmp -u user:pass -k --ftp-ssl ftp://<server_ip>:8000/file.txt
* About to connect() to 5.79.17.48 port 8000 (#0)
< 220-FileZilla Server version 0.9.41 beta
< 220-written by Tim Kosse (Tim.Kosse@gmx.de)
< 220 Please visit http://sourceforge.net/projects/filezilla/
> AUTH SSL
< 234 Using authentication type SSL
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> USER user
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< 331 Password required for user
> PASS pass
< 230 Logged on
> PBSZ 0
< 200 PBSZ=0
> PROT P
< 200 Protection level set to P
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||8007|)
*   Trying 5.79.17.48... connected
* Connecting to 5.79.17.48 (5.79.17.48) port 8007
> TYPE I
< 200 Type set to I
> SIZE file.txt
< 213 77200652
> RETR file.txt
< 150 Connection accepted
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL re-using session ID
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        start date: 2012-11-08 00:13:54 GMT
*        expire date: 2013-11-08 00:13:54 GMT
*        common name: www (does not match '5.79.17.48')
*        issuer: CN=www; C=11; ST=aaa; L=bbb; O=ddd; OU=aaa; emailAddress=a@a.com
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
* Maxdownload = -1
* Getting file with size: 77200652
{ [data not shown]

FileZilla server connection logs

As the client connects and start the session these are the logs we can observe on the serve.

Creating listen socket on port 8000...
Creating listen socket on port 990...
Server online
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> Connected, sending welcome message...
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 220-FileZilla Server version 0.9.41 beta
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 220-written by Tim Kosse (Tim.Kosse@gmx.de)
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 220 Please visit http://sourceforge.net/projects/filezilla/
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> AUTH SSL
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 234 Using authentication type SSL
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> SSL connection established
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> USER user
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> 331 Password required for user
(000022)11/12/2012 22:33:35 PM - (not logged in) (5.79.21.166)> PASS ********
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 230 Logged on
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> PBSZ 0
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 200 PBSZ=0
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> PROT P
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 200 Protection level set to P
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> PWD
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 257 "/" is current directory.
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> EPSV
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 229 Entering Extended Passive Mode (|||8007|)
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> TYPE I
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 200 Type set to I
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> SIZE c2900-universalk9-mz.SPA.152-1.T.bin
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 213 77200652
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> RETR c2900-universalk9-mz.SPA.152-1.T.bin
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> 150 Connection accepted
(000022)11/12/2012 22:33:35 PM - user (5.79.21.166)> SSL connection for data connection established
(000022)11/12/2012 22:34:33 PM - user (5.79.21.166)> 426 Connection closed; transfer aborted.

Summary

We can see that the EPSV extension can be used even on a server that has only IPv4 addresses. It is not a surprise as the RFC clearly defines that both protocols are supported (IPv6 and IPv4).

What is interesting is the server that once receives the EPSV command that is sent by the client using IPv4 it assumes this is the default protocol and defaults itself to IPv4 address.

References

Understanding network architecture in Openstack

With the new Folsom Openstack release we have now access to new networking architectures called Quantum. Even more, network is no longer only an internal part of nova but rather become a new core component [1].

It is going to take some time before Quantum become the main single option for network configuration. Many small to medium deployments will probably never require such a complicated mechanism to manage network creation, ports assignments or IPs allocation.

Problem

How many network configuration options do we have in Folsom.
How does the old nova-network work.

Solution

In Folsom we have a choice and we can use either nova-network [1] or quantum [3] service for our cloud infrastructure.

Depending on the complexity and flexibility we can use the the following configurations in nova-network [2]:

Configuring Flat Networking

Configuring Flat DHCP Networking

Configuring VLAN Networking

This picture shows an example interaction between 2 VMs. More in depth description of how the network flows look like and what role plays the hypervisor in routing or restricting the traffic can be found in [4].

References

http://www.mirantis.com/blog/openstack-networking-flatmanager-and-flatdhcpmanager/

http://www.mirantis.com/blog/openstack-networking-single-host-flatdhcpmanager/"

http://www.mirantis.com/blog/vlanmanager-network-flow-analysis/

http://www.mirantis.com/blog/openstack-networking-vlanmanager/

IBM OpenStack

Event flow when a cloud instance is provisioned in Openstack

Many people think that cloud is alike virtualizastion or other way around. Both technologies have a lot of things in common and with a sophisticated virtualization manager or orchestration tool (example can be a VMware vCenter or vCloud) we get a look and feel that cloud promise but in essence cloud is not the same like virtualization.

In this post instead to polemize who is right or not I would like to rather take a look at the events that are happening when we provision a cloud server, either in cloud or VMware. The inside view is only for the Openstack although.

Problem

What is happening in the Openstack cluster when a cloud server is provisioned and booted.

Solution

Before a VM can be marked as booted and fully provisioned there are number or messages and requests exchanged between the core openstack components[4].

This slides show event after event what is happening: Openstack cloud request flow [2].
This diagram [3] below is an additional visualization of interaction between the components.

References

Python Code level analyse
http://www.laurentluce.com/posts/openstack-nova-internals-of-instance-launching/
http://www.slideshare.net/mirantis/openstack-cloud-request-flow
Slides on Technical Architecture of Quantum (9/12) from the http://wiki.openstack.org/Quantum
http://www.openstack.org/software

Sunday, November 11, 2012

High level Openstack software architecture

This is an overwhelming task of tracking code changes for Openstack projects as well as to maintain an up to date view of current software architecture model. As the number of core Openstack projects increase it drives as well as the overall complexity and communication patters between the them.

Below is a nice diagram that through visualization helps to represent the complexities and simplify the communications view between all components in Folscom Openstack release.

Sources

New 2012 Gartner Magic Quadrant for Cloud Infrastructure as a Service (Iaas)

There is a new report from Gartner describing the IaaS main key player who drive the IaaS adoption and evolution as well as comparing the strengths and weakness they have.

The 2012 version of the document can be found here:

Magic Quadrant for Cloud Infrastructure as a Service
18 October 2012 ID:G00237002
Analyst(s): Lydia Leong, Douglas Toombs, Bob Gill, Gregor Petri, Tiny Haynes

With a little bit of history from the old Gartner's reports we can see as well as how the market has changed:

Friday, November 9, 2012

How to forcibly kill an established TCP connection in Linux

During one of the repros I needed to find a way to kill an already established TCP session but without killing the process that opened it.

Problem

How to kill an ESTABLISHED TCP connection in Linux.

Solution #1: passive mechanism

We can use a little tool called tcpkill that you can find desription and example usage here link

tcpkill -i eth0 { expression }

When started it first listens on a wire for any traffic matching your expression filter (that is compatible with tcpdump expressions). This is passive mechanism to learn about the TCP ACK and SEQ numbers. Once it knows the numbers it spoofs a TCP segments and try to reset the TCP session on both sides.

If the TCP session is idle or in generic there is no data being exchanged than this passive mechanism unfortunately is not going to work.

Solution #2: active mechanism

This killcx tool is taking a different approach and tries to generate traffic on the wire to discover what the ACK and SEQ numbers are. Once it forces the peer to replay to its spoofed traffic it kills the TCP session immediately after.

killcx.pl dest_ip:dest_port

Simple demonstration

tcpkill (passive)

On 2 different hosts we simulate a daemon that actively listens on incoming traffic and a client that initiate a connection to a server. Once a TCP session is created the client sent single byte. The commands show the outputs.

root@server:~# nc -v -l 7777                                                                                       
Connection from 164.177.146.87 port 7777 [tcp/*] accepted
a

root@client:~# nc -D 5.79.21.166 7777
a
b

Below are tcpdumps showing TCP handshaking and 1 byte exchange on both hosts.

root@server:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:15:59.049739 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [S], seq 2946797440, win 14600, options [mss 1460,sackOK,TS val 726987371 ecr 0,nop,wscale 2], length 0
23:15:59.049803 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [S.], seq 2303834803, ack 2946797441, win 14480, options [mss 1460,sackOK,TS val 350774705 ecr 726987371,nop,wscale 3], length 0
23:15:59.051394 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 726987375 ecr 350774705], length 0
23:16:00.815434 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 726987816 ecr 350774705], length 2
23:16:00.815493 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 3, win 1810, options [nop,nop,TS val 350775146 ecr 726987816], length 0

root@client:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:20:38.159839 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [S], seq 2946797440, win 14600, options [mss 1460,sackOK,TS val 726987371 ecr 0,nop,wscale 2], length 0
23:20:38.174837 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [S.], seq 2303834803, ack 2946797441, win 14480, options [mss 1460,sackOK,TS val 350774705 ecr 726987371,nop,wscale 3], length 0
23:20:38.174880 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 726987375 ecr 350774705], length 0
23:20:39.938895 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 726987816 ecr 350774705], length 2
23:20:39.939313 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 3, win 1810, options [nop,nop,TS val 350775146 ecr 726987816], length 0

At this point from another terminal on the server we can verify that the connection exists and start the tcpkill to try to kill it.

root@server:~# netstat -tulpan | grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      1338/nc
tcp        0      0 5.79.21.166:7777        164.177.146.87:50185    ESTABLISHED 1338/nc
root@server:~# tcpkill -i eth0 -9 port 50185

At this stage nothing is going to happen because neither the client or server sending any data. The tcpkill simply hangs and waits. The open TCP session continue to exists. But as soon as we sent another byte from the client to the server these is being logged on the tcpdump sessions for the tcpkill, client and server.

tcpkill

root@server:~# tcpkill -i eth0 -9 port 50185
tcpkill: listening on eth0 [port 50185]
164.177.146.87:50185 > 5.79.21.166:7777: R 2303834804:2303834804(0) win 0             
164.177.146.87:50185 > 5.79.21.166:7777: R 2303838454:2303838454(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303845754:2303845754(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303856704:2303856704(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303871304:2303871304(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303889554:2303889554(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303911454:2303911454(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303937004:2303937004(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303966204:2303966204(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946797445:2946797445(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946799255:2946799255(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946802875:2946802875(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946808305:2946808305(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946815545:2946815545(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946824595:2946824595(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946835455:2946835455(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946848125:2946848125(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946862605:2946862605(0) win 0

client

root@client:~# tcpdump -i any -nn port 7777
...
23:21:03.938535 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 3:5, ack 1, win 3650, options [nop,nop,TS val 726993816 ecr 350775146], length 2
23:21:03.943047 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 5, win 1810, options [nop,nop,TS val 350781146 ecr 726993816], length 0
23:21:03.943070 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303834804, win 0, length 0
23:21:03.943082 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303838454, win 0, length 0
23:21:03.943086 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303845754, win 0, length 0
23:21:03.943090 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303856704, win 0, length 0
23:21:03.943093 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303871304, win 0, length 0
23:21:03.943096 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303889554, win 0, length 0
23:21:03.943099 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303911454, win 0, length 0
23:21:03.943102 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303937004, win 0, length 0 
23:21:03.943106 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303966204, win 0, length 0

Server

root@server:~# tcpdump -i any -nn port 7777
...
23:16:24.815402 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 3:5, ack 1, win 3650, options [nop,nop,TS val 726993816 ecr 350775146], length 2
23:16:24.815449 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 5, win 1810, options [nop,nop,TS val 350781146 ecr 726993816], length 0
23:16:24.816040 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303834804, win 0, length 0
23:16:24.816080 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303838454, win 0, length 0
23:16:24.816103 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303845754, win 0, length 0
23:16:24.816130 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303856704, win 0, length 0
23:16:24.816153 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303871304, win 0, length 0
23:16:24.816177 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303889554, win 0, length 0
23:16:24.816199 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303911454, win 0, length 0
23:16:24.816225 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303937004, win 0, length 0
23:16:24.816245 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303966204, win 0, length 0
23:16:24.816280 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946797445, win 0, length 0
23:16:24.816314 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946799255, win 0, length 0
23:16:24.816339 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946802875, win 0, length 0
23:16:24.816362 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946808305, win 0, length 0
23:16:24.816384 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946815545, win 0, length 0
23:16:24.816407 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946824595, win 0, length 0
23:16:24.816429 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946835455, win 0, length 0
23:16:24.816451 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946848125, win 0, length 0
23:16:24.816474 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946862605, win 0, length 0

We can see that as soon as the new byte generate 2 TCP segment the tcpkill learns about the SEQ and ACK numbers and generate its RST packet to desynchronize and kill the TCP session.

killcx (active)

We start the 2 client and server sessions in a similar way. These are the command output showing what is happening on the wire this time.

server

root@server:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:46:06.228566 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 11257294, win 14600, options [mss 1460,sackOK,TS val 727439173 ecr 0,nop,wscale 2], length 0
23:46:06.228625 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [S.], seq 1319656020, ack 11257295, win 14480, options [mss 1460,sackOK,TS val 351226499 ecr 727439173,nop,wscale 3], length 0
23:46:06.233328 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 727439174 ecr 351226499], length 0
23:46:07.960259 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 727439605 ecr 351226499], length 2
23:46:07.960322 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351226932 ecr 727439605], length 0
# at this moment the TCP is estabilished and the server received one byte

# we see this as soon as the killcx is started
23:46:32.411167 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 10, win 65535, length 0
23:46:32.411194 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351233045 ecr 727439605,nop,nop,sack 1 {4283710012:4283710013}], length 0
23:46:32.413315 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [R], seq 11257297, win 65535, length 0
23:46:32.414945 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [R], seq 1319656021, win 65535, length 0

client

root@client:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:50:45.365610 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 11257294, win 14600, options [mss 1460,sackOK,TS val 727439173 ecr 0,nop,wscale 2], length 0
23:50:45.370015 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [S.], seq 1319656020, ack 11257295, win 14480, options [mss 1460,sackOK,TS val 351226499 ecr 727439173,nop,wscale 3], length 0
23:50:45.370059 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 727439174 ecr 351226499], length 0
23:50:47.097391 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 727439605 ecr 351226499], length 2
23:50:47.098888 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 35122693221:20:14439605], length 0
# at this moment the TCP is estabilished and the client sent one byte

# we see this as soon as the killcx is started
23:51:11.550122 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351233045 ecr 727439605,nop,nop,sack 1 {4283710012:4283710013}], length 0
23:51:11.552854 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [R], seq 1319656021, win 65535, length 0

killcx

root@server:~# netstat -tulpan | grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      10568/nc
tcp        0      0 5.79.21.166:7777        164.177.146.87:50189    ESTABLISHED 10568/nc

root@manage2:~# ./killcx.pl 164.177.146.87:50189
killcx v1.0.3 - (c)2009-2011 Jerome Bruandet - http://killcx.sourceforge.net/

[PARENT] checking connection with [164.177.146.87:50189]
[PARENT] found connection with [5.79.21.166:7777] (ESTABLISHED)
[PARENT] forking child
[CHILD]  interface not defined, will use [eth0]
[CHILD]  setting up filter to sniff ACK on [eth0] for 5 seconds
[PARENT] sending spoofed SYN to [5.79.21.166:7777] with bogus SeqNum
[CHILD]  hooked ACK from [5.79.21.166:7777]
[CHILD]  found AckNum [11257297] and SeqNum [1319656021]
[CHILD]  sending spoofed RST to [5.79.21.166:7777] with SeqNum [11257297]
[CHILD]  sending RST to remote host as well with SeqNum [1319656021]
[CHILD]  all done, sending USR1 signal to parent [10726] and exiting
[PARENT] received child signal, checking results...
         => success : connection has been closed !

As before the TCP session is established and idle waiting. As soon as we start the killcx it successfully spoofs a SYN packet on behalf of the client and sent it to the server. The server than replays with a valid TCP packet revealing the ACK and SEQ numbers. As soon as this this is on a wire the killcx sniffs this up and sent RST to kill the active session.

References

The Systems Engineer organized chaos

Search This Blog

Friday, November 30, 2012

Introduction into tunneling protocols when deploying cloud network for your cloud infrastructure

Friday, November 23, 2012

Installing Rackspace Private Cloud (Alamo) on a single virtual server using VMware Workstation

Thursday, November 22, 2012

How to install Rackspace Private Cloud (Alamo) on a single physical server

Monday, November 19, 2012

Openstack popularity trends over time since 2010

Sunday, November 18, 2012

What does Software Defined Data Center means

What your Dev, Engineering and sales teams can achieve by using Openstack cloud and Quantum network

How to extract captions from a youtube video

Thursday, November 15, 2012

What network topologies can I build with Openstack Quantum server

Wednesday, November 14, 2012

How many Linux distributions provide officially Openstack support

Videos from 2012 Openstack Summit in San Diego

Tuesday, November 13, 2012

What happens to FTPS data channel if client closes control connection

Monday, November 12, 2012

Does IPv4 based FTPS server supports EPSV FTP protocol extension

Understanding network architecture in Openstack

Event flow when a cloud instance is provisioned in Openstack

Sunday, November 11, 2012

High level Openstack software architecture

New 2012 Gartner Magic Quadrant for Cloud Infrastructure as a Service (Iaas)

Friday, November 9, 2012

How to forcibly kill an established TCP connection in Linux

Popular posts last week

Last posts

About Me

Linux enthusiast

My links

Friendly sites

Visitor locations

Total Pageviews

Blog Archive

Labels

Search This Blog

Friday, November 30, 2012

Friday, November 23, 2012

Thursday, November 22, 2012

Monday, November 19, 2012

Sunday, November 18, 2012

Thursday, November 15, 2012

Wednesday, November 14, 2012

Tuesday, November 13, 2012

Monday, November 12, 2012

Sunday, November 11, 2012

Friday, November 9, 2012

RSS/Atom feed

Popular posts last week

Last posts

About Me

Linux enthusiast

My links

Friendly sites

Visitor locations

Total Pageviews

Blog Archive

Labels