The ssh session to Rackspace Cloud Servers timeouts automaticaly and hangs

Problem description

After provisioning a Linux Rackspace Cloud Servers you can login to the server using ssh client.
If the ssh session is inactive for some time the underlying tcp session of the ssh connection will be automaticaly closed.

Impact

Often you work on the some server from multiple terminals or multiple ssh client sessions. When the ssh session timeouts you have to loggin again.

Depending on the ssh client you use the console output you had may be lost.

After logging again the bash history maybe lost.

Workaround

We can avoid the session to be terminated by trying to print something on the screen at a regular interval. The simple command below is going to manipulate the screen cursor position and prints a current data in the right bottom corner of the ssh session screen.

To execute it  for every new ssh session opened we have to cusomise the bash .profile config file.

cat >> .profile <<DONE
while true; do tput sc ; tput cup $(tput lines) $(tput cols); tput cub 8 ; echo -n $(date +%T); tput rc; sleep 10; done & 
DONE

References

man terminfo
How to: Change / Setup bash custom prompt (PS1)
Colours and Cursor Movement With tput
http://bashish.sourceforge.net/

TCP Source port change for the outgoing traffic from VM

When it comes to virtual  machines  (VM) management one of the big challenges to work with is the virtual network. When using VMware Workstation ( or Oracle VM VirtualBox) we can defined per VM how the added virtual network adapter should work.

There are multiple setting to choice for a single virtual network interface within VM:
 - Bridged
 - NAT
 - Host-only private network

Problem
Server received TCP network traffic has changed source port when comparing it to the original traffic that the VM sent out.

Impact
Applications that are port sensitive show wrong results (example: udptx )

Solution
By default a network adapter added to a VM uses NAT feature. It means that the outgoing traffic from the VM may be changed before it is going to be sent the next hope router in our real network. To overcome this behaviour we ave to change the network operational setting from NAT to Bridged.

If you lost the network connectivity after this change in your VM you have to additionally check and potentially change the default setting to what physical network device our default VMware bridge (VMnet0) is connected.

 - VMware workstation GUI -> Edit ->Virtual Network Editor : Vmnet0
 - Select the correct virtual network device (default is VMnet0)  from the list
 - Select a appropriate device from the device list under "Bridged to"

References
Workstation and Player Online Library
http://pubs.vmware.com/ws8/wwhelp/wwhimpl/js/html/wwhelp.htm#href=using_ws/GUID-BAFA66C3-81F0-4FCA-84C4-D9F7D258A60A.html

Workstation 8 doesn't work properly with bridge networking
http://communities.vmware.com/message/1843247

Hacking the UDP protocol - how to sent data using the UDP source port of UDP packets

Security has become one of the major concerns in current system architectures. The emerging new cloud technologies like Openstack ( IaaS cloud offering) try to address it in a better way in every new software release. When implementing and integrating bigger infrastructure solution we could say that the difficulty is proportional to the number of technologies that are involved to create a final solution.

One of the most important technology that works as a glue between the components is of course a network infrastructure and the supported protocol stack like TCP/IP that allows application to communicate. As our history of Internet proves a network layer has been since ever a fascination of many hackers. And as we know when it comes to programming and deciding what your program should do the sky is the limit.

As part of my network and security fascination i decided to conduct a small research project to confirm one of my ideas.

Research project description

How to transmit data using the UDP as transport protocols but without relaying on the UDP payload functionality.

Proposed solution description

The ideas is to use the available UDP headers to encode and to encapsulate the the data payload. In my example I decided to use the UDP source port value to store there my payload [1]

The UDP source port is a 16 bit length variable. That means that for a multi byte stream of data to transfer we have to implement effective way to divide it into chunks and sent them separately in many UDP datagrams. On the server site we have to de-encapsulate it and build up the original stream.

Examples and prove of concept (POC)
The client is a simple shell using the hping [2] to sent our specially designed UDP packets.

The server has been written in C to allow it an access to all functions of a network socket. The server has to implement the necessary logic to assemble the original data as well.

The example code can be found on github under the udptx project link [4]

Results presentation

For the below DemoA and DemoB illustrations we are going to use
 a) One Debian base Rackspace Cloud Server (RCS) [3].
 b) Locally hosted VM (in VMware Workstation) on my PC at home.

On any of the below mentioned machines we work on I assume you have already the necessary programs installed like hping2 (you can use the hping3 as well) and the compiler to compile the  server.c code.

This is only an example what you may need to install if you use Debian or Ubuntu base RCS.

debian-leny:~# aptitude install git-core #to check out our source code from github
debian-leny:~# aptitude install gcc
debian-leny:~# uname -a
debian-leny:~# aptitude install linux-headers-2.6.26-2-xen-amd64
debian-leny:~# aptitude install libc6-dev
debian-leny:~# aptitude install hping2

To check out the code and to compile the server code you can do:

debian-leny:~# git clone git://github.com/rtomaszewski/udptx.git
debian-leny:~# cd udptx
debian-leny:~/udptx# gcc -o server server.c

# after a successful compilation the code should look like this
debian-leny:~/udptx# ls -la server
-rwxr-xr-x 1 root root 11862 2012-01-05 22:15 server

debian-leny:~/udptx# ldd server
        linux-vdso.so.1 =>  (0x00007fff6bbff000)
        libc.so.6 => /lib/libc.so.6 (0x00007f95f19c3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f95f1d16000)

DemoA - using the loop back interface on the RCS

Open first SSH session to the RCS server and start the server.

ssh 31.222.171.250
debian-leny:~/udptx# cd udptx
debian-leny:~/udptx# ./server

If the server starts successfully the output will look like this.

debian-leny:~/udptx# ./server
created a UPD socket on port 5555
listening on our UPD socket

On the second SSH session on the RCS server let's run the client. The client requires 2 options: the ip of a host where the data should be sent and a string parameter representing the data we want to sent. In our example we use the loopback address and we try to sent a 3 character string "az;".

ssh  31.222.171.xyz
debian-leny:~/udptx# cd udptx

debian-leny:~/udptx# ./client.sh 127.0.0.1 'az;'
DEBUG: Output interface address: 219.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C BC A6 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 26 19 15 B3 00 08 C6 0F

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 191.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 11 B3 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 2F DD 15 B3 00 08 BC 4B

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 28.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C A9 27 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 17 41 15 B3 00 08 D4 E7

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 209.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 7F DE 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 00 35 15 B3 00 08 EB F3

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

If the data were sent successfully to the server we should see there the following output.

debian-leny:~/udptx# ./server
created a UPD socket on port 5555
listening on our UPD socket
received from ip: 127.0.0.1 port: hex 0x2619 dec 9753
received from ip: 127.0.0.1 port: hex 0x2fdd dec 12253
received from ip: 127.0.0.1 port: hex 0x1741 dec 5953
received from ip: 127.0.0.1 port: hex 0x35 dec 53
Received 3 characters:
msg[0]=char a hex 0x61 dec 97
msg[1]=char z hex 0x7a dec 122
msg[2]=char ; hex 0x3b dec 59

A tcpdump file with captured session between the client and sever can be found here udptx.lo.pcap.

debian-leny:~/udptx# tcpdump -s0 -nn -i lo -w /tmp/udptx.lo.pcap
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
4 packets captured
8 packets received by filter
0 packets dropped by kernel

DemoB - test over the Internet

The assumptions here are the same as before with the exception that the client is run from a local VM and it connects over the Internet to our Rackspace Cloud Server.

This time the output from the client is a little different than before.

root@ubuntuserver:~/udptx# uname  -a
Linux ubuntuserver 2.6.38-11-server #50-Ubuntu SMP Mon Sep 12 21:34:27 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntuserver:~/udptx# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:93:ee:ff brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.6/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::20c:29ff:fe93:eeff/64 scope link
       valid_lft forever preferred_lft forever

root@ubuntuserver:~/udptx# ls -la /usr/sbin/hping2
lrwxrwxrwx 1 root root 6 2012-01-05 23:28 /usr/sbin/hping2 -> hping3

root@ubuntuserver:~/udptx# tracepath  31.222.171.250

 1 :  192.168.0.6                                           0.207ms pmtu 1500
 1:  10.95.xyz.xyz                                           10.674ms
 1:  10.95.52.1                                           13.559ms
 2:  80.3.177.253                                         13.320ms
 3:  195.182.179.85                                       14.875ms asymm  5
 4:  212.43.163.117                                       13.964ms
 5:  212.43.162.218                                       12.821ms
 6:  213.105.159.34                                       18.455ms
 7:  62.253.184.2                                         26.880ms
 8:  195.66.224.116                                       15.280ms asymm  7
 9:  92.52.76.246                                         31.758ms asymm  8
10:  92.52.77.107                                         50.107ms asymm  9
11:  31.222.171.250                                       15.384ms reached
     Resume: pmtu 1500 hops 11 back 55

root@ubuntuserver:~/udptx# ./client.sh 31.222.171.250 'az;'
DEBUG: Output interface address: 0.0.0.0
DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 81 DF 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 26 19 15 B3 00 08 37 8B
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss

round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 0.0.0.0
DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 1F E2 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 2F DD 15 B3 00 08 2D C7 
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

DEBUG: Output interface address: 0.0.0.0
DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 9F 7F 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 17 41 15 B3 00 08 46 63 
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss

round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 0.0.0.0

DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 8C BC 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 00 35 15 B3 00 08 5D 6F 
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

The outputs from the server has changed as well.

debian-leny:~/udptx# ./server
created a UPD socket on port 5555
listening on our UPD socket
received from ip: 80.194.5.191 port: hex 0x2619 dec 9753
received from ip: 80.194.5.191 port: hex 0x2fdd dec 12253
received from ip: 80.194.5.191 port: hex 0x1741 dec 5953
received from ip: 80.194.5.191 port: hex 0x35 dec 53
Received 3 characters:
msg[0]=char a hex 0x61 dec 97
msg[1]=char z hex 0x7a dec 122
msg[2]=char ; hex 0x3b dec 59

A new tcpdump file with the captured session between the client and sever can be found here udptx.eth.pcap

debian-leny:~/udptx# tcpdump -s0 -nn -i eth0 -w /tmp/udptx.eth.pcap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19 packets captured
20 packets received by filter
0 packets dropped by kernel

Summary and further work

The presented code shows that it is possible to sent and receive data without relaying on the UDP payload mechanism. Of course the code is very simple and doesn't implement all the necessary protocol features to make it useful on a bigger scale. To name only a few missing key points that would have to be implemented:
 - packet lost
 - packet duplication
 - packets reordering

To better understand the limitation of our code and learn more about protocol implementation issues it is highly recommend to read at least the official RFC documents about for UDP and TCP protocols [5].

References

[1] http://www.networksorcery.com/enp/protocol/udp.htm
[2] http://www.hping.org/

[3]
http://www.rackspace.com/cloud/cloud_hosting_products/servers/
http://www.rackspace.com/cloud/cloud_hosting_faq/
http://www.rackspace.com/knowledge_center/node/1875

[4]  https://github.com/rtomaszewski/udptx
[5] http://www.diigo.com/user/rtomaszewski/rfc

One line Bash script debugging

Working on the Bash shell can be very effective. You can combine various command line programs and chain (pipe) them together to accomplish a bigger task. Sometimes you have to debug your one line scripts although.

When working on the CLI I wrote in a hurry a small command to find and check the value of the sched_autogroup_enabled Linux  kernel variable [1] under the proc file system.

To my first surprise it didn't work at all.

root@udesktop:/proc# find . -name \*sched\* 2>/dev/null  | grep -v [0-9]
root@udesktop:/proc# 

It is easy to find this file manualy and I did it. Below is the prove that the file exist that I was looking for.

root@udesktop:/proc# ls -la ./sys/kernel/sched_autogroup_enabled
-rw-r--r-- 1 root root 0 2012-01-01 21:38 ./sys/kernel/sched_autogroup_enabled

Problem
How to debug one line bash scripts. Or in general how to debug any bash script.

Solution
The problem is easy to see once we enable more verbose debug output from the Bash

root@udesktop:/proc# set -v -x
root@udesktop:/proc# find . -name \*sched\* 2>/dev/null  | grep -v [0-9]
find . -name \*sched\* 2>/dev/null  | grep -v [0-9]
+ find . -name '*sched*'
+ grep --color=auto -v 1 2 3 5 6 7 8 9

We see that the string '[0-9]' is extended by the bash before the command chain is actually executed.

Once we know that the problem is how our regular expression [2] is evaluated the fix is simple:

root@udesktop:/proc# find . -name \*sched\* 2>/dev/null  | grep -v '[0-9]'
find . -name \*sched\* 2>/dev/null  | grep -v '[0-9]'
+ find . -name '*sched*'
+ grep --color=auto -v '[0-9]'
./schedstat
./sched_debug
./sys/kernel/sched_child_runs_first
./sys/kernel/sched_min_granularity_ns
./sys/kernel/sched_latency_ns
./sys/kernel/sched_wakeup_granularity_ns
./sys/kernel/sched_tunable_scaling
./sys/kernel/sched_migration_cost
./sys/kernel/sched_nr_migrate
./sys/kernel/sched_time_avg
./sys/kernel/sched_shares_window
./sys/kernel/sched_rt_period_us
./sys/kernel/sched_rt_runtime_us
./sys/kernel/sched_compat_yield
./sys/kernel/sched_autogroup_enabled
./sys/kernel/sched_domain

References
[1]
Benefiting of sched_autogroup_enabled on the desktop
http://unix.stackexchange.com/questions/9069/benefiting-of-sched-autogroup-enabled-on-the-desktop

The ~200 Line Linux Kernel Patch That Does Wonders
http://www.phoronix.com/scan.php?page=article&item=linux_2637_video&num=1

[2]
Bash Reference Manual
http://www.gnu.org/software/bash/manual/bashref.html#Filename-Expansion

Debugging Bash scripts
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html

How effectively use Linux system tools to find out the default Linux kernel I/O scheduler

In many troubleshooting situation we overview and check large numbers of logs generated by one or another application. Kernel is nothing special here. The kernel log files can  usually be found under /var/log/kern.log.

Problem
How to find out and confirm all names of the used io kernel scheduler in the last several reboots.

Solution

$ zcat kern.log.3.gz  | egrep 'io scheduler.*\(default\)' 
Oct  8 22:20:16 udesktop kernel: [    0.888090] io scheduler cfq registered (default)
Oct 10 20:30:33 udesktop kernel: [    0.688367] io scheduler cfq registered (default)
Nov  3 23:31:05 udesktop kernel: [    0.872053] io scheduler cfq registered (default)

$ zcat kern.log.1.gz  | egrep 'io scheduler.*\(default\)' | sed -r 's/ +/ /g' | cut -d' ' -f10 | sort | uniq
cfg

Explanation
We first grep for the names of the configured io scheduler. Next we pars it trought sed to repalce all multi space string with a single space and at the end we cut of the scheduler name.

References
Choosing an I/O Scheduler for Red Hat® Enterprise Linux® 4 and the 2.6 Kernel
http://www.redhat.com/magazine/008jun05/features/schedulers/

How to connect over ssh to a server and automatically provide the user name and password using putty or ssh

With the benefit of easy of use of cloud services we can provision a new cloud base virtual server within a minute. Having the server ready we still have to login there to do our job or tests.

As an example you can take a look at the Cloud Servers from Rackspace [1].

By using the free windows ssh implementation like putty [2] or standard linux openssh ssh client we can easily login over a secure connection into our server by using the SSH protocol.

Problem
When you want to quickly and temporarily provision a cloud server it is time wasting when you have to provide the user name and password each time to login to do the work.
As the cloud server will be used for a limited (short) time we would like to be able to login with one 'click' or one command only.

Solution
The Multi-Tabbed PuTTY tool allow us to use a wrapper around the putty tool and provide an easy to use solution to open a new ssh session on demand without having to wory about he user name or password.

Multi-Tabbed PuTTY
http://www.ttyplus.com/

For the Linux system we can use an simple script using the 'expect' tool to start the ssh client.

SSH - Passing Unix login passwords through shell scripts
http://nixcraft.com/shell-scripting/4489-ssh-passing-unix-login-passwords-through-shell-scripts.html

Security
From the security point of view storing the passwords is always a bad idea. There are always some alternative methods. Some examples listed below:

Password-less logins with OpenSSH
http://www.debian-administration.org/articles/152
http://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/

References
[1]
http://www.rackspace.com/cloud/cloud_hosting_products/servers/

[2]
http://www.chiark.greenend.org.uk/~sgtatham/putty/

How to resize (increase) main NTFS system partition of your Windows base virtual mashine in VMware Workstation

Problem summary
When we create a VM we specify the various hardware component we want to virtualize. One of them is the HDD.  After some time you may find out that the hard drive your VM has is too small.

The VMware Workstation allows you to resize the disk (as long there are not snapshots).

Impact
You can't install any additional software because there is not enough free space.


Problem
Although there are many tools you can use to resize the NTFS file system many of them have some limitation (demo version doesn't write data to the disk) or require a comercial license to bought or can be use on the main partition (diskpart.exe) [2]



Solution
The task can be done with the help of [1]. We don't need license. It can resize the main NTFS partition.

Free EaseUS® Partition Master 9.1 Home Edition 

References
[1]
Free EaseUS® Partition Master 9.1 Home Edition
http://www.partition-tool.com/personal.htm

[2]

How to extend a data volume in Windows Server 2003, in Windows XP, in Windows 2000, and in Windows Server 2008
http://support.microsoft.com/kb/325590