The Systems Engineer organized chaos

TCP Source port change for the outgoing traffic from VM

2012-01-08T22:36:00.002Z

When it comes to virtual machines (VM) management one of the big challenges to work with is the virtual network. When using VMware Workstation ( or Oracle VM VirtualBox) we can defined per VM how the added virtual network adapter should work.

There are multiple setting to choice for a single virtual network interface within VM:
- Bridged
- NAT
- Host-only private network

Problem
Server received TCP network traffic has changed source port when comparing it to the original traffic that the VM sent out.

Impact
Applications that are port sensitive show wrong results (example: udptx )

Solution
By default a network adapter added to a VM uses NAT feature. It means that the outgoing traffic from the VM may be changed before it is going to be sent the next hope router in our real network. To overcome this behaviour we ave to change the network operational setting from NAT to Bridged.

If you lost the network connectivity after this change in your VM you have to additionally check and potentially change the default setting to what physical network device our default VMware bridge (VMnet0) is connected.

- VMware workstation GUI -> Edit ->Virtual Network Editor : Vmnet0
- Select the correct virtual network device (default is VMnet0) from the list
- Select a appropriate device from the device list under "Bridged to"

References
Workstation and Player Online Library
http://pubs.vmware.com/ws8/wwhelp/wwhimpl/js/html/wwhelp.htm#href=using_ws/GUID-BAFA66C3-81F0-4FCA-84C4-D9F7D258A60A.html

Workstation 8 doesn't work properly with bridge networking
http://communities.vmware.com/message/1843247

Hacking the UDP protocol - how to sent data using the UDP source port of UDP packets

2012-01-07T18:26:00.001Z

Security has become one of the major concerns in current system architectures. The emerging new cloud technologies like Openstack ( IaaS cloud offering) try to address it in a better way in every new software release. When implementing and integrating bigger infrastructure solution we could say that the difficulty is proportional to the number of technologies that are involved to create a final solution.

One of the most important technology that works as a glue between the components is of course a network infrastructure and the supported protocol stack like TCP/IP that allows application to communicate. As our history of Internet proves a network layer has been since ever a fascination of many hackers. And as we know when it comes to programming and deciding what your program should do the sky is the limit.

As part of my network and security fascination i decided to conduct a small research project to confirm one of my ideas.

Research project description

How to transmit data using the UDP as transport protocols but without relaying on the UDP payload functionality.

Proposed solution description

The ideas is to use the available UDP headers to encode and to encapsulate the the data payload. In my example I decided to use the UDP source port value to store there my payload [1]

The UDP source port is a 16 bit length variable. That means that for a multi byte stream of data to transfer we have to implement effective way to divide it into chunks and sent them separately in many UDP datagrams. On the server site we have to de-encapsulate it and build up the original stream.

Examples and prove of concept (POC)
The client is a simple shell using the hping [2] to sent our specially designed UDP packets.

The server has been written in C to allow it an access to all functions of a network socket. The server has to implement the necessary logic to assemble the original data as well.

The example code can be found on github under the udptx project link [4]

Results presentation

For the below DemoA and DemoB illustrations we are going to use
a) One Debian base Rackspace Cloud Server (RCS) [3].
b) Locally hosted VM (in VMware Workstation) on my PC at home.

On any of the below mentioned machines we work on I assume you have already the necessary programs installed like hping2 (you can use the hping3 as well) and the compiler to compile the server.c code.

This is only an example what you may need to install if you use Debian or Ubuntu base RCS.

debian-leny:~# aptitude install git-core #to check out our source code from github
debian-leny:~# aptitude install gcc
debian-leny:~# uname -a
debian-leny:~# aptitude install linux-headers-2.6.26-2-xen-amd64
debian-leny:~# aptitude install libc6-dev
debian-leny:~# aptitude install hping2

To check out the code and to compile the server code you can do:

debian-leny:~# git clone git://github.com/rtomaszewski/udptx.git
debian-leny:~# cd udptx
debian-leny:~/udptx# gcc -o server server.c

# after a successful compilation the code should look like this
debian-leny:~/udptx# ls -la server
-rwxr-xr-x 1 root root 11862 2012-01-05 22:15 server

debian-leny:~/udptx# ldd server
        linux-vdso.so.1 =>  (0x00007fff6bbff000)
        libc.so.6 => /lib/libc.so.6 (0x00007f95f19c3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f95f1d16000)

DemoA - using the loop back interface on the RCS

Open first SSH session to the RCS server and start the server.

ssh 31.222.171.250
debian-leny:~/udptx# cd udptx
debian-leny:~/udptx# ./server

If the server starts successfully the output will look like this.

debian-leny:~/udptx# ./server
created a UPD socket on port 5555
listening on our UPD socket

On the second SSH session on the RCS server let's run the client. The client requires 2 options: the ip of a host where the data should be sent and a string parameter representing the data we want to sent. In our example we use the loopback address and we try to sent a 3 character string "az;".

ssh  31.222.171.xyz
debian-leny:~/udptx# cd udptx

debian-leny:~/udptx# ./client.sh 127.0.0.1 'az;'
DEBUG: Output interface address: 219.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C BC A6 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 26 19 15 B3 00 08 C6 0F

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 191.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 11 B3 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 2F DD 15 B3 00 08 BC 4B

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 28.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C A9 27 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 17 41 15 B3 00 08 D4 E7

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 209.127.0.0
DEBUG: if lo: OK
using lo, addr: 127.0.0.1, MTU: 16436
DEBUG: Trying to open PF_PACKET socket... DEBUG: PF_PACKET, SOCK_RAW open OK
HPING 127.0.0.1 (lo 127.0.0.1): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 7F DE 00 00 40 11 00 00 7F 00 00 01 7F 00 00 01 00 35 15 B3 00 08 EB F3

--- 127.0.0.1 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

If the data were sent successfully to the server we should see there the following output.

debian-leny:~/udptx# ./server
created a UPD socket on port 5555
listening on our UPD socket
received from ip: 127.0.0.1 port: hex 0x2619 dec 9753
received from ip: 127.0.0.1 port: hex 0x2fdd dec 12253
received from ip: 127.0.0.1 port: hex 0x1741 dec 5953
received from ip: 127.0.0.1 port: hex 0x35 dec 53
Received 3 characters:
msg[0]=char a hex 0x61 dec 97
msg[1]=char z hex 0x7a dec 122
msg[2]=char ; hex 0x3b dec 59

A tcpdump file with captured session between the client and sever can be found here udptx.lo.pcap.

debian-leny:~/udptx# tcpdump -s0 -nn -i lo -w /tmp/udptx.lo.pcap
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
4 packets captured
8 packets received by filter
0 packets dropped by kernel

DemoB - test over the Internet

The assumptions here are the same as before with the exception that the client is run from a local VM and it connects over the Internet to our Rackspace Cloud Server.

This time the output from the client is a little different than before.

root@ubuntuserver:~/udptx# uname  -a
Linux ubuntuserver 2.6.38-11-server #50-Ubuntu SMP Mon Sep 12 21:34:27 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntuserver:~/udptx# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:93:ee:ff brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.6/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::20c:29ff:fe93:eeff/64 scope link
       valid_lft forever preferred_lft forever

root@ubuntuserver:~/udptx# ls -la /usr/sbin/hping2
lrwxrwxrwx 1 root root 6 2012-01-05 23:28 /usr/sbin/hping2 -> hping3

root@ubuntuserver:~/udptx# tracepath  31.222.171.250

 1 :  192.168.0.6                                           0.207ms pmtu 1500
 1:  10.95.xyz.xyz                                           10.674ms
 1:  10.95.52.1                                           13.559ms
 2:  80.3.177.253                                         13.320ms
 3:  195.182.179.85                                       14.875ms asymm  5
 4:  212.43.163.117                                       13.964ms
 5:  212.43.162.218                                       12.821ms
 6:  213.105.159.34                                       18.455ms
 7:  62.253.184.2                                         26.880ms
 8:  195.66.224.116                                       15.280ms asymm  7
 9:  92.52.76.246                                         31.758ms asymm  8
10:  92.52.77.107                                         50.107ms asymm  9
11:  31.222.171.250                                       15.384ms reached
     Resume: pmtu 1500 hops 11 back 55

root@ubuntuserver:~/udptx# ./client.sh 31.222.171.250 'az;'
DEBUG: Output interface address: 0.0.0.0
DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 81 DF 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 26 19 15 B3 00 08 37 8B
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss

round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 0.0.0.0
DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 1F E2 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 2F DD 15 B3 00 08 2D C7 
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

DEBUG: Output interface address: 0.0.0.0
DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 9F 7F 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 17 41 15 B3 00 08 46 63 
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss

round-trip min/avg/max = 0.0/0.0/0.0 ms
DEBUG: Output interface address: 0.0.0.0

DEBUG: if lo: The address doesn't match
DEBUG: if eth0: OK
using eth0, addr: 192.168.0.6, MTU: 1500
DEBUG: pcap_open_live(eth0, 99999, 0, 1, 0x628220)
DEBUG: dltype is 1
HPING 31.222.171.250 (eth0 31.222.171.250): udp mode set, 28 headers + 0 data bytes
45 00 00 1C 8C BC 00 00 40 11 00 00 C0 A8 00 06 1F DE AB FA 00 35 15 B3 00 08 5D 6F 
DEBUG: under pcap_recv()
DEBUG: under pcap_recv()
DEBUG: [pcap_recv] p = NULL

--- 31.222.171.250 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

The outputs from the server has changed as well.

debian-leny:~/udptx# ./server
created a UPD socket on port 5555
listening on our UPD socket
received from ip: 80.194.5.191 port: hex 0x2619 dec 9753
received from ip: 80.194.5.191 port: hex 0x2fdd dec 12253
received from ip: 80.194.5.191 port: hex 0x1741 dec 5953
received from ip: 80.194.5.191 port: hex 0x35 dec 53
Received 3 characters:
msg[0]=char a hex 0x61 dec 97
msg[1]=char z hex 0x7a dec 122
msg[2]=char ; hex 0x3b dec 59

A new tcpdump file with the captured session between the client and sever can be found here udptx.eth.pcap

debian-leny:~/udptx# tcpdump -s0 -nn -i eth0 -w /tmp/udptx.eth.pcap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19 packets captured
20 packets received by filter
0 packets dropped by kernel

Summary and further work

The presented code shows that it is possible to sent and receive data without relaying on the UDP payload mechanism. Of course the code is very simple and doesn't implement all the necessary protocol features to make it useful on a bigger scale. To name only a few missing key points that would have to be implemented:
- packet lost
- packet duplication
- packets reordering

To better understand the limitation of our code and learn more about protocol implementation issues it is highly recommend to read at least the official RFC documents about for UDP and TCP protocols [5].

References

[1] http://www.networksorcery.com/enp/protocol/udp.htm
[2] http://www.hping.org/

[3]
http://www.rackspace.com/cloud/cloud_hosting_products/servers/
http://www.rackspace.com/cloud/cloud_hosting_faq/
http://www.rackspace.com/knowledge_center/node/1875

[4] https://github.com/rtomaszewski/udptx
[5] http://www.diigo.com/user/rtomaszewski/rfc

One line Bash script debugging

2012-01-01T22:42:00.002Z

Working on the Bash shell can be very effective. You can combine various command line programs and chain (pipe) them together to accomplish a bigger task. Sometimes you have to debug your one line scripts although.

When working on the CLI I wrote in a hurry a small command to find and check the value of the sched_autogroup_enabled Linux kernel variable [1] under the proc file system.

To my first surprise it didn't work at all.

root@udesktop:/proc# find . -name \*sched\* 2>/dev/null  | grep -v [0-9]
root@udesktop:/proc#

It is easy to find this file manualy and I did it. Below is the prove that the file exist that I was looking for.

root@udesktop:/proc# ls -la ./sys/kernel/sched_autogroup_enabled
-rw-r--r-- 1 root root 0 2012-01-01 21:38 ./sys/kernel/sched_autogroup_enabled

Problem
How to debug one line bash scripts. Or in general how to debug any bash script.

Solution
The problem is easy to see once we enable more verbose debug output from the Bash

root@udesktop:/proc# set -v -x
root@udesktop:/proc# find . -name \*sched\* 2>/dev/null  | grep -v [0-9]
find . -name \*sched\* 2>/dev/null  | grep -v [0-9]
+ find . -name '*sched*'
+ grep --color=auto -v 1 2 3 5 6 7 8 9

We see that the string '[0-9]' is extended by the bash before the command chain is actually executed.

Once we know that the problem is how our regular expression [2] is evaluated the fix is simple:

root@udesktop:/proc# find . -name \*sched\* 2>/dev/null  | grep -v '[0-9]'
find . -name \*sched\* 2>/dev/null  | grep -v '[0-9]'
+ find . -name '*sched*'
+ grep --color=auto -v '[0-9]'
./schedstat
./sched_debug
./sys/kernel/sched_child_runs_first
./sys/kernel/sched_min_granularity_ns
./sys/kernel/sched_latency_ns
./sys/kernel/sched_wakeup_granularity_ns
./sys/kernel/sched_tunable_scaling
./sys/kernel/sched_migration_cost
./sys/kernel/sched_nr_migrate
./sys/kernel/sched_time_avg
./sys/kernel/sched_shares_window
./sys/kernel/sched_rt_period_us
./sys/kernel/sched_rt_runtime_us
./sys/kernel/sched_compat_yield
./sys/kernel/sched_autogroup_enabled
./sys/kernel/sched_domain

References
[1]
Benefiting of sched_autogroup_enabled on the desktop
http://unix.stackexchange.com/questions/9069/benefiting-of-sched-autogroup-enabled-on-the-desktop

The ~200 Line Linux Kernel Patch That Does Wonders
http://www.phoronix.com/scan.php?page=article&item=linux_2637_video&num=1

[2]
Bash Reference Manual
http://www.gnu.org/software/bash/manual/bashref.html#Filename-Expansion

Debugging Bash scripts
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html

How effectively use Linux system tools to find out the default Linux kernel I/O scheduler

2011-12-30T21:42:00.001Z

In many troubleshooting situation we overview and check large numbers of logs generated by one or another application. Kernel is nothing special here. The kernel log files can usually be found under /var/log/kern.log.

Problem
How to find out and confirm all names of the used io kernel scheduler in the last several reboots.

Solution

$ zcat kern.log.3.gz  | egrep 'io scheduler.*\(default\)' 
Oct  8 22:20:16 udesktop kernel: [    0.888090] io scheduler cfq registered (default)
Oct 10 20:30:33 udesktop kernel: [    0.688367] io scheduler cfq registered (default)
Nov  3 23:31:05 udesktop kernel: [    0.872053] io scheduler cfq registered (default)

$ zcat kern.log.1.gz  | egrep 'io scheduler.*\(default\)' | sed -r 's/ +/ /g' | cut -d' ' -f10 | sort | uniq
cfg

Explanation
We first grep for the names of the configured io scheduler. Next we pars it trought sed to repalce all multi space string with a single space and at the end we cut of the scheduler name.

References
Choosing an I/O Scheduler for Red Hat® Enterprise Linux® 4 and the 2.6 Kernel
http://www.redhat.com/magazine/008jun05/features/schedulers/

How to connect over ssh to a server and automatically provide the user name and password using putty or ssh

2011-12-30T00:34:00.004Z

With the benefit of easy of use of cloud services we can provision a new cloud base virtual server within a minute. Having the server ready we still have to login there to do our job or tests.

As an example you can take a look at the Cloud Servers from Rackspace [1].

By using the free windows ssh implementation like putty [2] or standard linux openssh ssh client we can easily login over a secure connection into our server by using the SSH protocol.

Problem
When you want to quickly and temporarily provision a cloud server it is time wasting when you have to provide the user name and password each time to login to do the work.
As the cloud server will be used for a limited (short) time we would like to be able to login with one 'click' or one command only.

Solution
The Multi-Tabbed PuTTY tool allow us to use a wrapper around the putty tool and provide an easy to use solution to open a new ssh session on demand without having to wory about he user name or password.

Multi-Tabbed PuTTY
http://www.ttyplus.com/

For the Linux system we can use an simple script using the 'expect' tool to start the ssh client.

SSH - Passing Unix login passwords through shell scripts
http://nixcraft.com/shell-scripting/4489-ssh-passing-unix-login-passwords-through-shell-scripts.html

Security
From the security point of view storing the passwords is always a bad idea. There are always some alternative methods. Some examples listed below:

Password-less logins with OpenSSH
http://www.debian-administration.org/articles/152
http://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/

References
[1]
http://www.rackspace.com/cloud/cloud_hosting_products/servers/

[2]
http://www.chiark.greenend.org.uk/~sgtatham/putty/

How to resize (increase) main NTFS system partition of your Windows base virtual mashine in VMware Workstation

2011-12-29T00:13:00.002Z

Problem summary
When we create a VM we specify the various hardware component we want to virtualize. One of them is the HDD. After some time you may find out that the hard drive your VM has is too small.

The VMware Workstation allows you to resize the disk (as long there are not snapshots).

Impact
You can't install any additional software because there is not enough free space.

Problem
Although there are many tools you can use to resize the NTFS file system many of them have some limitation (demo version doesn't write data to the disk) or require a comercial license to bought or can be use on the main partition (diskpart.exe) [2]

Solution
The task can be done with the help of [1]. We don't need license. It can resize the main NTFS partition.

Free EaseUS® Partition Master 9.1 Home Edition

References
[1]
Free EaseUS® Partition Master 9.1 Home Edition
http://www.partition-tool.com/personal.htm

[2]

How to extend a data volume in Windows Server 2003, in Windows XP, in Windows 2000, and in Windows Server 2008
http://support.microsoft.com/kb/325590

XenServer v6 installation problem: UNSUPPORTED_INSTALL_METHOD - other-config:install-repository was not set to an appropriate value, and this is required for the selected distribution type

2011-10-10T19:46:00.000+01:00

It has been a long time since I played with the XenServer for the last time.

I managed to install the XenServer without any problems. The XenConsole installation went flawlessly as well. But when I tried to create a new VS using the provided templates i run into the desribing problem:

What I did in XenCentre:

XenCentre
New VM
Ubuntu Lucid Lynx 10.04(32 bit)
Then use the default settings and once you see the screen with the:

"Select the installation method for the operating system software you want to install on the new VM"

You seen on the screen a page where you should select the desired installation source. In my example the option for "Install from ISO library or DVD drive" was grey but you can still select one of the ISO images from your storage pool (if you defined it previously).

Problem:
If you attempt to run the VS to start the installation you will get an error like this:

UNSUPPORTED_INSTALL_METHOD - other-config:install-repository was not set to an appropriate value, and this is required for the selected distribution type

There is not much on the Citrix support/technical knowledge page for this. The solution is here although.

Solution:

To create VM and start the installation from a ISO file saved on the attached storage (NFS or CIFS) you have to use the correct profile: Other install media
To use the Ubuntu Lucid Lynx 10.04(32 bit) profile you have to specify an URL for the data to be retrived (for example like http://mirror.clarkson.edu/ubuntu/ )

References:
http://lists.xensource.com/archives/html/xen-users/2011-06/msg00394.html

Screen trick how to simultaneously monitor network traffic and disk activity and processes in Linux

2011-05-19T23:15:00.001+01:00

As a system administrator sooner than later you will discover the 'screen' command. It is very powerful tool that allows you practically do anything with your (black) terminal screen or run commands in background for you.

The tools we will use for monitoring:
- htop is a top like process monitoring tool for text consoles
- nethogs is a network monitoring tool (needs root privileges!)
- iotop is a disk I/O monitoring tool

To start a single screen session, divided in 3 vertical smaller screens, each running its own monitoring program try this recipe:

# cat <<END >/tmp/myscreen.cfg
startup_message off
screen -t top htop
split
focus
screen -t nethogs  nethogs wlan0
split
focus
screen -t iotop iotop
END

# screen -c /tmp/myscreen.cfg

References:
Top 10 Awesome Linux Screen Tricks

By using Ubuntu how to create a bootable USB thumb drive with Fedora distro

2011-05-04T23:33:00.000+01:00

Although this it has already been possible to use USB thumb drive and boot from it for a longer time, there is still no one unified method to create this USB thumb drive.

Ubuntu has its own method:

"2) Burn your CD or create a USB drive"
http://www.ubuntu.com/download/server/download

Fedora has its own as well:

"3.2.1. Making Fedora USB Media on a Windows Operating System"
http://docs.fedoraproject.org/en-US/Fedora/14/html/Installation_Guide/Making_USB_Media.html

Fortunately for us a separate project unetbootin [1] comes in handy to do this job regardless what OS you are currently using.

To simply install and run follow this instructions bellow.

# aptitude install unetbootin # dpkg -L  unetbootin | grep bin/usr/bin/unetbootin# unetbootin

References:

[1]
UNetbootin - Homepage and Downloads

Using Fedora's liveusb-creator on Ubuntu Lucid Lynx

Gdisk error message: Caution: invalid main GPT header, but valid backup; regenerating main header from backup!

2011-05-01T23:06:00.000+01:00

At some point when experimenting with partitions on your disk you may get the following error message.

% gdisk /dev/sda
GPT fdisk (gdisk) version 0.7.1

Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!

Caution! After loading partitions, the CRC doesn't check out!
Warning! Main partition table CRC mismatch! Loaded backup partition table
instead of main partition table!

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: damaged

Found valid MBR and corrupt GPT. Which do you want to use? (Using the
GPT MAY permit recovery of GPT data.)
 1 - MBR
 2 - GPT
 3 - Create blank GPT

As always you have to be very careful depending what you did and what do you want to do as next.

The three options for gdisk gives you the choice what partition tables you will examine. Fortunately for us gpart is not going to write on disk anything until we say so by executing the w (write) command (example bellow). The three options only influences how the data on the disk are going to be interpreted and presented for our review.

# gdisk /dev/sda
Command (? for help): h
b    back up GPT data to a file
c    change a partition's name
d    delete a partition
i    show detailed information on a partition
l    list known partition types
n    add a new partition
o    create a new empty GUID partition table (GPT)
p    print the partition table
q    quit without saving changes
r    recovery and transformation options (experts only)
s    sort partitions
t    change a partition's type code
v    verify disk
w    write table to disk and exit
x    extra functionality (experts only)
?    print this menu

At this stage it is save to experiment and have a look how the partition tables look like. Depending on your choise different partitions may be printed ('p') .

REMEMBER to always quit the session with the 'q' (quit) and never with 'w' otherwise your experiments will be permanently saved on the disk.

In my case this error was misleading and confusing only. I could verify that my current partition schema was not GPT at all but the good old MSDOS one [1]. At this stage I knew that 'gdisk' is not the tool I wanted to use and I finished creating new partitions with 'parted'. I didn't have to use gdisk at all.

The reason why I had some parts of GPT data on my disk is unknown. I can only suspect that it was created when I played with the Windows tool 'EasyBCD'.

I could try to delete the GPT data but base on [2] never had to do it.

References
[1] How to find what type of disk partition schema do I use (msdos, gpt)

[2] Wiping Out Old GPT Data

How to find what type of disk partition schema do I use (msdos, gpt)

2011-05-01T21:50:00.000+01:00

There are many disk partition schemes that can be used. Very likely on most x86 base computer the choice is often limited by the operating system we use although.

The list of supported labels (partition schemes) in GNU parted tool can be found here [1]:


label-type must be one of these supported disk labels:

* bsd
* loop (raw disk access)
* gpt
* mac
* msdos
* pc98
* sun

To find out what type of partition schema we currently have on your system please run the bellow command and check the value for the 'Partition Table'.

% parted /dev/sda unit mb print
Model: ATA ST2000DL003-9VT1 (scsi)
Disk /dev/sda: 2000399MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start     End       Size      Type     File system  Flags
 1      0.03MB    104856MB  104856MB  primary  ntfs         boot
 2      104857MB  209714MB  104858MB  primary  ntfs
 3      209714MB  312115MB  102401MB  primary
                                      
root@sysresccd /tmp % parted /dev/sda unit s print
Model: ATA ST2000DL003-9VT1 (scsi)
Disk /dev/sda: 3907029168s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start       End         Size        Type     File system  Flags
 1      63s         204796619s  204796557s  primary  ntfs         boot
 2      204797952s  409597951s  204800000s  primary  ntfs
 3      409597952s  609599487s  200001536s  primary

References:

[1]
(parted) mklabel msdos

Others:

Parted User's Manual

GUID Partition Table

Make the most of large drives with GPT and Linux

Fun with GPT partitioning

Linux Creating a Partition Size Larger than 2TB

How to export/backup partition schema for a disk using old style disk partitions format like 'msdos' written into MBR sector

2011-05-01T21:17:00.000+01:00

To backup the partition schema of the disk you can execute:

# sfdisk -d /dev/sda > part.txt

# example output can look like
# cat part.txt
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=       63, size=204796557, Id= 7, bootable
/dev/sda2 : start=204797952, size=204800000, Id= 7
/dev/sda3 : start=409597952, size=200001536, Id=83
/dev/sda4 : start=        0, size=        0, Id= 0

To restore the part.txt you can use:

# sfdisk /dev/hda < part.txt

References:
man sfdisk
Wiping Out Old GPT Data

Howto capture and record the console screen output to a file on disk

2011-05-01T18:38:00.000+01:00

The basic of capturing a data to a file require simple redirection or using of 'tee' program for example.

Examples:

$ ls -la > /var/tmp/output.ls.txt
$ ls -la | tee /var/tmp/output.ls.txt

But sometimes programs can be interactive or we simply want to capture all our session without to worry to redirect the stdout to a file.

The solution is to use the 'screen' tool.

Example #1: Screen basic usage

# to start the session with a name 'example' run
$screen -h
$screen -S example

# to leave the screen session type: CONTROL-a d
# you are placed back in the original shell

# to reattach to the created session
$ screen -ls
$ screen -r example

Example #1: Enable screen logging

The options '-L' instruct screen to create a log file that will capture all the commands output in the screen session.

$ screen -S logexample -L

# inside the screen session
$ echo 'some output'

# to leave the screen session type: CONTROL-a d

# this is the default log file for screen
$ ls -la screenlog.0

$ cat screenlog.0

Automatic debugging session of a programs in C and the gdb analyse of its core dumps

2011-05-01T14:04:00.003+01:00

In our example we are going to concentrate on this little program bad_program to demonstrate our semi automated debug approach.

For debugging purposes we should first compile the program with debugging symbols. The possible ways are listed bellow and should be taken only as simple examples how to do this. More about this in [1].

# no debugging at all
 (0) $ gcc bad_program.c -o bad_program

# with debugging symbols  
 (1) $ gcc -g3     bad_program.c -o bad_program1
 (2) $ gcc -g      bad_program.c -rdynamic -o bad_program2
 (3) $ gcc -g  -O0 bad_program.c -rdynamic -o bad_program3
 (4) $ gcc -g3 -Os bad_program.c -o bad_program4

The test has been run on a following system.

$ gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.3-4ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-plugin --enable-objc-gc --enable-targets=all --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) 

$ gdb -v
GNU gdb (GDB) 7.1-ubuntu

The example program itself is written in such a way that it crushes and generates core dump each time when started.

Example 1

$ ./bad_program 1
Segmentation fault

To get the core dump file written on a disk that can be later analysed in gnu debugger (gdb) we need to first allow core dumps.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ ulimit -c unlimited

When we run the program again it will create the core file for later analyse.

Example 2

$ ./bad_program 1
Segmentation fault (core dumped)

$ ls -la core 
-rw------- 1 radoslaw radoslaw 151552 2011-04-24 19:50 core

In our example we only want to run the gdb 'where' command, but if needed the file my_session.gdb.cmds bellow can be extended for any number of commands we may be interested in. More about the useful gdb debugging commands can be found in [2].

$ cat my_session.gdb.cmds 
where

Our test session can look like:

Debugging session #1

$ export BAD_PROGRAM=bad_program
$ for i in $(seq 1 4); do rm -f core; echo;  echo " ---- ---- ---- [$i] starting the program ---- ---- ----"; ./$BAD_PROGRAM $i ; echo " ---- ---- ---- [$i] starting gdb ---- ---- ----"; gdb -batch -x my_session.gdb.cmds -n $BAD_PROGRAM core ; done | tee ${BAD_PROGRAM}.log

 ---- ---- ---- [1] starting the program ---- ---- ----
 ---- ---- ---- [1] starting gdb ---- ---- ----

warning: Can't read pathname for load map: Input/output error.[New Thread 16934]

1051 ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S: No such file or directory.
Core was generated by `./bad_program 1'.
Program terminated with signal 11, Segmentation fault.
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
 in ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
#1  0x00e9073c in ?? () from /lib/ld-linux.so.2
#2  0x0804844d in generate_core ()
#3  0x08048566 in main ()

 ---- ---- ---- [2] starting the program ---- ---- ----
 ---- ---- ---- [2] starting gdb ---- ---- ----
[New Thread 16938]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 2'.
Program terminated with signal 11, Segmentation fault.
#0  0x0804852b in try_core2 ()
#0  0x0804852b in try_core2 ()
#1  0x0804848e in generate_core ()
#2  0x08048566 in main ()

 ---- ---- ---- [3] starting the program ---- ---- ----
 ---- ---- ---- [3] starting gdb ---- ---- ----

warning: [New Thread 16942]
Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 3'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 ()
#0  0x08048541 in try_core3 ()
#1  0x080484cf in generate_core ()
#2  0x08048566 in main ()

 ---- ---- ---- [4] starting the program ---- ---- ----
 ---- ---- ---- [4] starting gdb ---- ---- ----
[New Thread 16946]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 4'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 ()
#0  0x08048541 in try_core3 ()
#1  0x080484cf in generate_core ()
#2  0x080484eb in generate_core ()
#3  0x08048566 in main ()

We can still see enough to say where the problem happened more or less but we get much better results when a program is run with debugging symbols.

Debugging session #2

$ export BAD_PROGRAM=bad_program1
$ for i in $(seq 1 4); do rm -f core; echo;  echo " ---- ---- ---- [$i] starting the program ---- ---- ----"; ./$BAD_PROGRAM $i ; echo " ---- ---- ---- [$i] starting gdb ---- ---- ----"; gdb -batch -x my_session.gdb.cmds -n $BAD_PROGRAM core ; done | tee ${BAD_PROGRAM}.log

 ---- ---- ---- [1] starting the program ---- ---- ----
 ---- ---- ---- [1] starting gdb ---- ---- ----

warning: [New Thread 17974]
Can't read pathname for load map: Input/output error.
1051 ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S: No such file or directory.
Core was generated by `./bad_program1 1'.
Program terminated with signal 11, Segmentation fault.
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
 in ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
#1  0x00f6973c in ?? () from /lib/ld-linux.so.2
#2  0x0804844d in generate_core (n=1) at bad_program.c:15
#3  0x08048566 in main (argc=2, argv=0xbfb572a4) at bad_program.c:52

 ---- ---- ---- [2] starting the program ---- ---- ----
 ---- ---- ---- [2] starting gdb ---- ---- ----

warning: [New Thread 17978]
Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 2'.
Program terminated with signal 11, Segmentation fault.
#0  0x0804852b in try_core2 (n=2) at bad_program.c:41
41   *ptr=n;
#0  0x0804852b in try_core2 (n=2) at bad_program.c:41
#1  0x0804848e in generate_core (n=2) at bad_program.c:20
#2  0x08048566 in main (argc=2, argv=0xbff99024) at bad_program.c:52

 ---- ---- ---- [3] starting the program ---- ---- ----
 ---- ---- ---- [3] starting gdb ---- ---- ----
[New Thread 17982]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 3'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
47   *(ptr+n)=n; 
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
#1  0x080484cf in generate_core (n=3) at bad_program.c:25
#2  0x08048566 in main (argc=2, argv=0xbf92fdd4) at bad_program.c:52

 ---- ---- ---- [4] starting the program ---- ---- ----
 ---- ---- ---- [4] starting gdb ---- ---- ----
[New Thread 17986]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 4'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
47   *(ptr+n)=n; 
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
#1  0x080484cf in generate_core (n=3) at bad_program.c:25
#2  0x080484eb in generate_core (n=4) at bad_program.c:29
#3  0x08048566 in main (argc=2, argv=0xbfdd2994) at bad_program.c:52

We can see in lines # 16, 27, 40, 53 the instructions that caused the core dump. We see as well the full arguments in functions what helps to better understand the program logic flow. One thing more as well to notice is the difference in debugging output when analysing the core from:

./bad_program  1   # no debugging symbols; versus
./bad_program1 1   # with debugging symbols

From the debugging session #1 we can hardly guess where the problem was, where in debugging session #2 we clearly see that the problem started with line 'bad_program.c:15'.

Example program

The "bad" example program that cores every time when run is bellow. More info about this in [3].

Source code of the bad_program.c

#include 
#include 
#include 
#include 
#include 

void try_core1(int n);
void try_core2(int n);
void try_core3(int n);
void generate_core(int n);

void generate_core( int n ) {
  if ( 1 == (n%10) ) { 
    try_core1(n);
    generate_core(n-1);
  } 
  
  if ( 2 == (n%10) ) { 
    try_core2(n);
    generate_core(n-1);
  } 
  
  if ( 3 == (n%10) ) { 
    try_core3(n);
    generate_core(n-1);
  } 
  
  generate_core(n-1);
}

void try_core1( int n ) {
  char *ptr=NULL;
  
  strcpy(ptr, "this is going to hurt ;)...");
}

void try_core2( int n ) {
  int *ptr=NULL;
  
  *ptr=n;
}

void try_core3( int n ) {
  int *ptr;
  
  *(ptr+n)=n; 
}


int main(int argc, char **argv) {
  generate_core( atoi( argv[1] ) );
}

References:
[1]
HowTo: Debug Crashed Linux Application Core Files Like A Pro
Debugging with GDB

[2]
Mastering Linux debugging techniques
Linux software debugging with GDB
GDB: The GNU Project Debugger

[3]
How to programatically cause a core dump in C/C++

How to create a bootable USB thumb drive for Windows 7 64bit using Windows XP 32bit machine.

2011-04-24T18:02:00.000+01:00

Problem:
We have the ISO of the Windows 7 64bit and want to install it using the USB Thumb drive installation method.
We have access only to our old Windows XP 32bit OS.

Solution:
Use the standard MS tool to format the USB Thumb drive [1].
It will failed at the 100% with an error message saying:


Status: Files copied successfully. However, we were unable to run bootsect to make the USB device
bootable. If you need assistance with bootsect ....#truncated#

Download and install the EasyBCD for your WinXP 32bit [2].
The installation will copy the bootsect.exe tool that can be used on winXP.

Run the bootsect.exe to make the USB Thumb drive bootable. The 'E:' is the driver letter assign by your XP!

C:\Program Files\NeoSmart Technologies\EasyBCD\bin>bootsect.exe /nt60 E:
Target volumes will be updated with BOOTMGR compatible bootcode.

E: (\\?\Volume{ba063d8e-7111-11e0-b640-aafb6b9edf9e})

    Successfully updated NTFS filesystem bootcode.

Bootcode was successfully updated on all targeted volumes.

As last, unmount the USB, reboot the system, change the BIOS and start the installation.

References:

[1] Windows 7 USB/DVD Download Tool

[2] EasyBCD

For troubleshooting purposes:
How To Boot And Install Windows 7 From USB Flash Drive

How to clone a partition to a remote NFS server

2011-04-24T16:24:00.048+01:00

Problem description.
You formatted and re-partitioned your hard drive.
You have a partition of the size about 100GB that you dedicated to your OS.
The installation of your OS went fine as well as the installation of all your necessary drivers.

After all you would like to have a copy of your partition data that can quickly restore in a case something wrong happens.

Solution:
Download the SystemRescueCd ISO img file [1]
Create bootable USB thumb drive [2]
Boot the system using your USB thumb drive. Once it is fully loaded you will get bash command line access.

Mount your remote NFS site image repository:

# mkdir /tmp/nfs
# mount -t nfs -o nolock 192.168.0.X:/your/path /tmp/nfs

Copy and compress the partition while coping it out for backup purposes [3]. Copy the partition schema as well.

# For test purposes you can shortly run the partclone.dd with '-d' option
# it will create a log file in /var/log directory.
# But don't run it like that otherwise it will quickly fill your log partition

# partclone.dd -c -d -s /dev/sda1 -o - | bzip2 -1 -c > /tmp/nfs/sda1.winXP.32bit.img

# Once tested run without -d option
#
# partclone.dd -c -s /dev/sda1 -o - | bzip2 -1 -c > /tmp/nfs/sda1.winXP.32bit.img
# parted -l > /tmp/nfs/log.parted.l.txt
# parted -l -m > /tmp/nfs/log.parted.lm.txt
#
# to backup the partition table from the MBR sector (don't use it on GPT partitioned system)
# sfdisk -d > /tmp/nfs/log.sfdisk.d.txt
#
# umount /tmp/nfs

At the end unmount the NFS partition and we are ready to reboot the system.

Later on, as a security precaution create md5sums and some level of par2 recovery files if the image should have ever get corrupted.

# md5sum -b sda1.winXP.32bit.img > sda1.winXP.32bit.img.md5
# par2create -r3 sda1.winXP.32bit.img

References:

[1] SystemRescueCd

[2] Sysresccd-manual-en How to install SystemRescueCd on an USB-stick

[3] partclone.dd

man 5 nfs

par2 home page
man par2

Less tricks in linux on the bash command line

2011-04-24T13:46:00.002+01:00

Some tricks that is worth to know when working on the linux bash cli that can save you a lot of time.

1. To tell less to normal print the raw control character of your stream you can use the '-r' or '-R' options.

$ ls -la --color | less -X 
total 3284
drwxr-xr-x 90 radoslaw radoslaw   4096 2011-04-24 10:22 ESC[0mESC[01;34m.ESC[0m
drwxr-xr-x  4 root     root       4096 2010-08-02 20:37 ESC[01;34m..ESC[0m
drwx------  5 radoslaw radoslaw   4096 2011-01-31 12:18 ESC[01;34m.adobeESC[0m
drwxr-xr-x  4 radoslaw radoslaw   4096 2009-11-07 20:53 ESC[01;34m.ankiESC[0m

$ ls -la --color | less -X -r
total 3284
drwxr-xr-x 90 radoslaw radoslaw   4096 2011-04-24 10:22 .
drwxr-xr-x  4 root     root       4096 2010-08-02 20:37 ..
drwx------  5 radoslaw radoslaw   4096 2011-01-31 12:18 .adobe
drwxr-xr-x  4 radoslaw radoslaw   4096 2009-11-07 20:53 .anki

2. To list only #n lines from the input with or without the raw characters being interpreted.

$ ls -la --color | cat -t | head -10
total 3284
drwxr-xr-x 90 radoslaw radoslaw   4096 2011-04-24 10:22 ^[[0m^[[01;34m.^[[0m
drwxr-xr-x  4 root     root       4096 2010-08-02 20:37 ^[[01;34m..^[[0m
drwx------  5 radoslaw radoslaw   4096 2011-01-31 12:18 ^[[01;34m.adobe^[[0m
drwxr-xr-x  4 radoslaw radoslaw   4096 2009-11-07 20:53 ^[[01;34m.anki^[[0m

$ ls -la --color | head -10
total 3284
drwxr-xr-x 90 radoslaw radoslaw   4096 2011-04-24 10:22 .
drwxr-xr-x  4 root     root       4096 2010-08-02 20:37 ..
drwx------  5 radoslaw radoslaw   4096 2011-01-31 12:18 .adobe
drwxr-xr-x  4 radoslaw radoslaw   4096 2009-11-07 20:53 .anki

3. To list files with line numbers.

$ less -N -X /etc/init.d/networking 
      1 #!/bin/sh -e
      2 ### BEGIN INIT INFO
      3 # Provides:          networking
      4 # Required-Start:    

$ ls -la --color | cat -n -t | head -5
     1 total 3284
     2 drwxr-xr-x 90 radoslaw radoslaw   4096 2011-04-24 10:22 ^[[0m^[[01;34m.^[[0m
     3 drwxr-xr-x  4 root     root       4096 2010-08-02 20:37 ^[[01;34m..^[[0m
     4 drwx------  5 radoslaw radoslaw   4096 2011-01-31 12:18 ^[[01;34m.adobe^[[0m
     5 drwxr-xr-x  4 radoslaw radoslaw   4096 2009-11-07 20:53 ^[[01;34m.anki^[[0m

$ ls -la --color | cat -n | head -5
     1 total 3284
     2 drwxr-xr-x 90 radoslaw radoslaw   4096 2011-04-24 10:22 .
     3 drwxr-xr-x  4 root     root       4096 2010-08-02 20:37 ..
     4 drwx------  5 radoslaw radoslaw   4096 2011-01-31 12:18 .adobe
     5 drwxr-xr-x  4 radoslaw radoslaw   4096 2009-11-07 20:53 .anki

Reference:

What does my bash ulimit function use to set/report the limitation of resources

2011-04-11T23:38:00.000+01:00

Everyone knows the output from the 'ulimit' bash, builtin function. But what functionality actually this function uses to mange all of this.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Let's find out if the 'strace' program come in handy for this job.

# let's create auxilary file we will use later
$ cat mylimits.sh
ulimit -a

$ strace bash mylimits.sh 1>/dev/null 2>/tmp/strace.out
$ tail -20 /tmp/strace.out 
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
write(1, "open files                      "..., 42) = 42
write(1, "pipe size            (512 bytes,"..., 39) = 39
getrlimit(RLIMIT_MSGQUEUE, {rlim_cur=800*1024, rlim_max=800*1024}) = 0
write(1, "POSIX message queues     (bytes,"..., 44) = 44
getrlimit(RLIMIT_RTPRIO, {rlim_cur=0, rlim_max=0}) = 0
write(1, "real-time priority              "..., 39) = 39
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
write(1, "stack size              (kbytes,"..., 42) = 42
getrlimit(RLIMIT_CPU, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
write(1, "cpu time               (seconds,"..., 47) = 47
getrlimit(RLIMIT_NPROC, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
write(1, "max user processes              "..., 47) = 47
getrlimit(RLIMIT_AS, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
write(1, "virtual memory          (kbytes,"..., 47) = 47
getrlimit(RLIMIT_LOCKS, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
write(1, "file locks                      "..., 47) = 47
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
read(255, "", 10)                       = 0
exit_group(0)                           = ?

So as we can see the limitations are managed by the 'getrlimit' function and enforced on the kernel level.

For more info about this take a look at man 2 setrlimit

For these who are not afraid of glibc library a quick look at they wrappers may help as well: glibc "Resource Usage And Limitation"

An obsolete kernel function man 3 ulimit

How to filter and dump in clear text http requests and responses from the network dump using the cli program tshark

2011-04-10T22:13:00.000+01:00

There are many programs that allow us to save network traffic into a file for later analyze. Once you have the file(s) we could like to quickly investigate the data inside and verify what we sent and what we revived.

If we are interested, let's say only in the HTTP traffic than the command line 'tshark' has only limited capabilities in a way to present us the data for review and investigation.

We can still play with the various options: '-Tfields' and multiple '-e' but still don't get the complete headers output.

The complete list that can be used with '-e' can be found here .
Alternatively we can experiment with the '-Tpdml' that will create alike XML file. But even with all this flexibility we still can't print a custom name header like in this curl request bellow:

curl -H "Rado: my_value" -v -o /tmp/page.html http://rtomaszewski.blogspot.com/2011/04/tshark-in-network-troubleshooting.html

To solve this little problem we have created a small program written in python. It takes the output from 'tshark -S -V' and parses it to present the data in a way we want.

Example how to use it:

# tshark -r /tmp/net.pcap -R http -V | parse.py -d
# tshark -r /tmp/net.pcap -R http -V | parse.py 
# tshark -nn -s0 -i any -w /tmp/net.pcap  -f tcp -R http -l -S -V | parse.py
# tshark -nn -s0 -i any -w /tmp/net.pcap  -f tcp -R http -l -S -V | parse.py -d

An example output from the 'parse.py' when running on the command line:

$ curl -H "Rado: my_value" -v -o /tmp/page.html http://rtomaszewski.blogspot.com/2011/04/tshark-in-network-troubleshooting.html 
* About to connect() to rtomaszewski.blogspot.com port 80 (#0)
*   Trying 209.85.229.132... connected
* Connected to rtomaszewski.blogspot.com (209.85.229.132) port 80 (#0)
> GET /2011/04/tshark-in-network-troubleshooting.html HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: rtomaszewski.blogspot.com
> Accept: */*
> Rado: my_value
> 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
< ETag: "b6d5837e-31bb-4473-95af-da3c1d466295"
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Server: GSE
< Age: 1202
< Date: Sun, 10 Apr 2011 19:14:00 GMT
< Expires: Sun, 10 Apr 2011 19:14:00 GMT
< Last-Modified: Sun, 10 Apr 2011 19:03:05 GMT
< Cache-Control: public, must-revalidate, proxy-revalidate, max-age=0
< Transfer-Encoding: chunked
< 
{ [data not shown]
100 49993    0 49993    0     0  28925      0 --:--:--  0:00:01 --:--:-- 57529* Connection #0 to host rtomaszewski.blogspot.com left intact

* Closing connection #0

# tshark -nn -s0 -i wlan0 -w /tmp/net.pcap -R http -f tcp -S -V | ./parse.py  
Running as user "root" and group "root". This could be dangerous.
Capturing on wlan0
Internet Protocol, Src: 192.168.43.111 (192.168.43.111), Dst: 209.85.229.132 (209.85.229.132)
Transmission Control Protocol, Src Port: 38391 (38391), Dst Port: 80 (80), Seq: 1, Ack: 1, Len: 226
    [Stream index: 1]
    GET /2011/04/tshark-in-network-troubleshooting.html HTTP/1.1\r\n
    User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15\r\n
    Host: rtomaszewski.blogspot.com\r\n
    Accept: */*\r\n
    Rado: my_value\r\n

Internet Protocol, Src: 209.85.229.132 (209.85.229.132), Dst: 192.168.43.111 (192.168.43.111)
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 38391 (38391), Seq: 49070, Ack: 227, Len: 1375
    [Stream index: 1]
    HTTP/1.1 200 OK\r\n
    Content-Type: text/html; charset=UTF-8\r\n
    ETag: "b6d5837e-31bb-4473-95af-da3c1d466295"\r\n
    X-Content-Type-Options: nosniff\r\n
    X-XSS-Protection: 1; mode=block\r\n
    Server: GSE\r\n
    Age: 1202\r\n
    Date: Sun, 10 Apr 2011 19:14:00 GMT\r\n
    Expires: Sun, 10 Apr 2011 19:14:00 GMT\r\n
    Last-Modified: Sun, 10 Apr 2011 19:03:05 GMT\r\n
    Cache-Control: public, must-revalidate, proxy-revalidate, max-age=0\r\n
    Transfer-Encoding: chunked\r\n

2 packets captured

The program can be downloaded form here: parse.py Additionally the source code of this short tool can be seen here:

#!/usr/bin/python 
#
## tested on python 2.6.5
#
# author : radoslaw tomaszewski

import sys
import re
import inspect


class ParseTsharkOut:
  no=0
  debugYes=0 
    
  ipRe=None
  tcpRe=None
  protRe=None
  
  ipInfo=[]
  tcpInfo=[]
  protInfo=[]
  
  auxStart=0
  tcpAuxStart=0

  protAux=1
  
  def __init__(self):
    self.ipRe="Internet Protocol,(.*)$"
    self.tcpRe=["Transmission Control Protocol,(.*)$", "    (\[Stream index:.*)$"]
    self.protRe=["Hypertext Transfer Protocol", "    ([^\[ ].*)$", "^$| *(\\\\r|\\\\n)"]
    
  def debug(self, s):
    if self.debugYes : 
      parent=inspect.stack()[1][3]
      #parent=inspect.stack()
      print("debug:[" + str(parent) + "] " +  s.rstrip())
  
  def usage(self):
    print("todo")   
  
  def ipParse(self,s):
    self.debug(s)

    tmp=re.match(self.ipRe, s)
#    tmp=re.match("..", s)
    if tmp is None :
       return 0
    else:
       self.ipInfo.append(tmp.group(0))
       return 1
       
  def tcpParse(self,s):
    self.debug(s)
    ret=0

    tmp=re.match(self.tcpRe[self.tcpAuxStart], s)
    if tmp is None :
       return 0
    else:
       self.tcpInfo.append(tmp.group(0))
       
       ret=self.tcpAuxStart
       self.tcpAuxStart=(self.tcpAuxStart + 1 ) % 2
       
       return ret
  
  def protParse(self,s):
    self.debug(s)

    if ( self.protAux ) :
       if re.match(self.protRe[0], s):
         self.protAux=0
         return 0
         
    else :
       if re.match(self.protRe[2], s):
         self.protAux=1
         self.show()
         return 1

       tmp=re.match(self.protRe[1], s)
       if tmp is None :
         return 0
       else:
         self.protInfo.append(tmp.group(0))
         return 0
    
  def parse(self, s):
    funcs=[self.ipParse, self.tcpParse, self.protParse]
    
    if funcs[self.auxStart](s):
      self.auxStart= ( self.auxStart + 1 ) % 3 
  
  def show(self):
    self.debug("start")
    
    for i in self.ipInfo:
      print(i)
      
    for i in self.tcpInfo:
      print(i)

    for i in self.protInfo:
      print(i)  
      
    print("")
      
    self.ipInfo=[]
    self.tcpInfo=[]
    self.protInfo=[]      
  
  def main(self):
    try:
      if sys.argv[1] == '-d':
        self.debugYes=1
        self.debug("debuging is turn on")
    except (IndexError):
      None 
      
    self.debug("main start")

    for l in sys.stdin:
      self.parse(l)  
      
    return 0  
      
  

if __name__ == "__main__":
        sys.exit(ParseTsharkOut().main())

aaa

tshark in network troubleshooting

2011-04-10T20:00:00.000+01:00

As described in another of my posts, there is not good and elegant way of printing on stdout and to a file the data captured on the wire when using 'tcpdump'.

In help the 'tshark' that comes with the 'wireshark' packet sniffer comes handy to do the job in a simple and elegant way.

To capture all network packets and to write them to a file you would use a syntax similar to this one:

# tshark -nn -i any -w /tmp/net.pcap
Running as user "root" and group "root". This could be dangerous.
Capturing on Pseudo-device that captures on all interfaces
24

If you don't like the packet counter constantly being updated when tshark runs use the '-q' options like that:

# tshark -nn -i any -w /tmp/net.pcap.pcap -q

And finally when you want to observe the packet in live and in the same time have a raw copy of the network dump on your disk use the '-S' option like that:

# tshark -nn -i any -w /tmp/net.pcap.pcap -S
Running as user "root" and group "root". This could be dangerous.
Capturing on Pseudo-device that captures on all interfaces
  0.000000    127.0.0.1 -> 127.0.0.1    ICMP Echo (ping) request
  0.028908    127.0.0.1 -> 127.0.0.1    ICMP Echo (ping) request
  0.297138 192.168.122.1 -> 192.168.122.255 UDP Source port: 17500  Destination port: 17500
  1.036921    127.0.0.1 -> 127.0.0.1    ICMP Echo (ping) request
  2.045086    127.0.0.1 -> 127.0.0.1    ICMP Echo (ping) request
5 packets captured

After all, when you are done with the troubleshooting and need to convenience someone you can sent him the saved file /tmp/net.pcap for review.

More info
man tshark

How to understand Apache MPM processes during the starting phase

2011-04-05T00:46:00.000+01:00

One of the great flexibility that comes with Apache HTTP server is its incredible modularity. But sometimes it take a little time and experimenting before you understand how your favourite MPM model works.

So let's take a look how many processes there are going to be created when we start Apache after not customized installation?

All descriptions and examples are based on the default configuration in the Ubuntu:

# cat /etc/lsb-release

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=10.04

DISTRIB_CODENAME=lucid

DISTRIB_DESCRIPTION="Ubuntu 10.04.1 LTS"

The installation itself is as simple as:

# aptitude install apache2

By default when you start Apache with its default config it will look like this:

# ps -AHl | egrep 'apach|httpd'
1 S     0 25528     1  0  80   0 -  1348 poll_s ?        00:00:00   apache2
5 S    33 25530 25528  0  80   0 -  1291 skb_re ?        00:00:00     apache2
5 S    33 25532 25528  0  80   0 - 56702 pipe_w ?        00:00:00     apache2
5 S    33 25533 25528  0  80   0 - 56702 pipe_w ?        00:00:00     apache2

A closer look at the configuration in /etc/apache2/apache2.conf reveals more then one section that could be in control of your http server. So which one is our?

# grep -B7 'IfModule mpm_prefork_module'  apache2.conf 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves


# grep -A8 'IfModule mpm_'  apache2.conf 

    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0


# worker MPM
--

    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0

--

    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxRequestsPerChild   0

Well, if you paid attention when installing the server then you know what MPM you installed. If you don't remember than we can find this by running:

# apache2ctl -l
Compiled in modules:
  core.c
  mod_log_config.c
  mod_logio.c
  worker.c  < - - - this is the one we can customize
  http_core.c
  mod_so.c

From the doc Apache MPM worker we know what the above options should mean as well.


    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0

Base on line # 2 (StartServers) we should see 2 processes. But when you scroll up we can see 4! So what is going on there? To understand the situation we need to look at the processes and threads statistics together.

# ps -lFmC apache2
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN    RSS PSR STIME TTY          TIME CMD
1 - root     25528     1  0   -   - -  1348 -       2580   - Apr04 ?        00:00:00 /usr/sbin/apache2 -k start
1 S root         -     -  0  80   0 -     - poll_s     -   0 Apr04 -        00:00:00 -
5 - www-data 25530 25528  0   -   - -  1291 -       1784   - Apr04 ?        00:00:00 /usr/sbin/apache2 -k start
5 S www-data     -     -  0  80   0 -     - skb_re     -   0 Apr04 -        00:00:00 -
5 - www-data 25532 25528  0   -   - - 56736 -       2776   - Apr04 ?        00:00:00 /usr/sbin/apache2 -k start
5 S www-data     -     -  0  80   0 -     - pipe_w     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - inet_c     -   1 Apr04 -        00:00:00 -
5 - www-data 25533 25528  0   -   - - 56736 -       2768   - Apr04 ?        00:00:00 /usr/sbin/apache2 -k start
5 S www-data     -     -  0  80   0 -     - pipe_w     -   0 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - futex_     -   1 Apr04 -        00:00:00 -
1 S www-data     -     -  0  80   0 -     - inet_c     -   1 Apr04 -        00:00:00 -

At the end everything is as it suppose to be. From all the 4 Apache processes the first 2 are for management and communication and then the last 2 are actually the one we defined in our config file to process the client's requests. All lines with the 'futex' show as well that there is 25 worker threads in the last to processes so this is controlled by the parameter ThreadsPerChild and is correct as well.

tcpdump wrapper for all network and systems troubleshooter

2011-03-26T20:00:00.001Z

UPDATE:
This other post describe a more elegant way how to do the job tshark in network troubleshooting

Origin post:

One of the tool that a network engineers relay every day is a network sniffer. One of the most famous I believe is the 'tcpdump'.

Very often when you troubleshoot a problem you run it many time to verify the traffic on the wire. Let say that at some point you see where the problem may be and you need to sent an email with your analyze to another person.

Documenting your results can be very time consuming. To minimize our time and increase the quality of the results we would like to attach the dump files we review ourself of course. Unfortunately it can be a little annoying if we need to repeat our troubleshooting again only to save the dumps on the disk this time. Often sending the analyzed text output form tcpdumps is not enough as well.

This small tcpdump wrapper bellow can save you a lot of time by saving the tcpdump data to file and still letting you to follow the data on the screen in a live troubleshooting.

For couple of examples how to run in please scroll down.

The file with source code can be found here mytcpdump.sh

# you can define the filter and options in your bash variables
# example: 
# T_FILTER='arp or icmp or not ip ( net 10.0.0.0/8 )
# T_OPTIONS='-s0 -nn'

# ------------------------------------------

# arg1 - filter to the wireshark
# arg2 - options to wireshark
mytcpdump () {
 # parse args
 
 DEFAULT_OPT='-s0 -l -nn -w - -i any'
 
 if [ 'x-h' = x"$1" ] ; then 
  echo 
  echo "usage: mytcpdump [arg1] [arg2]"
  echo " arg1 - wireshark network filter, by example: 'arp and (net 10/8)'"
  echo " arg2 - wireshark options, default: '$DEFAULT_OPT'"
  echo ""
  echo " example:"
  echo "   mytcpdump"
  echo "   mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22'"
  echo "   mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22' '-s0 -l -nn -i eth0 -w -' "
  echo
  
  return 
 fi
 
 # filters
 if [ '1' != 1"$1" ] ; then 
  filter="$1"
 elif [ '2' != 2"$T_FILTER" ]; then
  filter=$T_FILTER
 else
  filter=""
 fi

 # options
 if [ '1' != 1"$2" ] ; then 
  opts="$2"
 elif [ '2' != 2"$T_OPTIONS" ]; then
  opts=$T_OPTIONS
 else
  opts="$DEFAULT_OPT"
 fi 
 
 t=`date +%s`;
 echo "[$t]: timestamp is $t" 
 echo "[$t]: wireshark optoins are <$opts>"
 echo "[$t]: wireshark filter is <$filter>"

 cmd="tcpdump $opts $filter"
 echo "[$t]: tcpdump cmd is <$cmd>"
 
 f="/var/tmp/tcpdump.$t.pcap"
 echo "[$t]: tcpdump pcap file <$f>"
 
 chain="$cmd | tee $f | tcpdump -r- -nn"
 echo "[$t]: running the bash command chains <$chain>" 
 
 $cmd | tee $f | tcpdump -r- -nn
}

alias myt='mytcpdump'

Usage help

# myt -h

usage: mytcpdump [arg1] [arg2]
 arg1 - wireshark network filter, by example: 'arp and (net 10/8)'
 arg2 - wireshark options, default: '-s0 -l -nn -w - -i any'

 example:
   mytcpdump
   mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22'
   mytcpdump '(net 10.0.0.0/8 and not net 11.0.0.0/8) and port 22' '-s0 -l -nn -i eth0 -w -'

Examples:

These 2 examples bellow show how to use this small wrapper. Each time we can monitor live traffic on the console output from tcpdump and in the same time be sure that a copy of the raw tcpdump data is written to the disk.

The file you may want to copy then later is shown at the beginning after the header '[timestmap]'. In our examples the file names are:

/var/tmp/tcpdump.1301166859.pcap
/var/tmp/tcpdump.1301166869.pcap

# myt 
[1301166859]: timestamp is 1301166859
[1301166859]: wireshark optoins are <-s0 -l -nn -w - -i any>
[1301166859]: wireshark filter is <>
[1301166859]: tcpdump cmd is <tcpdump -s0 -l -nn -w - -i any >
[1301166859]: tcpdump pcap file </var/tmp/tcpdump.1301166859.pcap>
[1301166859]: running the bash command chains <tcpdump -s0 -l -nn -w - -i any  | tee /var/tmp/tcpdump.1301166859.pcap | tcpdump -r- -nn>
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
reading from file -, link-type LINUX_SLL (Linux cooked)
19:14:19.837101 IP 192.168.43.111 > 212.77.100.101: ICMP echo request, id 42106, seq 5396, length 64
^Ctcpdump: pcap_loop: error reading dump file: Interrupted system call
4 packets captured
6 packets received by filter
0 packets dropped by kernel

# myt 'icmp or arp'
[1301166869]: timestamp is 1301166869
[1301166869]: wireshark optoins are <-s0 -l -nn -w - -i any>
[1301166869]: wireshark filter is <icmp or arp>
[1301166869]: tcpdump cmd is <tcpdump -s0 -l -nn -w - -i any icmp or arp>
[1301166869]: tcpdump pcap file </var/tmp/tcpdump.1301166869.pcap>
[1301166869]: running the bash command chains <tcpdump -s0 -l -nn -w - -i any icmp or arp | tee /var/tmp/tcpdump.1301166869.pcap | tcpdump -r- -nn>
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
reading from file -, link-type LINUX_SLL (Linux cooked)
19:14:29.847649 IP 192.168.43.111 > 212.77.100.101: ICMP echo request, id 42106, seq 5406, length 64
^C2 packets captured
2 packets received by filter
0 packets dropped by kernel
tcpdump: pcap_loop: error reading dump file: Interrupted system call

Funny way to print lines in reverse order

2011-03-12T15:48:00.000Z

This is completely something you need see at least once in your live to know what you never should do ;).

The complete code examples bellow are taken from the original post at Print lines in reverse order

The recommended way:

$ tac file

The more funny way to achieve the same result ;)

#example1
$ cat -n file | sort -k1,1rn | cut -f 2-

#example2
$ perl -e 'print reverse <>' file

#example3; sorry, this one is not a one line command
$ i=0
$ while IFS= read -r arr[i]; do ((i++)); done < file
$ for((j=i-1;j>=0;j--)); do printf "%s\n" "${arr[$j]}"; done

LVM is reporting I/O errors after creating a snapshots and modifying it

2011-03-11T21:45:00.000Z

Once you create a snapshot for one of your logical volumes you can mount it in rw mode and use it as a normal available file system/block device. After some time you may get errors like the one bellow unfortunately.

# lvs
  /dev/group1/snap: read failed after 0 of 4096 at 0: Input/output error
  Volume group group2 is exported
  LV         VG     Attr   LSize  Origin Snap%  Move Log             Copy%  Convert
  l1         group1 owi-ao 12.00m                                                  
  mirror.new group1 mwi-a- 12.00m                    mirror.new_mlog 100.00        
  snap       group1 Swi-I-  4.00m l1     100.00                                    
  snap2      group1 swi-a- 20.00m l1       0.23                                    
  snap3      group1 swi-ao  4.00m l1       1.17                                    
  snap4      group1 swi-a-  4.00m l1       1.17

The key part to understand this is to look at the attribute descriptions of the lvs command.

$ man lvs | less -R  +2/lv_attr -X
1  Volume  type:  (m)irrored,  (M)irrored  without  initial  sync,  (o)rigin,  (p)vmove,  (s)napshot, invalid (S)napshot,
                 (v)irtual, mirror (i)mage, mirror (I)mage out-of-sync, under (c)onversion

              2  Permissions: (w)riteable, (r)ead-only

              3  Allocation policy: (c)ontiguous, c(l)ing, (n)ormal, (a)nywhere, (i)nherited This is capitalised if the volume is  cur‐
                 rently locked against allocation changes, for example during pvmove (8).

              4  fixed (m)inor

              5  State:  (a)ctive,  (s)uspended,  (I)nvalid  snapshot,  invalid  (S)uspended  snapshot, mapped (d)evice present without
                 tables, mapped device present with (i)nactive table

              6  device (o)pen

It means that our snapshot device /dev/group1/snap has run out of free space. The only way to fix it is to remove the snapshot device because we can't trust unfortunately its content any longer. Next time make sure that the additional free space for the snapshot logical volume is big enough when you plan changing a lot of data on it.

# lvremove /dev/group1/snap4
# lvcreate -n snap -s -L group1/l1

Reference:
3.8. Snapshots
LVM is reporting I/O errors, but the disk reports no problems

What storage solution is better – quick comparison between LVM mirroring and software RAID1 in Linux

2011-03-11T19:42:00.002Z

With the introduction of LVM2 we have now the ability to create mirrors in LVM. It is relatively easy to set up configuration and works similar to RAID1.

At some point when playing with it you start asking yourself what is actually the technical difference between them two in Linux:

LVM mirroring
RAID1

For the end user it looks that both provide the same functionality of creating and maintaining a redundant block device. But are they very different from each other? Before I will give you couple of ideas to think about let me refresh just briefly how these 2 technologies are defined.

1. What is RAID (taken from the Red Hat docs)
13.1. What is RAID

...
RAID allows information to be spread across several disks. RAID uses techniques such as disk striping (RAID Level 0), disk mirroring (RAID Level 1), and disk striping with parity (RAID Level 5) to achieve redundancy, lower latency, increased bandwidth, and maximized ability to recover from hard disk crashes.

RAID distributes data across each drive in the array by breaking it down into consistently-sized chunks (commonly 256K or 512k, although other values are acceptable). Each chunk is then written to a hard drive in the RAID array according to the RAID level employed. When the data is read, the process is reversed, giving the illusion that the multiple drives in the array are actually one large drive.
....

2. Logical Volumes (taken from the Red Hat docs)
1.2. Logical Volumes

...
Volume management creates a layer of abstraction over physical storage, allowing you to create logical storage volumes. This provides much greater flexibility in a number of ways than using physical storage directly. With a logical volume, you are not restricted to physical disk sizes. In addition, the hardware storage configuration is hidden from the software so it can be resized and moved without stopping applications or unmounting file systems.
...

As we see the descriptions are quite different. So where the differences are going to be visible? Well, I was trying to find a good technical summary how this 2 techniques work but found only many small articles around on Google and in various docs.

Bellow is only a quick summary what could be good to know when trying to answer my question. For your information as well, the list is not meant to be complete and reflects only my personal state of knowledge.

How will the cache size in modern hard drives impacts the write and read operation of my block device? ( Storage Admin guide )

Is the read/write performance the same for LVM and RAID block devices? Looks, like it is not equal and RAID can manage a faster read throughput when accessing the data ( Linux md vs. LVM performance )

In RAID1 by default both disks are going to be used for read operations. This is the opposite for LVM, where by default only the ‘primary’ disk is used.

With LVM mirroring you can create more then only one mirror. A command like below will create for you 2 separate mirror block devices. In RAID1 you can have only one mirror and other devices will be consider as spare disk only.

# lvcreate -m 2 ...

You can detach the mirror from the ‘primary’ logical volume and start using it for whatever you want (backup, testing, ...). Doing this gives you immediately a copy of your primary logical volume but of course will remove the redundancy, at least temporarily. If you want to have the redundancy back you have to enable it again with a command like:

# lvchange -m ...

LVM supports snapshotting feature on logical volumes. At the time of writing this blog it can’t be used on the mirroring volume but you can always detach the mirror and start using it as a normal logical volume. As soon as it becomes a normal logical volume you can start creating snapshots.

# lvconvert --splitmirrors ...

# lvconvert -m1 ...

As long you have free space in your volume group you can have more then one snapshot.

For LVM mirroring you need by default more devices (+1 for the mirror device and +1 for the logs what together when you want to have only 1 mirror on a logical device you need 3 devices for it). This allocation policy, especially for logs can be changed and is controlled with the following options:

--mirrorlog (lvcreate)
--alloc anywhere (vgcreate)

Having a mirror in place don't limit you in use of other LVM features. It means you should be able to extend you logical volume by extending the physical volume it is base on. Once the local volume grows your file system on this block device can grow as well. All this can be done online without having to take the server off line or down.

Depending on what mounting point are you trying to add to your system you may have some trouble during the boot phase with the default initrd (often it comes when you play with the /boot and / file systems as described here rhelv5-list - LVM mirroring )

Reference:

Red Hat documentation and especially the books of “Logical Volume Manager Administration", “Storage Administratoin Guide”
Heinz Mauelshagen, Consulting Development Engineer, Cluster and Storage Development
The discussion about the technical design between RAID1 and LVM mirroring [linux-lvm] Mirroring implementation
LVM mirroring VS RAID1
Linux md vs. LVM performance
LVM mirroring VS RAID1
Barriers and journaling filesystems
A simple introduction to working with LVM on Debian Administration (especialy the commends are very interesting :) )
RAID verses LVM

Searching trick for the Linux less tool

2011-03-07T23:50:00.000Z

Event after so many year of working on the command line I constantly find some trick ;). This is one of them for the less tool.

With the syntax bellow the less will do a normal search like using the '/' char, but will look for the 2th occurrence of our regex expression and once found will jump to it immediately to display.

$ man tune2fs | less -R  +2/-U -X
      -U UUID
              Set   the   universally   unique   identifier   (UUID)   of   the  filesystem  to  UUID.   The  format  of  the  UUID  is  a  series  of  hex  digits  separated  by  hyphens,  like  this:
              "c1b9d5a2-f162-11cf-9ece-0020afc76f16".  The UUID parameter may also be one of the following:

                   clear  clear the filesystem UUID

                   random generate a new randomly-generated UUID

                   time   generate a new time-based UUID

              The UUID may be used by mount(8), fsck(8), and /etc/fstab(5) (and possibly others) by specifying UUID=uuid instead of a block special device name like /dev/hda1.

              See uuidgen(8) for more information.  If the system does not have a good random number generator such as /dev/random or /dev/urandom, tune2fs will  automatically  use  a  time-based  UUID
              instead of a randomly-generated UUID.

The source of it can be found on the FAQ page of the less itself: Less FAQ

How to sort data on more then one column with Linux sort command

2011-03-07T22:14:00.001Z

You caught always yourself looking for old recipes you did years ago. It is time to document this once and for all.

This will produce a list of services maintained by the upstart init daemon on the Linux system that:

1st is sorted alphabetically base on the status description
2th is sorted numerically base on the pid value
3th is sorted alphabetically base on the service name

$ initctl  list | sed 's/ (/(/g'  | sort -k2,2d -k4,4n -k1,1d | column -t  
bridge-network-interface(eth0)    start/running
bridge-network-interface(lo)      start/running
bridge-network-interface(pan0)    start/running
bridge-network-interface(virbr0)  start/running
bridge-network-interface(wlan0)   start/running
network-interface(eth0)           start/running
network-interface(lo)             start/running
network-interface(pan0)           start/running
network-interface-security        start/running
network-interface(virbr0)         start/running
network-interface(wlan0)          start/running
ufw                               start/running
upstart-udev-bridge               start/running,  process  358
udev                              start/running,  process  366
smbd                              start/running,  process  870
rsyslog                           start/running,  process  889
dbus                              start/running,  process  891
ssh                               start/running,  process  897
gdm                               start/running,  process  904
network-manager                   start/running,  process  905
avahi-daemon                      start/running,  process  912
tty4                              start/running,  process  1046
tty5                              start/running,  process  1051
tty2                              start/running,  process  1060
tty3                              start/running,  process  1062
tty6                              start/running,  process  1065
acpid                             start/running,  process  1070
cron                              start/running,  process  1084
atd                               start/running,  process  1085
libvirt-bin                       start/running,  process  1096
tty1                              start/running,  process  1627
nmbd                              start/running,  process  2156
alsa-mixer-save                   stop/waiting
anacron                           stop/waiting
apport                            stop/waiting
console-setup                     stop/waiting
control-alt-delete                stop/waiting
dmesg                             stop/waiting
failsafe-x                        stop/waiting
hostname                          stop/waiting
hwclock                           stop/waiting
hwclock-save                      stop/waiting
irqbalance                        stop/waiting
module-init-tools                 stop/waiting
mountall                          stop/waiting
mountall-net                      stop/waiting
mountall-reboot                   stop/waiting
mountall-shell                    stop/waiting
mounted-dev                       stop/waiting
mounted-tmp                       stop/waiting
mounted-varrun                    stop/waiting
networking                        stop/waiting
plymouth                          stop/waiting
plymouth-log                      stop/waiting
plymouth-splash                   stop/waiting
plymouth-stop                     stop/waiting
procps                            stop/waiting
rc                                stop/waiting
rcS                               stop/waiting
rc-sysinit                        stop/waiting
screen-cleanup                    stop/waiting
udev-finish                       stop/waiting
udevmonitor                       stop/waiting
udevtrigger                       stop/waiting
ureadahead                        stop/waiting
ureadahead-other                  stop/waiting
usplash                           stop/waiting

Google and system documentation is your friend.

$ man sort
$ info coreutils 'sort invocation'

Sorting based on Multiple columns

Sort Files Like A Master With The Linux Sort Command

How to highlight source text on your blog

2011-03-07T18:45:00.003Z

There are many time situation when you would like to publish a piece of code that should have a special formatting for the viewer.

This blog describe one of the possible solution base on the javascript library: SyntaxHighlighter

To integrate it with your blog follow the description there:

Adding a Syntax Highlighter to your Blogger blog

The list of brushes (javascript scripts) that you have to copy to your blog design template depends on your needs. If you want to use these 2 like 'plain' and 'bash' brushes you have to extend the list.

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'/>

<script language='javascript'>

The list of all brushes can be found at Bundled Brushes.

To convert the 'raw html' into the 'escaped' html code you can use this convert tool .

A basic example use in your blog post can look like this:

<pre class="brush:html; highlight: [1];">
<h2>This is an example.</h2>
</pre>

pretty printing for bash command outputs

2011-03-07T17:05:00.007Z

Some commands like to print the output in a way that is hard to read. Try this 'column' tool that turns the unstructured output into a pretty printed output.

$ mount | column -t
/dev/sda2         on  /                         type  ext3                   (rw,relatime,errors=remount-ro)
proc              on  /proc                     type  proc                   (rw)
none              on  /sys                      type  sysfs                  (rw,noexec,nosuid,nodev)
none              on  /sys/fs/fuse/connections  type  fusectl                (rw)
none              on  /sys/kernel/debug         type  debugfs                (rw)
none              on  /sys/kernel/security      type  securityfs             (rw)
...

How to upgrade ubuntu distro

2009-11-08T01:20:00.004Z

I had to update and upgrade the Ubuntu on my laptop. There are 2 well know options how you can do this.

Using the graphical GUI
# update-manager

Using the text command GUI

# type -a do-release-upgrade
do-release-upgrade is /usr/bin/do-release-upgrade
# dpkg -S /usr/bin/do-release-upgrade
update-manager-core: /usr/bin/do-release-upgrade
root@radexlaptop:/home/radoslaw# do-release-upgrade --help
Usage: do-release-upgrade [options]

Options:
-h, --help show this help message and exit
-d, --devel-release Check if upgrading to the latest devel release is
possible
-p, --proposed Try upgrading to the latest release using the upgrader
from $distro-proposed
-m MODE, --mode=MODE Run in a special upgrade mode. Currently 'desktop' for
regular upgrades of a desktop system and 'server' for
server systems are supported.
-f FRONTEND, --frontend=FRONTEND
Run the specified frontend
-s, --sandbox Test upgrade with a sandbox aufs overlay
root@radexlaptop:/home/radoslaw# do-release-upgrade -d
Checking for a new ubuntu release
No new release found

Linux trick with grep tool

2009-11-07T22:30:00.002Z

Command line tools are like grep are very useful when working a lot with text files.

One unknown feature is the --color options which shows you in different colour the matching string grep found.

user@host:~$ egrep '(vmx|svm)' --color=always /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida tpr_shadow vnmi flexpriority

Which CPU for XenServer do I need

2009-11-04T16:33:00.005Z

Citrix XenServer to be able to virtualise Windows OS need some support on the hardware site. You have to have a CPU which bring the "virtualisation feature". It's mean: Intel-VT or Amd-V.

To see if your processor is 64 bit, you can run the following command:

# egrep --color ' lm ' /proc/cpuinfo

How you can check if your CPU has one of these necessary features:

# egrep --color '^flags.*(vmx|svm)' /proc/cpuinfo

How to check if your CPU supports hardware virtualization
Hyper-V: Will My Computer Run Hyper-V? Detecting Intel VT and AMD-V
Does your CPU run Intel-VT or AMD-V?
XenServer 3.2 Hardware Support FAQ

Small trick with a 'stat' tool in Linux

2009-11-04T16:33:00.003Z

I used many times 'stat' tool to find out information about a physical file. Normally the output looks like that:

user@host:/tmp$ stat myfile.txt
File: `myfile.txt'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 325124 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/radoslaw) Gid: ( 1000/radoslaw)
Access: 2009-11-04 16:15:07.000000000 +0100
Modify: 2009-11-04 16:15:07.000000000 +0100
Change: 2009-11-04 16:15:07.000000000 +0100

There is another very useful options, -f, that shows different information.

user@host:/tmp$ stat -f myfile.txt
File: "myfile.txt"
ID: a0fb1614334b9c81 Namelen: 255 Type: ext2/ext3
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 2460827 Free: 647809 Available: 522804
Inodes: Total: 625856 Free: 407715

In the output you can find a lot of file system related information, like the block size or file system name.

And When you want to find only the file system name you can run:

user@host:/tmp$ stat -f -c "%T" myfile.txt
ext2/ext3

How to configure a Linux (Ubuntu) notebook to use a mobile phone with UMTS card as a access modem to Internet

2009-11-04T16:32:00.001Z

You have a mobile phone, like this htc advantag 7510 and you want to use it as a Internet modem.

Everything what you have to do is:

Configure bluetooth in phone

enable the bluetooth

turn on the "Internet sharing" application
estabilish the connection

Configure a Linux host

pand --connect 00:17:E6:E7:E3:D2
dhclient -1 -d bnep0

More info about the configuration for Ubuntu can you find here:

http://www.geekzone.co.nz/content.asp?contentid=1524

http://ubuntuforums.org/showthread.php?p=4145133#post4145133

http://forum.eeeuser.com/viewtopic.php?pid=23948#p23948

http://ubuntu-utah.ubuntuforums.org/showthread.php?t=896244

Linux, date tool trick

2009-11-04T16:31:00.001Z

How many times did you have to get same date and do something with it. To calculate this dates is very annoying, unless you know a few tricks with date command :).

How to find quickly some dates:

$ date -d yesterday
$ date -d '2 days ago'
$ date -d '1 week ago'
$ date -d tomorrow
$ date -d '2 days'
$ date -d '1 week'

For example, when did you born:

$ date
Di 12 Aug 11:44:57 CEST 2008
$ date -d "11 days ago 7 month ago 28 years ago"
Di 1 Jan 10:43:49 CET 1980

Linux Admin bash command diary

2009-11-04T16:30:00.000Z

External hard disk drive "CnMemory HDD 2,5 " and NTFS file system problem

The external HDD drivers can bring some problem when mounting if they have been formatted and use the NTFS file system. To workaround some annoying warning when the device has not been "clean" removed from the windows or linux system use the mount command with following options:

root@radex:/# cat /etc/issue
Ubuntu 7.10 \n \l
root@host:/# mount -t ntfs-3g /dev/sdb1 /mnt/tmp -o force
$LogFile indicates unclean shutdown (0, 0)
WARNING: Forced mount, reset $LogFile.

Configuration Management and DB2 parameters

2009-11-04T16:29:00.001Z

Configuration Management is one of the most important thing in problem and incident management. Even in a simple situation when our db server with important db's will totally demanded we will need some "secured", high tuned parameters for db, application or for the operating system.

To be able to collect this db parameters we can write some simple script which will do this for us. The most important thing in this script that we need to thing about are: looking and deadlock situation in the db.

For IBM db2 we can use following peace of code in Perl, which will use "uncommitted lock" during the connection with db to get the db parameters:

446: $id= getpwnam("$user_name");
447: @tmp=getpwuid($id);
448: $home_dir=$tmp[7];
449: $cmd="db2 change isolation to ur ; " .
       "db2 connect to $db_name ; " .
       "db2 list tables for all show detail; db2 connect reset";
450:    
451: @tmp=`sudo su - $user_name -c "$cmd" 2>&1`;

The complete script can be found there db2-cfg-info.pl

Linux, find tool trick

2009-11-04T16:28:00.002Z

Many times when you are working with some application which create some lock/log/shared/record files you want to know what is happening for a sort time ago for example. The quick trick with the find tool allow you to find for example all files which ware created/changed for _xyz_ minutes ago.

How to find quickly new created files in some directory:

$ find /tmp/ -cmin -1 -maxdepth 1
$ find /tmp/ -cmin -10 -maxdepth 1

Potential problems with "Write Back Caching"

2009-11-04T16:28:00.001Z

For the modern hd drives there is something what can make some problems when the operating system crushed. It is also got to know for troubleshooting and tuning of os and application like data base for example.

Try it for your self:

# man hdparm
# man sdparm

To disable write cache execute:
#hdparm -W0 /dev/hdX | device
#sdparm --set WCE=0

To enable write cache execute:
#hdparm -W1 device
#sdparm --set WCE=1

# sdparm -a -e /dev/sda
# sdparm --page=ca --long /dev/sda
/dev/sda: ATA Hitachi HTS72201 DCDO
Direct access device specific parameters: WP=0 DPOFUA=0
Caching (SBC) [ca] mode page:
IC 0 Initiator control
ABPF 0 Abort pre-fetch
CAP 0 Caching analysis permitted
DISC 0 Discontinuity
SIZE 0 Size enable
WCE 1 Write cache enable
MF 0 Multiplication factor
RCD 0 Read cache disable
DRRP 0 Demand read retention priority
WRP 0 Write retention priority
DPTL 0 Disable pre-fetch transfer length
MIPF 0 Minimum pre-fetch
MAPF 0 Maximum pre-fetch
MAPFC 0 Maximum pre-fetch ceiling
FSW 0 Force sequential write
LBCSS 0 Logical block cache segment size
DRA 0 Disable read ahead
NV_DIS 0 Non-volatile cache disable
NCS 0 Number of cache segments
CSS 0 Cache segment size

a few links in net:
The sdparm utility
Praxis-Know-how: RAID-Controller optimal konfigurieren
MySql InnoDB Configuration
Experiments on Disk Write Back Caches in Linux
The Linux Page Cache and pdflush: Theory of Operation and Tuning for Write-Heavy Loads
Linux IDE Problem beim Write Back Caching
turning on dma and write back cache on SATA

Convert source code to syntax highlighted document

2009-11-04T16:26:00.002Z

How to conver some source code to nice looking [x]html code :)?

The debian apt-cache tool is your friend. Try the following commands:

apt-cache search syntax highlighted
apt-cache search source converter
apt-cache search source color html

Some of packages, which I found:

code2html - Syntax highlighter
src2tex - A converter from source program files to TeX format files
highlight - An universal source code to formatted text converter
txt2html - Text to HTML converter
src2tex - A converter from source program files to TeX format files
c2html - Highlight C sources for WWW presentation
source-highlight - convert source code to syntax highlighted document
webcpp - configurable utility to convert source code to HTML

Example:

$ source-highlight -n -d -o sample.html -s python -i sample.py


01: #!/usr/bin/python 
02:
03: import sys, re
04:
05: class HelloWorld:
06:     def helloWorld(self):
07:         print "hallo world :)"
08:
09:
10: if __name__ == "__main__":
11:     helloWorld=HelloWorld()
12:     helloWorld.helloWorld()

Bash commands statistics

2009-11-04T16:26:00.001Z

Looking for some MySQL & performance stuff is found on the page from Alexey N. Kovyrin something interested :).

To see, what are you doing on our Linux box, type the following code on bash console:

$ history 1000 | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head

150 ls
30 for
26 du
20 tar
19 java
19 strace
...

autodiag too: how to generate uml diagram for perl code

2009-11-04T16:24:00.000Z

Howto generate xml "class" diagram for perl code.

$ autodia.pl -z -r -d printshop/

getting pattern for perl using directory : printshop/ recursively searching files.. AutoDia - version 2.00(c) Copyright 2003 A Trevena getting handlers.. using language : perl ..using Autodia::Handler::Perl opening printshop/monitor.pl opening printshop/checkdata.pl opening printshop/modules/knownfile.pm opening printshop/modules/common.pm opening printshop/modules/logs.pm opening printshop/modules/processfiles.pm opening printshop/modules/processfiles_phase2.pm opening printshop/modules/processprotocols.pm opening printshop/modules/mylock.pm opening printshop/modules/monitordb.pm opening printshop/modules/filetypes/standardbriefe.pm opening printshop/modules/filetypes/csv.pm opening printshop/modules/filetypes/hermespayrem.pm opening printshop/modules/filetypes/brandmobile_invoice.pm opening printshop/modules/filetypes/asnstandardbriefe.pm opening printshop/modules/filetypes/otherspayrem.pm opening printshop/modules/filetypes/xml.pm opening printshop/modules/filetypes/hermes.pm opening printshop/modules/filetypes/brandmobile_pr.pm opening printshop/modules/filetypes/invoices.pm opening printshop/modules/filetypes/acnpayrem.pm opening printshop/modules/protocoltypes/hermesmanung.pm opening printshop/modules/protocoltypes/hermesrechnung.pm opening printshop/modules/protocoltypes/brandmobile_telefactory.pm opening printshop/modules/protocoltypes/acoreuspayrem.pm opening printshop/modules/protocoltypes/skeleton.pm opening printshop/modules/protocoltypes/acnmahnung.pm opening printshop/modules/protocoltypes/invoices.pm Diagram.pm : add_class : ignoring duplicate classprotocoltypes::acoreuspayrem opening printshop/modules/protocoltypes/itelaobit.pm opening printshop/modules/protocoltypes/acnrechnung.pm Diagram.pm : Inheritances : no Inheritances to be printed - ignoring.. written outfile : autodia.out.xml successfully complete. (processed 30 files)

The diagram can looks like:

Modules with out function:

Modules with function:

Festplatte auf Fehler überprüfen

2009-11-04T16:23:00.000Z

Problem: Festplatte auf Fehler überprüfen

Problemlösung:

Linuxdistribution mithilfe einer Live CD booten.
Root werden
apt-get install smartmontools
smartctl -s on
smartctl -a -d ATA /dev/PLATTE
die Anzahl der Relocatet Sectors notieren
badblocks -w /dev/PLATTE
smartctl -a -d ATA /dev/PLATTE
Anzahl der Relocatet Sectors vergleichen

any good solution for undelete for ext3 :(

2009-11-04T16:22:00.000Z

http://jblevins.org/computing/linux/ext3-undelete

...
So, to recover plain text data one can simply do a brute force grep of the
filesystem to find a unique string in the deleted file. My Fortran files all
contained a unique string, the name of a module I wrote called
“auxpf_filter.” Thus, using the following command I was able to eventually
find the files I needed.

grep --binary-files=text -300 "auxpf_filter" /dev/hda4 > output.txt

dieM4tr1x-Trilogie von bitchhacker

2009-11-04T16:20:00.000Z

Eine echte wahre Geschichte aus dem Imternet : dieM4tr1x-Trilogie

Zitat:
<Elch> Du kennst dich ja voll mit Rechner aus.
<bitchchecker> halts maul ich hack dich
<Elch> ok, ich bin ja schon ruhig, nicht dass du uns zeigst was für ein toller Hacker du bist ^^
<bitchchecker> sag mir deine netzwerk nummer man dann bist du tot
<Elch> öhm die ist 129.0.0.1
<Elch> oder war es 127.0.0.1
<Elch> ja genau die war es: 127.0.0.1 Ich warte dann mal auf einen dollen Hackerangriff
<bitchchecker> in fünf minuten ist deine fest platte gelöscht
<Elch> Da habe ich jetzt aber Angst
<bitchchecker> halts maul du bist gleich weg
<bitchchecker> ich hab hier ein program da gebe ich deine ip ein und du bist tot
<bitchchecker> sag schon mal auf wieder sehen
<Elch> zu wem?
<bitchchecker> zu dir mann
<bitchchecker> buy buy
<Elch> Ich zitter schon förmlich vor einem so krassen Hack0r wie du einer bist
* bitchchecker (~java@euirc-61a2169c.dip.t-dialin.net) Quit (Ping timeout#)

Die komplete Geschichte ist unten zu finden:
http://www.stophiphop.com/modules/news/article.php?storyid=184

db2daig log

2009-11-04T16:19:00.000Z

The internet is great, some very interested post (all info is at post ):

Hi,

some developers seem to have a good sense of humor. The following
was found after (stupid enough) restoring a DB created with FP9a on a
FP8 test-system (DIAGLEVEL=4):

2006-03-17-13.13.47.909513+060 I54072G437 LEVEL: Info
PID : 22929 TID : 1024 PROC : db2agent (TEST)
INSTANCE: db2inst1 NODE : 000 DB : TEST
APPHDL : 0-300 APPID: XXXXXXXX.XXXX.XXXXXXXXXXXX
FUNCTION: DB2 UDB, access plan manager, sqlra_compile_var, probe:20
RETCODE : ZRC=0x80310004=-2144272380=SQLNN_E_NFOUND
"did not find specified thing in whatever"

Not too bad. It finally led me to the initial mistake AND gave me a
refreshing laugh. Thank you guys! ;-)

tcb,
Norbert