Search This Blog

Thursday, May 19, 2011

Screen trick how to simultaneously monitor network traffic and disk activity and processes in Linux

As a system administrator sooner than later you will discover the 'screen' command. It is very powerful tool that allows you practically do anything with your (black) terminal screen or run commands in background for you.

The tools we will use for monitoring:
- htop is a top like process monitoring tool for text consoles
- nethogs is a network monitoring tool (needs root privileges!)
- iotop is a disk I/O monitoring tool

To start a single screen session, divided in 3 vertical smaller screens, each running its own monitoring program try this recipe:

# cat <<END >/tmp/myscreen.cfg
startup_message off
screen -t top htop
split
focus
screen -t nethogs  nethogs wlan0
split
focus
screen -t iotop iotop
END

# screen -c /tmp/myscreen.cfg

References:
Top 10 Awesome Linux Screen Tricks

Wednesday, May 4, 2011

By using Ubuntu how to create a bootable USB thumb drive with Fedora distro

Although this it has already been possible to use USB thumb drive and boot from it for a longer time, there is still no one unified method to create this USB thumb drive.

Ubuntu has its own method:

"2) Burn your CD or create a USB drive"
http://www.ubuntu.com/download/server/download

Fedora has its own as well:

"3.2.1. Making Fedora USB Media on a Windows Operating System"
http://docs.fedoraproject.org/en-US/Fedora/14/html/Installation_Guide/Making_USB_Media.html

Fortunately for us a separate project unetbootin [1] comes in handy to do this job regardless what OS you are currently using.

To simply install and run follow this instructions bellow.

# aptitude install unetbootin 
# dpkg -L  unetbootin | grep bin
/usr/bin/unetbootin
# unetbootin


References:

[1]
UNetbootin - Homepage and Downloads

Using Fedora's liveusb-creator on Ubuntu Lucid Lynx

Sunday, May 1, 2011

Gdisk error message: Caution: invalid main GPT header, but valid backup; regenerating main header from backup!

At some point when experimenting with partitions on your disk you may get the following error message.

% gdisk /dev/sda
GPT fdisk (gdisk) version 0.7.1

Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!

Caution! After loading partitions, the CRC doesn't check out!
Warning! Main partition table CRC mismatch! Loaded backup partition table
instead of main partition table!

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: damaged

Found valid MBR and corrupt GPT. Which do you want to use? (Using the
GPT MAY permit recovery of GPT data.)
 1 - MBR
 2 - GPT
 3 - Create blank GPT

As always you have to be very careful depending what you did and what do you want to do as next.

The three options for gdisk gives you the choice what partition tables you will examine. Fortunately for us gpart is not going to write on disk anything until we say so by executing the w (write) command (example bellow). The three options only influences how the data on the disk are going to be interpreted and presented for our review.

# gdisk /dev/sda
Command (? for help): h
b    back up GPT data to a file
c    change a partition's name
d    delete a partition
i    show detailed information on a partition
l    list known partition types
n    add a new partition
o    create a new empty GUID partition table (GPT)
p    print the partition table
q    quit without saving changes
r    recovery and transformation options (experts only)
s    sort partitions
t    change a partition's type code
v    verify disk
w    write table to disk and exit
x    extra functionality (experts only)
?    print this menu

At this stage it is save to experiment and have a look how the partition tables look like. Depending on your choise different partitions may be printed ('p') .

REMEMBER to always quit the session with the 'q' (quit) and never with 'w' otherwise your experiments will be permanently saved on the disk.

In my case this error was misleading and confusing only. I could verify that my current partition schema was not GPT at all but the good old MSDOS one [1]. At this stage I knew that 'gdisk' is not the tool I wanted to use and I finished creating new partitions with 'parted'. I didn't have to use gdisk at all.

The reason why I had some parts of GPT data on my disk is unknown. I can only suspect that it was created when I played with the Windows tool 'EasyBCD'.

I could try to delete the GPT data but base on [2] never had to do it.

References
[1] How to find what type of disk partition schema do I use (msdos, gpt)

[2] Wiping Out Old GPT Data

How to find what type of disk partition schema do I use (msdos, gpt)

There are many disk partition schemes that can be used. Very likely on most x86 base computer the choice is often limited by the operating system we use although.

The list of supported labels (partition schemes) in GNU parted tool can be found here [1]:


label-type must be one of these supported disk labels:

* bsd
* loop (raw disk access)
* gpt
* mac
* msdos
* pc98
* sun


To find out what type of partition schema we currently have on your system please run the bellow command and check the value for the 'Partition Table'.

% parted /dev/sda unit mb print
Model: ATA ST2000DL003-9VT1 (scsi)
Disk /dev/sda: 2000399MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start     End       Size      Type     File system  Flags
 1      0.03MB    104856MB  104856MB  primary  ntfs         boot
 2      104857MB  209714MB  104858MB  primary  ntfs
 3      209714MB  312115MB  102401MB  primary
                                      
root@sysresccd /tmp % parted /dev/sda unit s print
Model: ATA ST2000DL003-9VT1 (scsi)
Disk /dev/sda: 3907029168s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start       End         Size        Type     File system  Flags
 1      63s         204796619s  204796557s  primary  ntfs         boot
 2      204797952s  409597951s  204800000s  primary  ntfs
 3      409597952s  609599487s  200001536s  primary


References:

[1]
(parted) mklabel msdos

Others:

Parted User's Manual

GUID Partition Table

Make the most of large drives with GPT and Linux

Fun with GPT partitioning

Linux Creating a Partition Size Larger than 2TB

How to export/backup partition schema for a disk using old style disk partitions format like 'msdos' written into MBR sector

To backup the partition schema of the disk you can execute:

# sfdisk -d /dev/sda > part.txt

# example output can look like
# cat part.txt
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=       63, size=204796557, Id= 7, bootable
/dev/sda2 : start=204797952, size=204800000, Id= 7
/dev/sda3 : start=409597952, size=200001536, Id=83
/dev/sda4 : start=        0, size=        0, Id= 0

To restore the part.txt you can use:

# sfdisk /dev/hda < part.txt 
 
References:
man sfdisk 
Wiping Out Old GPT Data 

Howto capture and record the console screen output to a file on disk

The basic of capturing a data to a file require simple redirection or using of 'tee' program for example.

Examples:

$ ls -la > /var/tmp/output.ls.txt
$ ls -la | tee /var/tmp/output.ls.txt

But sometimes programs can be interactive or we simply want to capture all our session without to worry to redirect the stdout to a file.

The solution is to use the 'screen' tool.

Example #1: Screen basic usage

# to start the session with a name 'example' run
$screen -h
$screen -S example

# to leave the screen session type: CONTROL-a d
# you are placed back in the original shell

# to reattach to the created session
$ screen -ls
$ screen -r example

Example #1: Enable screen logging

The options '-L' instruct screen to create a log file that will capture all the commands output in the screen session.

$ screen -S logexample -L

# inside the screen session
$ echo 'some output'

# to leave the screen session type: CONTROL-a d

# this is the default log file for screen
$ ls -la screenlog.0

$ cat screenlog.0

Automatic debugging session of a programs in C and the gdb analyse of its core dumps

In our example we are going to concentrate on this little program bad_program to demonstrate our semi automated debug approach.

For debugging purposes we should first compile the program with debugging symbols. The possible ways are listed bellow and should be taken only as simple examples how to do this. More about this in [1].

# no debugging at all
 (0) $ gcc bad_program.c -o bad_program

# with debugging symbols  
 (1) $ gcc -g3     bad_program.c -o bad_program1
 (2) $ gcc -g      bad_program.c -rdynamic -o bad_program2
 (3) $ gcc -g  -O0 bad_program.c -rdynamic -o bad_program3
 (4) $ gcc -g3 -Os bad_program.c -o bad_program4

The test has been run on a following system.

$ gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.3-4ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-plugin --enable-objc-gc --enable-targets=all --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) 

$ gdb -v
GNU gdb (GDB) 7.1-ubuntu

The example program itself is written in such a way that it crushes and generates core dump each time when started.

Example 1

$ ./bad_program 1
Segmentation fault

To get the core dump file written on a disk that can be later analysed in gnu debugger (gdb) we need to first allow core dumps.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ ulimit -c unlimited 

When we run the program again it will create the core file for later analyse.

Example 2

$ ./bad_program 1
Segmentation fault (core dumped)

$ ls -la core 
-rw------- 1 radoslaw radoslaw 151552 2011-04-24 19:50 core

In our example we only want to run the gdb 'where' command, but if needed the file my_session.gdb.cmds bellow can be extended for any number of commands we may be interested in. More about the useful gdb debugging commands can be found in [2].

$ cat my_session.gdb.cmds 
where

Our test session can look like:

Debugging session #1

$ export BAD_PROGRAM=bad_program
$ for i in $(seq 1 4); do rm -f core; echo;  echo " ---- ---- ---- [$i] starting the program ---- ---- ----"; ./$BAD_PROGRAM $i ; echo " ---- ---- ---- [$i] starting gdb ---- ---- ----"; gdb -batch -x my_session.gdb.cmds -n $BAD_PROGRAM core ; done | tee ${BAD_PROGRAM}.log

 ---- ---- ---- [1] starting the program ---- ---- ----
 ---- ---- ---- [1] starting gdb ---- ---- ----

warning: Can't read pathname for load map: Input/output error.[New Thread 16934]

1051 ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S: No such file or directory.
Core was generated by `./bad_program 1'.
Program terminated with signal 11, Segmentation fault.
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
 in ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
#1  0x00e9073c in ?? () from /lib/ld-linux.so.2
#2  0x0804844d in generate_core ()
#3  0x08048566 in main ()

 ---- ---- ---- [2] starting the program ---- ---- ----
 ---- ---- ---- [2] starting gdb ---- ---- ----
[New Thread 16938]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 2'.
Program terminated with signal 11, Segmentation fault.
#0  0x0804852b in try_core2 ()
#0  0x0804852b in try_core2 ()
#1  0x0804848e in generate_core ()
#2  0x08048566 in main ()

 ---- ---- ---- [3] starting the program ---- ---- ----
 ---- ---- ---- [3] starting gdb ---- ---- ----

warning: [New Thread 16942]
Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 3'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 ()
#0  0x08048541 in try_core3 ()
#1  0x080484cf in generate_core ()
#2  0x08048566 in main ()

 ---- ---- ---- [4] starting the program ---- ---- ----
 ---- ---- ---- [4] starting gdb ---- ---- ----
[New Thread 16946]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 4'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 ()
#0  0x08048541 in try_core3 ()
#1  0x080484cf in generate_core ()
#2  0x080484eb in generate_core ()
#3  0x08048566 in main ()

We can still see enough to say where the problem happened more or less but we get much better results when a program is run with debugging symbols.

Debugging session #2

$ export BAD_PROGRAM=bad_program1
$ for i in $(seq 1 4); do rm -f core; echo;  echo " ---- ---- ---- [$i] starting the program ---- ---- ----"; ./$BAD_PROGRAM $i ; echo " ---- ---- ---- [$i] starting gdb ---- ---- ----"; gdb -batch -x my_session.gdb.cmds -n $BAD_PROGRAM core ; done | tee ${BAD_PROGRAM}.log

 ---- ---- ---- [1] starting the program ---- ---- ----
 ---- ---- ---- [1] starting gdb ---- ---- ----

warning: [New Thread 17974]
Can't read pathname for load map: Input/output error.
1051 ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S: No such file or directory.
Core was generated by `./bad_program1 1'.
Program terminated with signal 11, Segmentation fault.
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
 in ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
#1  0x00f6973c in ?? () from /lib/ld-linux.so.2
#2  0x0804844d in generate_core (n=1) at bad_program.c:15
#3  0x08048566 in main (argc=2, argv=0xbfb572a4) at bad_program.c:52

 ---- ---- ---- [2] starting the program ---- ---- ----
 ---- ---- ---- [2] starting gdb ---- ---- ----

warning: [New Thread 17978]
Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 2'.
Program terminated with signal 11, Segmentation fault.
#0  0x0804852b in try_core2 (n=2) at bad_program.c:41
41   *ptr=n;
#0  0x0804852b in try_core2 (n=2) at bad_program.c:41
#1  0x0804848e in generate_core (n=2) at bad_program.c:20
#2  0x08048566 in main (argc=2, argv=0xbff99024) at bad_program.c:52

 ---- ---- ---- [3] starting the program ---- ---- ----
 ---- ---- ---- [3] starting gdb ---- ---- ----
[New Thread 17982]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 3'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
47   *(ptr+n)=n; 
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
#1  0x080484cf in generate_core (n=3) at bad_program.c:25
#2  0x08048566 in main (argc=2, argv=0xbf92fdd4) at bad_program.c:52

 ---- ---- ---- [4] starting the program ---- ---- ----
 ---- ---- ---- [4] starting gdb ---- ---- ----
[New Thread 17986]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 4'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
47   *(ptr+n)=n; 
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
#1  0x080484cf in generate_core (n=3) at bad_program.c:25
#2  0x080484eb in generate_core (n=4) at bad_program.c:29
#3  0x08048566 in main (argc=2, argv=0xbfdd2994) at bad_program.c:52

We can see in lines # 16, 27, 40, 53 the instructions that caused the core dump. We see as well the full arguments in functions what helps to better understand the program logic flow. One thing more as well to notice is the difference in debugging output when analysing the core from:

./bad_program  1   # no debugging symbols; versus
./bad_program1 1   # with debugging symbols

From the debugging session #1 we can hardly guess where the problem was, where in debugging session #2 we clearly see that the problem started with line 'bad_program.c:15'.

Example program

The "bad" example program that cores every time when run is bellow. More info about this in [3].

Source code of the bad_program.c 

#include 
#include 
#include 
#include 
#include 

void try_core1(int n);
void try_core2(int n);
void try_core3(int n);
void generate_core(int n);

void generate_core( int n ) {
  if ( 1 == (n%10) ) { 
    try_core1(n);
    generate_core(n-1);
  } 
  
  if ( 2 == (n%10) ) { 
    try_core2(n);
    generate_core(n-1);
  } 
  
  if ( 3 == (n%10) ) { 
    try_core3(n);
    generate_core(n-1);
  } 
  
  generate_core(n-1);
}

void try_core1( int n ) {
  char *ptr=NULL;
  
  strcpy(ptr, "this is going to hurt ;)...");
}

void try_core2( int n ) {
  int *ptr=NULL;
  
  *ptr=n;
}

void try_core3( int n ) {
  int *ptr;
  
  *(ptr+n)=n; 
}


int main(int argc, char **argv) {
  generate_core( atoi( argv[1] ) );
}

References:
[1]
HowTo: Debug Crashed Linux Application Core Files Like A Pro
Debugging with GDB

[2]
Mastering Linux debugging techniques
Linux software debugging with GDB
GDB: The GNU Project Debugger

[3]
How to programatically cause a core dump in C/C++