Automatic debugging session of a programs in C and the gdb analyse of its core dumps

In our example we are going to concentrate on this little program bad_program to demonstrate our semi automated debug approach.

For debugging purposes we should first compile the program with debugging symbols. The possible ways are listed bellow and should be taken only as simple examples how to do this. More about this in [1].

# no debugging at all
 (0) $ gcc bad_program.c -o bad_program

# with debugging symbols  
 (1) $ gcc -g3     bad_program.c -o bad_program1
 (2) $ gcc -g      bad_program.c -rdynamic -o bad_program2
 (3) $ gcc -g  -O0 bad_program.c -rdynamic -o bad_program3
 (4) $ gcc -g3 -Os bad_program.c -o bad_program4

The test has been run on a following system.

$ gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.3-4ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-plugin --enable-objc-gc --enable-targets=all --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) 

$ gdb -v
GNU gdb (GDB) 7.1-ubuntu

The example program itself is written in such a way that it crushes and generates core dump each time when started.

Example 1

$ ./bad_program 1
Segmentation fault

To get the core dump file written on a disk that can be later analysed in gnu debugger (gdb) we need to first allow core dumps.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ ulimit -c unlimited 

When we run the program again it will create the core file for later analyse.

Example 2

$ ./bad_program 1
Segmentation fault (core dumped)

$ ls -la core 
-rw------- 1 radoslaw radoslaw 151552 2011-04-24 19:50 core

In our example we only want to run the gdb 'where' command, but if needed the file my_session.gdb.cmds bellow can be extended for any number of commands we may be interested in. More about the useful gdb debugging commands can be found in [2].

$ cat my_session.gdb.cmds 
where

Our test session can look like:

Debugging session #1

$ export BAD_PROGRAM=bad_program
$ for i in $(seq 1 4); do rm -f core; echo;  echo " ---- ---- ---- [$i] starting the program ---- ---- ----"; ./$BAD_PROGRAM $i ; echo " ---- ---- ---- [$i] starting gdb ---- ---- ----"; gdb -batch -x my_session.gdb.cmds -n $BAD_PROGRAM core ; done | tee ${BAD_PROGRAM}.log

 ---- ---- ---- [1] starting the program ---- ---- ----
 ---- ---- ---- [1] starting gdb ---- ---- ----

warning: Can't read pathname for load map: Input/output error.[New Thread 16934]

1051 ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S: No such file or directory.
Core was generated by `./bad_program 1'.
Program terminated with signal 11, Segmentation fault.
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
 in ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
#1  0x00e9073c in ?? () from /lib/ld-linux.so.2
#2  0x0804844d in generate_core ()
#3  0x08048566 in main ()

 ---- ---- ---- [2] starting the program ---- ---- ----
 ---- ---- ---- [2] starting gdb ---- ---- ----
[New Thread 16938]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 2'.
Program terminated with signal 11, Segmentation fault.
#0  0x0804852b in try_core2 ()
#0  0x0804852b in try_core2 ()
#1  0x0804848e in generate_core ()
#2  0x08048566 in main ()

 ---- ---- ---- [3] starting the program ---- ---- ----
 ---- ---- ---- [3] starting gdb ---- ---- ----

warning: [New Thread 16942]
Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 3'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 ()
#0  0x08048541 in try_core3 ()
#1  0x080484cf in generate_core ()
#2  0x08048566 in main ()

 ---- ---- ---- [4] starting the program ---- ---- ----
 ---- ---- ---- [4] starting gdb ---- ---- ----
[New Thread 16946]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program 4'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 ()
#0  0x08048541 in try_core3 ()
#1  0x080484cf in generate_core ()
#2  0x080484eb in generate_core ()
#3  0x08048566 in main ()

We can still see enough to say where the problem happened more or less but we get much better results when a program is run with debugging symbols.

Debugging session #2

$ export BAD_PROGRAM=bad_program1
$ for i in $(seq 1 4); do rm -f core; echo;  echo " ---- ---- ---- [$i] starting the program ---- ---- ----"; ./$BAD_PROGRAM $i ; echo " ---- ---- ---- [$i] starting gdb ---- ---- ----"; gdb -batch -x my_session.gdb.cmds -n $BAD_PROGRAM core ; done | tee ${BAD_PROGRAM}.log

 ---- ---- ---- [1] starting the program ---- ---- ----
 ---- ---- ---- [1] starting gdb ---- ---- ----

warning: [New Thread 17974]
Can't read pathname for load map: Input/output error.
1051 ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S: No such file or directory.
Core was generated by `./bad_program1 1'.
Program terminated with signal 11, Segmentation fault.
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
 in ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S
#0  __memcpy_ssse3 () at ../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:1051
#1  0x00f6973c in ?? () from /lib/ld-linux.so.2
#2  0x0804844d in generate_core (n=1) at bad_program.c:15
#3  0x08048566 in main (argc=2, argv=0xbfb572a4) at bad_program.c:52

 ---- ---- ---- [2] starting the program ---- ---- ----
 ---- ---- ---- [2] starting gdb ---- ---- ----

warning: [New Thread 17978]
Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 2'.
Program terminated with signal 11, Segmentation fault.
#0  0x0804852b in try_core2 (n=2) at bad_program.c:41
41   *ptr=n;
#0  0x0804852b in try_core2 (n=2) at bad_program.c:41
#1  0x0804848e in generate_core (n=2) at bad_program.c:20
#2  0x08048566 in main (argc=2, argv=0xbff99024) at bad_program.c:52

 ---- ---- ---- [3] starting the program ---- ---- ----
 ---- ---- ---- [3] starting gdb ---- ---- ----
[New Thread 17982]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 3'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
47   *(ptr+n)=n; 
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
#1  0x080484cf in generate_core (n=3) at bad_program.c:25
#2  0x08048566 in main (argc=2, argv=0xbf92fdd4) at bad_program.c:52

 ---- ---- ---- [4] starting the program ---- ---- ----
 ---- ---- ---- [4] starting gdb ---- ---- ----
[New Thread 17986]

warning: Can't read pathname for load map: Input/output error.
Core was generated by `./bad_program1 4'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
47   *(ptr+n)=n; 
#0  0x08048541 in try_core3 (n=3) at bad_program.c:47
#1  0x080484cf in generate_core (n=3) at bad_program.c:25
#2  0x080484eb in generate_core (n=4) at bad_program.c:29
#3  0x08048566 in main (argc=2, argv=0xbfdd2994) at bad_program.c:52

We can see in lines # 16, 27, 40, 53 the instructions that caused the core dump. We see as well the full arguments in functions what helps to better understand the program logic flow. One thing more as well to notice is the difference in debugging output when analysing the core from:

./bad_program  1   # no debugging symbols; versus
./bad_program1 1   # with debugging symbols

From the debugging session #1 we can hardly guess where the problem was, where in debugging session #2 we clearly see that the problem started with line 'bad_program.c:15'.

Example program

The "bad" example program that cores every time when run is bellow. More info about this in [3].

Source code of the bad_program.c 

#include 
#include 
#include 
#include 
#include 

void try_core1(int n);
void try_core2(int n);
void try_core3(int n);
void generate_core(int n);

void generate_core( int n ) {
  if ( 1 == (n%10) ) { 
    try_core1(n);
    generate_core(n-1);
  } 
  
  if ( 2 == (n%10) ) { 
    try_core2(n);
    generate_core(n-1);
  } 
  
  if ( 3 == (n%10) ) { 
    try_core3(n);
    generate_core(n-1);
  } 
  
  generate_core(n-1);
}

void try_core1( int n ) {
  char *ptr=NULL;
  
  strcpy(ptr, "this is going to hurt ;)...");
}

void try_core2( int n ) {
  int *ptr=NULL;
  
  *ptr=n;
}

void try_core3( int n ) {
  int *ptr;
  
  *(ptr+n)=n; 
}


int main(int argc, char **argv) {
  generate_core( atoi( argv[1] ) );
}

References:
[1]
HowTo: Debug Crashed Linux Application Core Files Like A Pro
Debugging with GDB

[2]
Mastering Linux debugging techniques
Linux software debugging with GDB
GDB: The GNU Project Debugger

[3]
How to programatically cause a core dump in C/C++