Using GDB to explore a core file

Before we go any further, please note that these are instructions for a Centos 6.3 OS.  They should work in a similar way on Redhat and Fedora.

Recently, I've been experiencing issues with pacemaker immediately after ugprading from one version to another.  To make a long story short, crmd, which is part of the pacemaker package, was crashing every 15 minutes and dumping a core file.  It became necessary to read the details of the core dump using gdb.  To be fair though, the service itself stayed up fine but kept respawning child processes.  So, even though it complains and dumps a core file, it is still robust enough to continue operating.

Now the issue becomes, how to get the error content out of the core file?  We first need to make sure the right repositories are available for our OS:

# DEBUGINFO
[debuginfo]
name=debuginfo
baseurl=http://debuginfo.centos.org/$releasever/$basearch
enabled=0
gpgcheck=0

You will need a tool called 'debuginfo-install' which is part of a yum utility package.

# yum install yum-utils

Install gdb

# yum install gdb

Run gdb against your executable and it's core file.  For example:

# gdb /usr/libexec/pacemaker/crmd ./core.26688

If you are missing any debuginfo file, gdb will let you know which ones.  You can then install them as required.  For example:

Missing separate debuginfos, use: debuginfo-install audit-libs-2.2-2.el6.x86_64

Simply run the suggested command to get the right debuginfo files.

# debuginfo-install audit-libs-2.2-2.el6.x86_64 --enablerepo=debuginfo

Once you've got all your debug info files, run the gdb debugger once again.

# gdb /usr/libexec/pacemaker/crmd ./core.26688
...
Core was generated by `/usr/libexec/pacemaker/crmd'.
Program terminated with signal 6, Aborted.
#0  0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install libtool-ltdl-2.2.6-15.5.el6.x86_64
(gdb)

At this point you are in the gdb prompt.  Use "bt" to output the backtrace of the error.

(gdb) bt
#0  0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f81896ae085 in abort () at abort.c:92
#2  0x00007f818bb8a56b in crm_abort (file=0x7f818bba9d58 "xml.c", function=0x7f818bbab6b4 "string2xml", line=650,
    assert_condition=0x7f818bbaa01a "String parsing error", do_core=, do_fork=) at utils.c:1073
#3  0x00007f818bb933af in string2xml (
    input=0x1e745f8 "#4  0x00007f818b76a2fc in lrmd_ipc_dispatch (buffer=, length=, userdata=0x1e72910) at lrmd_client.c:310
#5  0x00007f818bba2e90 in mainloop_gio_callback (gio=, condition=G_IO_IN, data=0x1e73be0) at mainloop.c:585
#6  0x00007f8188fbbf0e in g_main_dispatch (context=0x1d4f120) at gmain.c:1960
#7  IA__g_main_context_dispatch (context=0x1d4f120) at gmain.c:2513
#8  0x00007f8188fbf938 in g_main_context_iterate (context=0x1d4f120, block=1, dispatch=1, self=) at gmain.c:2591
#9  0x00007f8188fbfd55 in IA__g_main_loop_run (loop=0x1e734a0) at gmain.c:2799
#10 0x00000000004052ce in crmd_init () at main.c:154
#11 0x00000000004055cc in main (argc=1, argv=0x7fffe77a4f88) at main.c:120
(gdb)

This is the information we were looking for.

Thanks to Andrew Beekhof of  http://clusterlabs.org/ for pointing me in the right direction with GDB.

Debugging JavaServer Faces

While attempting to test some session handling code for JavaServer Faces, I ran into a Java Null Pointer Exception.  The code causing the error was written directly into the jsp file itself instead of in a backing bean.  The cause of the error is still eluding me as I did not yet delve into it; however I discovered that debugging jsp files is not as hard as it seems.  Initially I was stumped because the output from glassfish stated that the exception error occurred on line 82:  My JSP file only has 61 lines.  Being experienced with .NET it's easy to understand why the error numbers would differ.  Normally in .NET one would use codebehind and the error's line number would match your code's line number.  Similarly the normal way to code in JavaServer Faces is to ensure all your logic code is in a backing bean;  then the error's line number matches with the code's line number.

In any case I wanted to know the location of the error in my code  (JSF generates a .java code file from your JSP file when it is compiled.)  The only trick was to find the location of this generated file and the line number from the error matches the line number in the .java file.

Here is a sample of the code from my JSP page and the "translation" from the .java file.  The location of the .Java file may vary, but mine was located at: /home/<username>/.netbeans/6.9/config/GF3_62/domain1/generated/jsp/various_tests/org/apache/jsp/index_jsp.java

JSP code:

21 22 23 24 25

System.out.println("reached point 1"); FacesContext ctx = FacesContext.getCurrentInstance(); HttpSession mySession = (HttpSession) ctx.getExternalContext().getSession(false); String s = mySession.getAttribute("test").toString(); System.out.println(s);

Java code:

80 81 82 83 84

System.out.println("reached point 1"); FacesContext ctx = FacesContext.getCurrentInstance(); HttpSession mySession = (HttpSession) ctx.getExternalContext().getSession(false); String s = mySession.getAttribute("test").toString(); System.out.println(s);

Line 82 is highlighted in red and it's equivalent in the JSP file is line 23.

IcedTea Java Console: How to debug Java Applets on Unix / Linux based systems

If you've ever used the IcedTea Java plugin, you may have noticed that it does not have a console like it's Windows based cousin.

How do you debug or see the Java output without a console? The answer is very simple: If you are using IcedTea it's because you are running a Unix / Linux based system. You have built-in consoles all around you!

THE QUICK SOLUTION: (for the impatient)

Before running your applet in a browser, open a terminal window and type:

$ watch -n 1 'cat $HOME/.icedteaplugin/java.stdout'

Now run your applet and see the output fill up the screen.

THE FULL EXPLANATION:

Open a terminal and go to your icedteaplugin personal settings folder. It should be named .icedteaplugin in your $HOME directory:

$ cd $HOME/.icedteaplugin

Do a directory listing and you will see a few files and a directory.

$ ls

cache
java.stdout
java.stderr

cache is the directory which contains java applets that have been accessed before. If you ever debug an applet you will know that sometimes it's useful to 'clear' the cache. You can individually remove the cached applet that you are debugging, to force IcedTea to download the latest version, by deleting the file.

java.stdout contains standard output from a running java applet, or if none is running it may contain data from the last running applet.

java.stderr contains standard error output.

Now that you know where all the data is, how can you see the output in real time?

Well, you can. At least you can see the output in near real time by using the useful Unix/Linux watch and cat commands.

If you are already in your $HOME/.icedteaplugin directory, type:

$ watch -n 1 'cat ./java.stdout'

If you open another terminal window, type:

$ watch -n 1 'cat $HOME/.icedteaplugin/java.stderr'

Use man watch and man cat for all the optional parameters, but note that the -n option for watch specifies the update rate in seconds.