Enable Linux Core Dump

Enable Linux Core Dumps One of our applications (Freeswitch) just randomly crashed for no apparent reason and didn’t write anything to it’s log files. The service we’re trialling is currently in Beta so there’s room to muck about and do some diagnostics. I want to make the kernel dump a core file whenever Freeswitch dies, in case it happens again, so that we have some stuff to work with after the fact. It’ll also shut up my QA manager.

Check The Current Linux Core Dump Limits

ulimit is used to specify the maximum size of generated coredumps, this is to stop apps chewing up a million GB of RAM and then blowing your disk up, by default it’s a 0, which means nothing gets written to disk and no dump is created!

hstaging:~ # ulimit -c
0

Change The Linux Core Dump Limits To Something Awesome

To set the size limit of the linux core files to 75000 bytes, you can do something like this

hstaging:~ # ulimit -c 75000
hstaging:~ # ulimit -c
75000

but I’m a maverick, this does exactly what you think it does

hstaging:~ # ulimit -c unlimited
hstaging:~ # ulimit -c
unlimited

Enable Linux Core Dump For Application Crashes And Segfaults And Things

Ok, so we want this to persist across reboots so that basically means we have to stick the ulimit command in /etc/profile, i’m putting this at the bottom of mine:

#corefile stuff
ulimit -c unlimited > /dev/null 2>&1

this will stop anything weird getting spat out to the screen and nicely tells us that it’s core file stuff.

For our next trick we’ll set some sysctl flags so in /etc/sysctl.conf add

#corefile stuff
kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t
fs.suid_dumpable = 2

this basically says when the application crashes create a coredump file in /tmp with a useful name pattern

kernel.core_uses_pid = 1 - add the pid of the crashed app to the filename.
fs.suid_dumpable = 2 - enable linux core dumps for setuid processes.
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t - crazy naming pattern for a successful core dump, here's roughly what all the bits mean:
%e - executable filename
%s - number of signal causing dump
%u - real UID of dumped process
%g - real GID of dumped process
%p - PID of dumped process
%t - time of dump (seconds since 0:00h, 1 Jan 1970)

super usefuls. Then run sysctl -p so it takes effect yo!

hstaging:~ # sysctl -p
<snip>
kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t
fs.suid_dumpable = 2

Enabling Linux Core Dump For All Apps

Now here’s the last part. When you want an application to core dump you create an environment variable, before you start it, telling the kernel to sort itself out and get ready to dump, if you want all apps on the server to generate core dumps then you’re going to want to specify this variable somewhere near the top of the process chain. The best place for this on a redhat style box is /etc/sysconfig/init, so stick the following in that file

DAEMON_COREFILE_LIMIT='unlimited'

now might be an idea to reboot to force it to be set across all applications and things

Enabling Linux Core Dumps For A Specific Application

This is the slightly less rebooty version of the above. Rather than force the environment variable to be loaded when the box starts, we just stick it in the init script for the deamon, and then restart the daemon.

In /etc/init.d/functions the RedHat guys have already stuck in

corelimit="ulimit -S -c ${DAEMON_COREFILE_LIMIT:-0}"

So we need to make sure we put our DEAMON_COREFILE_LIMIT above that. Simples. In our case it’s in /etc/init.d/freeswitch with

DAEMON_COREFILE_LIMIT='unlimited'

Distros That Aren’t RedHat

DAEMON_COREFILE_LIMIT is a RedHatism. If you’re running something cool, like Ubuntu, you’ll want to use

ulimit -c unlimited >/dev/null 2>&1
echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern

instead.

Testing Core Dumps

This is EASY, we just start the deamon, send a segfault signal, look in the right place!!

hstaging:tmp # /etc/init.d/freewitch start
hstaging:tmp # /etc/init.d/freeswitch status
freeswitch (pid 8257) is running...
hstaging:tmp # kill -s SIGSEGV 8257
hstaging:tmp # ls /tmp/core*
core-freeswitch-11-493-492-8257-1371823178

Now you give this file to your developers and take a bow!

About rus

Hi, I'm Rus and after 10 years of Linux administration and software development, in September 2014 I quit my day job, sold all my stuff and moved to Thailand to live the good life.My new website, http://www.ruspow.com shows how I'm making money online as well as what life can be like when you're not sat behind a desk and are free to make your own decisions!

,

banner