How to use gdb to debug postgres

Sometimes we need to use a debugger on on postgres to see what is happening. The purpose of this article is to show how to use debugger (gdb) to get a stack trace of a running Postgres process

If PostgreSQL was installed from packages, the debuginfo packages must be installed. See the linked related article "installing debuginfo packages". Debuginfo is important, otherwise the results are much less useful.

You will also need the package gdb installed.

Then gdb can be run with the following command to examine the stack of the $pid of the process in question:

sudo gdb -p ${pid} -ex 'set pagination off' -ex 'set logging file gdb.txt' -ex 'set logging on' -ex 'set confirm off' -ex 'bt' -ex 'bt full'--eval-command='shell sleep 5' -ex 'bt full' --eval-command='shell sleep 5' -ex 'bt full' -ex 'info threads' -ex 'quit'

This captures the stack information into the file gdb.txt, taking three samples over a short period.

Please attach gdb.txt to a ticket to stop email clients from mangling the formatting.

Check the resulting file. The output should show details about source file names (.c files) and line numbers, should show function arguments, etc. Like this:

#8 PostgresMain (argc=<optimized out>, argv=argv@entry=0x13644d8, dbname=<optimized out>, username=<optimized out>) at postgres.c:4003
#9 0x000000000046cdd6 in BackendRun (port=0x138e300) at postmaster.c:4312
#10 BackendStartup (port=0x138e300) at postmaster.c:3986
#11 ServerLoop () at postmaster.c:1705
#12 0x00000000006673c9 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x13635d0) at postmaster.c:1313
#13 0x000000000046e4f8 in main (argc=3, argv=0x13635d0) at main.c:228

If you instead see brief output that looks like this:

...
#5 0x00000000006bfb60 in PostgresMain ()
#6 0x000000000046cdd6 in ServerLoop ()
#7 0x00000000006673c9 in PostmasterMain ()
#8 0x000000000046e4f8 in main ()

then your debug symbol information is not present, so please install debug info packages and try again. (If you built from source, that's also covered in the debug info article).

In some specific situations, for example to diagnose memory leaks in PostgreSQL processes, the following command will help collecting memory context statistics useful for that purpose:

sudo gdb -p ${pid} \
        -ex 'set unwindonsignal on' \
        -ex 'set pagination off' \
        -ex 'call MemoryContextStats(TopMemoryContext)' \
        -ex 'detach' \
        -ex 'quit' >> /tmp/gdb_library_information.txt

This gdb command will not produce a stack trace, but will capture the memory dump in the postgres logs. Tail the log before executing the gdb command to confirm that data is correctly being fed into the logs, e.g:

tail -f postgresql.log | grep -E "[0-9]{1,} used"

TopMemoryContext: 68704 total in 5 blocks; 11736 free (15 chunks); 56968 used
pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1456 free (0 chunks); 6736 used
TopTransactionContext: 8192 total in 1 blocks; 7744 free (1 chunks); 448 used
TableSpace cache: 8192 total in 1 blocks; 2096 free (0 chunks); 6096 used
Type information cache: 24352 total in 2 blocks; 2624 free (0 chunks); 21728 used
...
PrivateRefCount: 8192 total in 1 blocks; 2624 free (0 chunks); 5568 used
MdSmgr: 8192 total in 1 blocks; 7696 free (0 chunks); 496 used
LOCALLOCK hash: 8192 total in 1 blocks; 560 free (0 chunks); 7632 used
Timezones: 104120 total in 2 blocks; 2624 free (0 chunks); 101496 used
ErrorContext: 8192 total in 1 blocks; 7936 free (0 chunks); 256 used
Grand total: 1159328 bytes in 178 blocks; 235976 free (120 chunks); 923352 used

Please attach the full postgres log(s) and the /tmp/gdb_library_information.txt file.