Scripting GDB with Python

Every tutorial on how to use Python to control GDB (the GNU Debugger) starts with a line like "say you have hundreds of threads."

Well, I have hundreds of threads. But I also have hundreds of socket objects in the C++ program I'm looking at! I can't really afford to examine them one at a time and get any insight. I'm not even sure what I'm looking for in the deadlock situation I'm investigating. Python's a good language for this sort of exploratory data analysis. (I have not yet checked whether I can get pandas and matplotlib to work within GDB--- an adventure for another time.)

The documentation on using Python inside GDB can be found here: https://sourceware.org/gdb/onlinedocs/gdb/Python.html

You type python, enter some Python code, finish with end, and it gets executed by a Python interpreter embedded in the GDB. Just in case you have a burning need to do some bignum arithmetic:

(gdb) python
>print pow( 2, 104 )
>end
20282409603651670423947251286016

The gdb package provided by GDB contains an enormous breadth of functionality. You can define new commands, access types and values, decorate stack frames, even replace an existing C++ method.

Inside Python, you can call gdb.parse_and_eval to use GDB's ability to parse C++ syntax and evaluate it. This is usually the starting point--- "print the value of some variable." But what you get back is actually an object, and most of what I need is in that Values object. This does magic so that fields in an object show up as dictionary entries in the resulting Python object.

(gdb) python
>foo = gdb.parse_and_eval( "this->mutex_" )
>print foo['cleanupPolicy_']
>end
MutexBasic::Destroy

It turns out this doesn't always work the way you'd expect. While the documentation claims that types that can be converted to Python basic types will be, I experienced some very strange behavior trying to call max() and do subtraction on 64-bit values. I worked around this by explicitly converting to long.

Also. C++ strings failed to convert to Python strings:

(gdb) python
>foo = gdb.parse_and_eval( "*('Framework::PthreadThread' *)(this->thread_.ptr_)" )
>bar = foo['name_'].string()
>print bar[7:]
>end
Traceback (most recent call last):
  File "<string>", line 2, in <module>
gdb.error: Trying to read string with inappropriate type `String'.
Error while executing Python code.

This might just be a problem with the rather old compiler and debugger combination I happen to be using. A workaround is to cast to a char * (which happens to work with the way the GCC standard library implements std::basic_string.)

(gdb) python
>char_ptr = gdb.lookup_type( 'char' ).pointer()
>foo = gdb.parse_and_eval( "*('Framework::PthreadThread' *)(this->thread_.ptr_)" )
>bar = foo['name_'].cast( char_ptr ).string()
>print bar[7:]
>end
/GSReceiveThread

But, one thing that does work is incrementing a pointer by 1 to move to the next object; the Python wrapper overloads addition so that the "right thing" happens.

Here's the script I ended up using to walk an array of threads and an array of sockets in each thread, and print a few key stats neatly.

def sendSockets():
    sockets = [] 
    maxTime = 0
    
    char_ptr = gdb.lookup_type( 'char' ).pointer()
    ggss = gdb.parse_and_eval( 'GlobalGenericSocketState' )
    for i in xrange( 0, ggss['dataSendEpollThreadCount_'] ):
        thread = (ggss['dataSendEpollThreadArray_'] + i).dereference()
        for j in xrange( 0, thread['socketCount_'] ):
            socket = ( thread['socketArray_'] + j ).dereference().dereference()
            client = socket['client_'].cast( char_ptr ).string()
            sockets.append( (client, socket) )
            maxTime = max( maxTime,
                           long( socket['lastEpollWaitRecvNs_'] ),
                           long( socket['lastEpollWaitSendNs_'] ) )

    sockets.sort( key=lambda x:x[0] )
    print len( sockets), "sockets"
    
    for ( c, socket ) in sockets:
            print "{0:20} {1:4} {2:15} lastWaitRecv {3:7} lastWaitSend {4:7}".format(
                c, socket['fd_'], socket['socketStats_']['sendBytes_'],
                ( long( socket['lastEpollWaitRecvNs_'] ) - maxTime ) / 1000000000, 
                ( long( socket['lastEpollWaitSendNs_'] ) - maxTime ) / 1000000000 )