Gdb script to spew malloc backtraces

We've got an app where "perf top" is showing malloc being called a lot. Now we need to figure out who is calling it.

Couple of ideas are obviously to use tcmalloc or something like this:

http://man7.org/linux/man-pages/man3/malloc_hook.3.html

However these are live game instances running on a server somewhere in Germany (for example), and it's not easy to restart and preload DSOs. People playing on that server tend to frown upon this behavior also. :)

Came up with the idea to try using gdb to connect, set a breakpoint and log the next X hundred of the callstacks to a logfile. Script is down below. I've tried it on glxspheres and it seems like it'll work. I'm sure others have done this - if so and you have any comments / suggestions, please let me know.

#
# Use via something like this:
#    gdb -x ~/bin/spewmallocscript.gdb --pid=$(pidof glxspheres64)
#

# Make sure we never get asked y/n
set pagination off

# Overwrite logfile
set logging overwrite on
set logging file /tmp/gdb_malloc_calls.txt
set logging on

printf "\n\nSetting malloc breakpoint\n"
break malloc

set $i=0
set $count=100

printf "Logging %d callstacks...\n\n", $count

# add command for malloc breakpoint
commands
silent
bt 16

if $i < $count
  printf "\ncall #%d:\n", $i
  set $i=$i+1
else
  set logging redirect off
  quit
end

c
end

# send output to log file only
set logging redirect on
c

Man (and less) colors...

I had the following set in my environment.

export LESS='-QRS'
export LESS_TERMCAP_mb=$'\E[01;31m'
export LESS_TERMCAP_md=$'\E[01;38;5;74m'
export LESS_TERMCAP_me=$'\E[0m'
export LESS_TERMCAP_so="$(printf 'rev\nbold\nsetaf 3\n' | tput -S)"
export LESS_TERMCAP_se="$(tput sgr0)"
export LESS_TERMCAP_us=$'\E[04;38;5;146m'
export LESS_TERMCAP_ue=$'\E[0m'

Made for nice colored man pages. Unfortunately, when I used the "env" command, environment variables following the LESS_TERMCAP_* variables got mucked up: underlined, etc.

So I unset them and now use this alias:

alias man="LESS_TERMCAP_mb=$'\E[01;31m' LESS_TERMCAP_md=$'\E[01;38;5;74m' LESS_TERMCAP_me=$'\E[0m' LESS_TERMCAP_se=$'\E[0m' LESS_TERMCAP_so=$'\E[48;5;3
m\E[38;5;0m' LESS_TERMCAP_ue=$'\E[0m' LESS_TERMCAP_us=$'\E[04;38;5;146m' man"

alias less="LESS_TERMCAP_mb=$'\E[01;31m' LESS_TERMCAP_md=$'\E[01;38;5;74m' LESS_TERMCAP_me=$'\E[0m' LESS_TERMCAP_se=$'\E[0m' LESS_TERMCAP_so=$'\E[48;5;3
m\E[38;5;0m' LESS_TERMCAP_ue=$'\E[0m' LESS_TERMCAP_us=$'\E[04;38;5;146m' less"

Not sure it's the best solution, but it seemed better than the lesskey stuff. Also found the prompt option and added that:

export LESS='-QRS --prompt="%f (%pb\%, %lmth line, %L lines)$ '

The final fairly useful thing from poking in the code is the LESS_TERMCAP_DEBUG environment variable. It highlights the section names which is pretty darn useful for figuring the colors out. Example:

mikesart@mikesart-petra:~/dev$ LESS_TERMCAP_DEBUG=1 man strstr

<ti><ks><cr>STRSTR(3) Linux Programmer's Manual

<md>NAME<me>

       strstr, strcasestr - locate a substring

<md>SYNOPSIS<me>

       <md>#include<me> <md><string.h><me>

... strstr(3) line 1/43 (END) (press h for help or q to quit)<se><ce>