glibtop: This machine has 56 CPUs, 32 are being monitored

Fully building TF2 dedicated server was taking around 13 minutes, 10 seconds on this:

Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
mikesart@mikesart-petra:~$ grep -c model.name /proc/cpuinfo
12

I tend to muck with the lower level libraries which meant I was spending quite a bit of my day waiting for these builds to finish.

TF2 dedicated server has:
 - 1347 cpp files
 - 13 DSOs (with 13 .dbg files)
 - 12 static libraries

I looked at several things: ccache, precompiled headers, distcc, unity builds. We are using ccache fulltime, but when I switch between debug / release or add a new define here and there, it wasn't helping as much as I'd like. Distcc meant I'd have to manage several other machines, and the unity builds seemed like a whole bunch of work also. The precompiled headers gave a decent win (down to around 7 minutes), but (from what I could tell), the .gch file really wants to be where the main header file lives. We do debug/release/client/dedicated server builds all out of the same tree and this just won't work.

So I got one of these:

(2) Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
mikesart@mikesart-mint:~/valvesrc/ValveGames/staging/game$ grep -c model.name /proc/cpuinfo
56

The build time is now 3 minutes, 30 seconds.

I tried creating a ramdisk and putting the entire source tree and tmp directory there, and the build time was still 3m30s (compared to building from a Samsung SSD 850 PRO 512GB). I think a decent chunk of time right now is going towards linking - so I'm going to investigate the gold linker and see if I can get a bit more time back from that.

We are currently using gcc 4.6.1 crosstools w/ glibc-2.11.1. I'm going to investigate newer versions of gcc and clang when I get more time to play with this stuff. Have some bugs and features before I get to play more - hopefully turnaround will go a bit faster now though...

Gdb script to spew malloc backtraces

We've got an app where "perf top" is showing malloc being called a lot. Now we need to figure out who is calling it.

Couple of ideas are obviously to use tcmalloc or something like this:

http://man7.org/linux/man-pages/man3/malloc_hook.3.html

However these are live game instances running on a server somewhere in Germany (for example), and it's not easy to restart and preload DSOs. People playing on that server tend to frown upon this behavior also. :)

Came up with the idea to try using gdb to connect, set a breakpoint and log the next X hundred of the callstacks to a logfile. Script is down below. I've tried it on glxspheres and it seems like it'll work. I'm sure others have done this - if so and you have any comments / suggestions, please let me know.

#
# Use via something like this:
#    gdb -x ~/bin/spewmallocscript.gdb --pid=$(pidof glxspheres64)
#

# Make sure we never get asked y/n
set pagination off

# Overwrite logfile
set logging overwrite on
set logging file /tmp/gdb_malloc_calls.txt
set logging on

printf "\n\nSetting malloc breakpoint\n"
break malloc

set $i=0
set $count=100

printf "Logging %d callstacks...\n\n", $count

# add command for malloc breakpoint
commands
silent
bt 16

if $i < $count
  printf "\ncall #%d:\n", $i
  set $i=$i+1
else
  set logging redirect off
  quit
end

c
end

# send output to log file only
set logging redirect on
c

Man (and less) colors...

I had the following set in my environment.

export LESS='-QRS'
export LESS_TERMCAP_mb=$'\E[01;31m'
export LESS_TERMCAP_md=$'\E[01;38;5;74m'
export LESS_TERMCAP_me=$'\E[0m'
export LESS_TERMCAP_so="$(printf 'rev\nbold\nsetaf 3\n' | tput -S)"
export LESS_TERMCAP_se="$(tput sgr0)"
export LESS_TERMCAP_us=$'\E[04;38;5;146m'
export LESS_TERMCAP_ue=$'\E[0m'

Made for nice colored man pages. Unfortunately, when I used the "env" command, environment variables following the LESS_TERMCAP_* variables got mucked up: underlined, etc.

So I unset them and now use this alias:

alias man="LESS_TERMCAP_mb=$'\E[01;31m' LESS_TERMCAP_md=$'\E[01;38;5;74m' LESS_TERMCAP_me=$'\E[0m' LESS_TERMCAP_se=$'\E[0m' LESS_TERMCAP_so=$'\E[48;5;3
m\E[38;5;0m' LESS_TERMCAP_ue=$'\E[0m' LESS_TERMCAP_us=$'\E[04;38;5;146m' man"

alias less="LESS_TERMCAP_mb=$'\E[01;31m' LESS_TERMCAP_md=$'\E[01;38;5;74m' LESS_TERMCAP_me=$'\E[0m' LESS_TERMCAP_se=$'\E[0m' LESS_TERMCAP_so=$'\E[48;5;3
m\E[38;5;0m' LESS_TERMCAP_ue=$'\E[0m' LESS_TERMCAP_us=$'\E[04;38;5;146m' less"

Not sure it's the best solution, but it seemed better than the lesskey stuff. Also found the prompt option and added that:

export LESS='-QRS --prompt="%f (%pb\%, %lmth line, %L lines)$ '

The final fairly useful thing from poking in the code is the LESS_TERMCAP_DEBUG environment variable. It highlights the section names which is pretty darn useful for figuring the colors out. Example:

mikesart@mikesart-petra:~/dev$ LESS_TERMCAP_DEBUG=1 man strstr

<ti><ks><cr>STRSTR(3) Linux Programmer's Manual

<md>NAME<me>

       strstr, strcasestr - locate a substring

<md>SYNOPSIS<me>

       <md>#include<me> <md><string.h><me>

... strstr(3) line 1/43 (END) (press h for help or q to quit)<se><ce>

32-bit Valgrind Notes

1. When running ./configure to build valgrind, look for this line:

    checking for 32 bit build support... yes

You can also use --enable-only64bit or --enable-only32bit if you only care about a specific platform.

2. Adding "-d" to valgrind will spew out a bunch of debug information, including which tool it's launching. Something like this:

--4603:1:launcher no client specified, defaulting platform to 'amd64-linux'
--4603:1:launcher launching /usr/local/lib/valgrind/memcheck-amd64-linux

"valgrind --verbose" can also be useful, of course.

3. Archive of valgrind users mailing list is here:

http://valgrind.10908.n7.nabble.com/Valgrind-Users-f33662i35.html

4. If you see this error when valgrind'ing a 32-bit application:

valgrind:  Fatal error at startup: a function redirection
valgrind:  which is mandatory for this platform-tool combination
valgrind:  cannot be set up.  Details of the redirection are:
valgrind:
valgrind:  A must-be-redirected function
valgrind:  whose name matches the pattern:      strlen
valgrind:  in an object with soname matching:   ld-linux.so.2
valgrind:  was not found whilst processing
valgrind:  symbols from the object with soname: ld-linux.so.2
valgrind:
valgrind:  Possible fixes: (1, short term): install glibc's debuginfo
valgrind:  package on this machine.  (2, longer term): ask the packagers
valgrind:  for your Linux distribution to please in future ship a non-
valgrind:  stripped ld.so (or whatever the dynamic linker .so is called)
valgrind:  that exports the above-named function using the standard
valgrind:  calling conventions for this platform.  The package you need
valgrind:  to install for fix (1) is called
valgrind:
valgrind:    On Debian, Ubuntu:                 libc6-dbg
valgrind:    On SuSE, openSuSE, Fedora, RHEL:   glibc-debuginfo
valgrind:
valgrind:  Cannot continue -- exiting now.  Sorry.

You most likely need the 32-bit libc-dbg package. On Debian based systems this should fix it:

sudo apt-get install libc6-dbg:i386

Valgrind Notes

Couple notes.

1. Command line for running valgrind with vogl capturing glxspheres64 from the vogl_build/bin directory:

valgrind --tool=memcheck --leak-check=full --error-limit=no --trace-children=yes --time-stamp=yes --log-file=/tmp/blah.log -- ../../bin/steamlauncher.sh --amd64 --gameid ./glxspheres64

2. Found some good stuff. Also a few things like this... :)

                // Get some entropy from the heap.
                p[i] = vogl_malloc(65536 * (i + 1));
                gen.update_obj_bits(p[i]);
                if (p[i])
                {
                    for (uint j = 0; j < 16; j++)
                        gen.update_obj_bits(reinterpret_cast<const uint64_t *>(p)[j]);
                }

2. Adding --track-origins=yes to the command line slows Valgrind down quite a bit but can really help. It added the line in bold for this stack trace (which wasn't making sense until we got this hint):

Uninitialised byte(s) found during client check request                                                                                                                             
   at 0x5422873: vogl_trace_stream_start_of_file_packet::compute_crc() const (vogl_trace_stream_types.h:185)
   by 0x54227B1: vogl_trace_stream_start_of_file_packet::check_crc(unsigned int) const (vogl_trace_stream_types.h:231)
   by 0x5421EE8: vogl_trace_stream_start_of_file_packet::full_validation(unsigned int) const (vogl_trace_stream_types.h:242)
   by 0x5420CB7: vogl_trace_file_writer::open(char const*, vogl_archive_blob_manager*, bool, bool, unsigned int) (vogl_trace_file_writer.cpp:82)
   by 0x517BAC0: vogl_global_init() (vogl_intercept.cpp:799) 
   by 0x92E236F: pthread_once (pthread_once.S:103)
   by 0x517A970: vogl_entrypoint_prolog(gl_entrypoint_id_t) (vogl_intercept.cpp:865) 
   by 0x50B3382: vogl_glXChooseVisual(_XDisplay const*, int, int const*) (gl_glx_func_defs.inc:91640)
   by 0x50B3302: glXChooseVisual (gl_glx_func_defs.inc:91635)
   by 0x403C84: main (glxspheres.c:716)
 Address 0x59632bd is 149 bytes inside data symbol "_ZZL21get_vogl_trace_writervE19s_vogl_trace_writer"
 Uninitialised value was created by a stack allocation
   at 0x5536214: vogl::init_uuid() (vogl_uuid.cpp:53)

3. And finally, if that doesn't do it, you can use code like this to help even more:

      #include "memcheck.h"
    ...  
      uintptr_t addr = VALGRIND_CHECK_MEM_IS_DEFINED(ptr, len);
      if (addr)
      {
          printf("VALGRIND_CHECK_MEM failed: %p %u\n", ptr, len);
          printf("  addr = %p\n", (void *)addr);
      }

Documentation for these markups (and much, much more) here:

http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs

Just grab valgrind.h and memcheck.h. We've checked them into the extlib/valgrind directory in vogl.

Bash Symbols

Debugging an issue where our preloaded vogl shared object is crashing bash. These are the steps I did to get the bash symbols on Linux Mint 16:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

# NOTE: Since I'm on Linux Mint (Petra) I then had to edit ddebs.list and change petra to saucy.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01

wget -q http://ddebs.ubuntu.com/dbgsym-release-key.asc

sudo apt-key add dbgsym-release-key.asc

sudo apt-get update

apt-cache policy bash

# Returns something like this:

> bash:
>   Installed: 4.2-5ubuntu3
>   Candidate: 4.2-5ubuntu3
> ...

sudo apt-get install bash-dbgsym=4.2-5ubuntu3

# Grab the source:

apt-get source bash

cd bash-4.2

# Untar the source (I use atool, feel free to use tar xf or whatever):

atool -x bash-4.2.tar.xz

# Now in gdb (or your .gdbinit) you can point to the bash source. For me:

directory /home/mikesart/src/bash-4.2/bash-4.2

# Here are a couple of good of good links for all this.

https://wiki.ubuntu.com/DebuggingProgramCrash

http://yaapb.wordpress.com/2012/12/28/debugging-your-running-kernel-in-ubuntu/

http://randomascii.wordpress.com/2013/01/08/symbols-on-linux-part-one-g-library-symbols/

QtCreator and Ninja warning/error parsing

We have a custom build binary that launches ninja and we couldn't get QtCreator to parse the warnings from that output. Errors/warnings show up, they just weren't being parsed. I finally got QtCreator building with symbols and tracked it down to this in makestep.cpp:

 267     if (m_useNinja)
 268         AbstractProcessStep::stdError(line);
 269     else
 270         AbstractProcessStep::stdOutput(line);

Ninja is processing stderr and regular makefiles are processing stdout. So I added this to our custom shell script and it's working now:

Command: .../bin/mkvogl.sh
Arguments: --amd64 --debug 3>&1 1>&2 2>&3

I guess if you ever find QtCreator isn't parsing your output, try swapping stderr and stdout. :)

If anyone wants to get QtCreator building release with symbols on something vaguely resembling 64-bit Linux Mint 16, this is what I wound up doing:

apt-get install: libgl1-mesa-dev libxml2-dev libglib2.0-dev libxslt1-dev libglib2.0-dev libgstreamer-plugins-base0.10-dev libgstreamer0.10-dev

diff --git a/qtcreator.pri b/qtcreator.pri
index 1750705..9be9c03 100644
--- a/qtcreator.pri
+++ b/qtcreator.pri
@@ -180,6 +180,9 @@ unix {

     RCC_DIR = $${OUT_PWD}/.rcc
     UI_DIR = $${OUT_PWD}/.uic
+
+    QMAKE_CXXFLAGS_RELEASE += -g
+    QMAKE_CFLAGS_RELEASE += -g
 }

 win32-msvc* {

cd ~/dev/qt-creator/src
qmake -r
make

If there is a better way of achieving this, please let me know - I don't know much about qmake and couldn't find anything...