Valgrind Notes

Couple notes.

1. Command line for running valgrind with vogl capturing glxspheres64 from the vogl_build/bin directory:

valgrind --tool=memcheck --leak-check=full --error-limit=no --trace-children=yes --time-stamp=yes --log-file=/tmp/blah.log -- ../../bin/steamlauncher.sh --amd64 --gameid ./glxspheres64

2. Found some good stuff. Also a few things like this... :)

                // Get some entropy from the heap.
                p[i] = vogl_malloc(65536 * (i + 1));
                gen.update_obj_bits(p[i]);
                if (p[i])
                {
                    for (uint j = 0; j < 16; j++)
                        gen.update_obj_bits(reinterpret_cast<const uint64_t *>(p)[j]);
                }

2. Adding --track-origins=yes to the command line slows Valgrind down quite a bit but can really help. It added the line in bold for this stack trace (which wasn't making sense until we got this hint):

Uninitialised byte(s) found during client check request                                                                                                                             
   at 0x5422873: vogl_trace_stream_start_of_file_packet::compute_crc() const (vogl_trace_stream_types.h:185)
   by 0x54227B1: vogl_trace_stream_start_of_file_packet::check_crc(unsigned int) const (vogl_trace_stream_types.h:231)
   by 0x5421EE8: vogl_trace_stream_start_of_file_packet::full_validation(unsigned int) const (vogl_trace_stream_types.h:242)
   by 0x5420CB7: vogl_trace_file_writer::open(char const*, vogl_archive_blob_manager*, bool, bool, unsigned int) (vogl_trace_file_writer.cpp:82)
   by 0x517BAC0: vogl_global_init() (vogl_intercept.cpp:799) 
   by 0x92E236F: pthread_once (pthread_once.S:103)
   by 0x517A970: vogl_entrypoint_prolog(gl_entrypoint_id_t) (vogl_intercept.cpp:865) 
   by 0x50B3382: vogl_glXChooseVisual(_XDisplay const*, int, int const*) (gl_glx_func_defs.inc:91640)
   by 0x50B3302: glXChooseVisual (gl_glx_func_defs.inc:91635)
   by 0x403C84: main (glxspheres.c:716)
 Address 0x59632bd is 149 bytes inside data symbol "_ZZL21get_vogl_trace_writervE19s_vogl_trace_writer"
 Uninitialised value was created by a stack allocation
   at 0x5536214: vogl::init_uuid() (vogl_uuid.cpp:53)

3. And finally, if that doesn't do it, you can use code like this to help even more:

      #include "memcheck.h"
    ...  
      uintptr_t addr = VALGRIND_CHECK_MEM_IS_DEFINED(ptr, len);
      if (addr)
      {
          printf("VALGRIND_CHECK_MEM failed: %p %u\n", ptr, len);
          printf("  addr = %p\n", (void *)addr);
      }

Documentation for these markups (and much, much more) here:

http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs

Just grab valgrind.h and memcheck.h. We've checked them into the extlib/valgrind directory in vogl.

Bash Symbols

Debugging an issue where our preloaded vogl shared object is crashing bash. These are the steps I did to get the bash symbols on Linux Mint 16:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

# NOTE: Since I'm on Linux Mint (Petra) I then had to edit ddebs.list and change petra to saucy.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01

wget -q http://ddebs.ubuntu.com/dbgsym-release-key.asc

sudo apt-key add dbgsym-release-key.asc

sudo apt-get update

apt-cache policy bash

# Returns something like this:

> bash:
>   Installed: 4.2-5ubuntu3
>   Candidate: 4.2-5ubuntu3
> ...

sudo apt-get install bash-dbgsym=4.2-5ubuntu3

# Grab the source:

apt-get source bash

cd bash-4.2

# Untar the source (I use atool, feel free to use tar xf or whatever):

atool -x bash-4.2.tar.xz

# Now in gdb (or your .gdbinit) you can point to the bash source. For me:

directory /home/mikesart/src/bash-4.2/bash-4.2

# Here are a couple of good of good links for all this.

https://wiki.ubuntu.com/DebuggingProgramCrash

http://yaapb.wordpress.com/2012/12/28/debugging-your-running-kernel-in-ubuntu/

http://randomascii.wordpress.com/2013/01/08/symbols-on-linux-part-one-g-library-symbols/

QtCreator and Ninja warning/error parsing

We have a custom build binary that launches ninja and we couldn't get QtCreator to parse the warnings from that output. Errors/warnings show up, they just weren't being parsed. I finally got QtCreator building with symbols and tracked it down to this in makestep.cpp:

 267     if (m_useNinja)
 268         AbstractProcessStep::stdError(line);
 269     else
 270         AbstractProcessStep::stdOutput(line);

Ninja is processing stderr and regular makefiles are processing stdout. So I added this to our custom shell script and it's working now:

Command: .../bin/mkvogl.sh
Arguments: --amd64 --debug 3>&1 1>&2 2>&3

I guess if you ever find QtCreator isn't parsing your output, try swapping stderr and stdout. :)

If anyone wants to get QtCreator building release with symbols on something vaguely resembling 64-bit Linux Mint 16, this is what I wound up doing:

apt-get install: libgl1-mesa-dev libxml2-dev libglib2.0-dev libxslt1-dev libglib2.0-dev libgstreamer-plugins-base0.10-dev libgstreamer0.10-dev

diff --git a/qtcreator.pri b/qtcreator.pri
index 1750705..9be9c03 100644
--- a/qtcreator.pri
+++ b/qtcreator.pri
@@ -180,6 +180,9 @@ unix {

     RCC_DIR = $${OUT_PWD}/.rcc
     UI_DIR = $${OUT_PWD}/.uic
+
+    QMAKE_CXXFLAGS_RELEASE += -g
+    QMAKE_CFLAGS_RELEASE += -g
 }

 win32-msvc* {

cd ~/dev/qt-creator/src
qmake -r
make

If there is a better way of achieving this, please let me know - I don't know much about qmake and couldn't find anything...

QtCreator projects

I've switched to using QtCreator 3.0 as my main editor. I'm really liking it (and FakeVim!), but one big issue we've run into is projects. What files are loaded, what defines are set, files not listed in our makefiles (.sh), listing include files in our makefiles just so they're in the project, etc. It also likes to blast .user files where you open your makefile and now we're dealing with getting mercurial or git to ignore those, etc.

I wrote the below which just finds everything under our vogl project directory with specified extensions. It also takes some patterns to remove files after the fact (possibly someone can tell me a more optimal way of doing this?).

Next I'm going to figure out how to get QtCreator to parse ninja build output. (Grumble, grumble. :)

In any case, throwing it up here in case someone might find it useful...

#

# VoglProj QtCreator cmake file.

#

# Do the following in the directory above your vogl enlistment:

#

#  ln -s vogl/bin/qtcreator/CMakeLists.txt

#

# Then open it up with QtCreator and you should be off and running.

#

project(VoglProj)

cmake_minimum_required(VERSION 2.8)

# List of file extensions that we search for.

set(EXTLIST *.i *.sh *.inl *.inc *.txt *.vs *.vp *.frag *.vert *.py *.m *.c* *.h* *.S)

# Vogl directory.

set(VOGL_DIR "${CMAKE_CURRENT_SOURCE_DIR}/vogl")

if(NOT EXISTS "${VOGL_DIR}/")

    message("\nERROR: ${VOGL_DIR} does not exist. Please put this script one level up from your vogl enlistement.\n")

    message(FATAL_ERROR "Exiting...")

endif()

# Create list of vogl directorie plus extensions.

set(GLOBSPEC)

foreach(ext ${EXTLIST})

    list(APPEND GLOBSPEC ${VOGL_DIR}/${ext})

endforeach()

# Search for all the files.

file(GLOB_RECURSE vogl_srcs

    RELATIVE ${CMAKE_CURRENT_SOURCE_DIR}

    ${GLOBSPEC}

    )

# Macro to remove files based on regex pattern.

macro(RemoveSrcFiles pat)

    set(result)

    foreach(file ${vogl_srcs})

        if(file MATCHES ${pat})

        else()

            list(APPEND result ${file})

        endif()

    endforeach()

    set(vogl_srcs ${result})

endmacro()

# Remove all files under .git and .hg directories.

RemoveSrcFiles("/[.]git/")

RemoveSrcFiles("/[.]hg/")

# Spew out all files we've found.

set(count 0)

foreach(file ${vogl_srcs})

    message("${file}")

    math(EXPR count "${count} + 1")

endforeach()

message("${count} files added.\n")

add_executable(VoglProj ${vogl_srcs})

set_target_properties(VoglProj PROPERTIES LINKER_LANGUAGE C)

GCC 4.8 on Ubuntu 12.04 x64

Ran into a bug in libstdc++ 4.7 building LLDB with Clang 3.3. So these are the notes on getting GCC 4.8 installed.

Here is a link talking about the libstdc++ bug:
http://stackoverflow.com/questions/15747223/why-does-this-basic-thread-program-fail-with-clang-but-pass-in-g

Good askubuntu link:
http://askubuntu.com/questions/193513/problem-adding-a-ppa-to-install-gcc-4-7

Ubuntu Toolchain PPA:
https://launchpad.net/~ubuntu-toolchain-r/+archive/test

Steps:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test

If that doesn't work, you can create the file manually:

mikesart@mikesart64:~/data/src/blah/build64$ cat /etc/apt/sources.list.d/toolchain.list
# https://launchpad.net/~ubuntu-toolchain-r/+archive/test
deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu precise main   
deb-src http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu precise main

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1E9377A2BA9EF27F
sudo apt-get update
sudo apt-get install gcc-4.8 g++-4.8

I then added gcc 4.8 to my alternatives list.

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50
sudo update-alternatives --install /usr/bin/cpp cpp-bin /usr/bin/cpp-4.8 50

Here are some clang / gcc commands to view various options, include paths, etc.

# Show default options and commands, plus include paths
mikesart@mikesart64:~/data/src/llvm.svn/build$ clang -v -fsyntax-only -x c++ /dev/null 2>&1
clang version 3.3 (tags/RELEASE_33/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
 "/home/mikesart/data/src/clang3.3/bin/clang" -cc1 -triple x86_64-unknown-linux-gnu -fsyntax-only -disable-free -disable-llvm-verifier -main-file-name null -mrelocation-model static -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -target-linker-version 2.20.1 -v -resource-dir /home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/x86_64-linux-gnu -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/x86_64-linux-gnu/c++/4.8 -internal-isystem /usr/local/include -internal-isystem /home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdeprecated-macro -fdebug-compilation-dir /home/mikesart/data/src/llvm.svn/build -ferror-limit 19 -fmessage-length 181 -mstackrealign -fobjc-runtime=gcc -fobjc-default-synthesize-properties -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -backend-option -vectorize-loops -x c++ /dev/null
clang -cc1 version 3.3 based upon LLVM 3.3 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/x86_64-linux-gnu"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/backward
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/x86_64-linux-gnu/c++/4.8
 /usr/local/include
 /home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

# Print the paths used for finding libraries and programs
mikesart@mikesart64:~/data/src/llvm.svn/build$ clang -print-search-dirs | tr : '\n'
programs
 =/home/mikesart/bin
/home/mikesart/data/src/clang3.3/bin
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../x86_64-linux-gnu/bin
libraries
 =/home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3
/usr/lib/gcc/x86_64-linux-gnu/4.8
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu
/lib/x86_64-linux-gnu
/lib/../lib64
/usr/lib/x86_64-linux-gnu
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../..
/lib
/usr/lib

# list all preprocessor definitions
clang -dM -E - < /dev/null
#define _LP64 1
#define __ATOMIC_ACQUIRE 2
#define __ATOMIC_ACQ_REL 4
#define __ATOMIC_CONSUME 1
#define __ATOMIC_RELAXED 0
#define __ATOMIC_RELEASE 3
#define __ATOMIC_SEQ_CST 5

#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__
...

Link for gcc options which control kind of output:
http://gcc.gnu.org/onlinedocs/gcc-4.3.5/gcc/Overall-Options.html

This one really useful: "If the -Q option appears on the command line before the --help= option, then the descriptive text displayed by --help= is changed. Instead of describing the displayed options, an indication is given as to whether the option is enabled, disabled or set to a specific value (assuming that the compiler knows this at the point where the --help= option is used)."

# See what gcc enables with native flag (sse, avx, etc)
# (Clang 3.3 doesn't appear to support the --help=XX stuff)
gcc -march=native -Q --help=target -v

--help=XX supports the following:

  optimizers: display all optimization options supported by the compiler.

  warnings: display all options controlling warning messages produced by the compiler.

  target: display target-specific options.

  params: display values recognized by the --param option.

  common: display options that are common to all languages.

  language: display options supported for language, where language = c++, etc.

You can add undocumented to list all undocumented target-specific switches as well. Ie:

/usr/bin/gcc-4.8 -march=native -Q --help=target,undocumented -v

/usr/bin/gcc-4.8 -march=native -Q --help=c++,undocumented -v

Simple SSE/AVX/MMX sample source code...

For testing a bunch of register stuff in LLDB. Shoved it up here also:

https://gist.github.com/mikesart/6832418#file-gistfile1-txt

// Output from my cmake VERBOSE=1 command for building:
// c++ -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -march=native -g -O0 -std=c++0x -g -o sse.cpp.o -c sse.cpp
// c++ -march=native -g -O0 -std=c++0x -g sse.cpp.o -o sse -rdynamic -ldl -lpthread

// SSE
//
#include <stdio.h>
#include <stdlib.h>

// #include <mmintrin.h> // MMX
// #include <xmmintrin.h> // SSE
// #include <emmintrin.h> // SSE2
// #include <pmmintrin.h> // SSE3
// #include <tmmintrin.h> // SSSE3
// #include <nmmintrin.h> // SSE4.1
// #include <ammintrin.h> // SSE4.2
// #include <wmmintrin.h> // AES/PCMUL
// #include <immintrin.h> // AVX
#include <x86intrin.h>      // Pulls in all of the above based on compiler switches (-march)

// AVX, SSE intrinsics, etc.:
// http://chessprogramming.wikispaces.com/AVX

// Intrinsics for Advanced Vector Extensions:
// http://software.intel.com/sites/products/documentation/hpc/composerxe/en-us/2011Update/cpp/lin/intref_cls/common/intref_bk_advectorext.htm

// Intrinsics for Advanced Vector Extensions 2:
// http://software.intel.com/sites/products/documentation/hpc/composerxe/en-us/2011Update/cpp/lin/intref_cls/common/intref_bk_advectorext2.htm

#ifndef __AVX__
#error AVX not defined
#endif

int main( int argc, char *argv[] )
{
    float a = 16.0f;
    float b = 9.0f;

    __m128 SSE0 = _mm_setzero_ps();
    __m128 SSEa = _mm_set_ps1(a);   // _mm_load1_ps(&a);
    __m128 SSEb = _mm_set_ps1(b);   // _mm_load1_ps(&b);
    __m128 SSEv = _mm_add_ps(SSEa, SSEb);

    __m256 AVX0 = _mm256_setzero_ps();
    __m256 AVXa = _mm256_set1_ps(a);
    __m256 AVXb = _mm256_set1_ps(b);
    __m256 AVXv = _mm256_add_ps(AVXa, AVXb);

    __m64 MMX0 = _mm_setzero_si64();
    __m64 MMXa = _mm_setr_pi32(16, 16);
    __m64 MMXb = _mm_setr_pi32(9, 9);
    __m64 MMXv = _mm_add_pi32(MMXa, MMXb);

    float temp[4] __attribute__((aligned(16)));
    _mm_store_ps(&temp[0], SSEv);
    printf("tempsse is %.2f %.2f %.2f %.2f\n", temp[0], temp[1], temp[2], temp[3]);

    float temp2[8] __attribute((aligned(32)));
    _mm256_store_ps(&temp2[0], AVXv);
    printf("tempavx is %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f\n",
        temp2[0], temp2[1], temp2[2], temp2[3],
        temp2[4], temp2[5], temp2[6], temp2[7]);

    printf("%d\n", _mm_cvtsi64_si32(MMXv));

    return 0;
}

More on Linux Threads

Got Linux thread names working in LLDB. "thread list" will now display the proper thread name and will be updated after calling pthread_setname_np(), etc. Still need thread-events, but that's a bit lower priority right now.

Couple of interesting notes & questions.

1. I initially implemented this by reading the "/proc/[pid]/task/[tid]/comm" file. Matt Kopec pointed out this could be read from "/proc/[pid]/comm" as well, even though "/proc/[tid]" isn't visible using ls in the terminal. This directory existing makes sense as threads are just light-weight processes, I just had never thought or read about it anywhere before. (Although to be fair, Pierre-Loup said he mentioned it to me at some point.)

2. For the curious, "/proc/self" has process granularity. Ie, I read "/proc/self/comm" from a background thread and it was the name of the process.

3. The "man proc" page for "/proc/[pid]/task" has this warning:

In a multithreaded process, the contents of the /proc/[pid]/task directory are not available if the main thread has already terminated (typically by calling pthread_exit(3)).

If anyone knows a system where this is true, I'd love to hear about it.

4. Gdb uses this libthread_db library to get notifications about new threads, and it looks like this is quite the doozy to set up and get running. Some great ( and only other than source? :) info on that here:

http://timetobleed.com/notes-about-an-odd-esoteric-yet-incredibly-useful-library-libthread_db/


LLDB doesn't use libthread_db though - it uses signals. Source code can be found in ProcessMonitor.cpp if you search for the "case (SIGTRAP | (PTRACE_EVENT_CLONE << 8))" statement in ProcessMonitor::MonitorSIGTRAP().

https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/Process/Linux/ProcessMonitor.cpp

My question would be: why on earth go through all the trouble to use libthread_db if signals will work just as well?

There is an intriguing note in the libthread_db post where he mentions accessing thread local data:

Now you can use the library

At this point, you’ve done enough setup to be able to dlsym search for and call various functions to iterate over the threads in a remote process, to be notified asynchronously when threads are created or destroyed, and to access thread local data if you want to.

Now that could be incredibly useful... but from what I can tell, gdb doesn't use this feature. Getting to tls data in gdb (unless I've missed something) is a bit of a pain in the backside.

I'm going to put these on the backburner for now and start trying to track down some stack tracing bugs. Which means diving in and trying to understand CIE and FDEs: http://www.airs.com/blog/archives/460

Good times!