GCC 4.8 on Ubuntu 12.04 x64

Ran into a bug in libstdc++ 4.7 building LLDB with Clang 3.3. So these are the notes on getting GCC 4.8 installed.

Here is a link talking about the libstdc++ bug:
http://stackoverflow.com/questions/15747223/why-does-this-basic-thread-program-fail-with-clang-but-pass-in-g

Good askubuntu link:
http://askubuntu.com/questions/193513/problem-adding-a-ppa-to-install-gcc-4-7

Ubuntu Toolchain PPA:
https://launchpad.net/~ubuntu-toolchain-r/+archive/test

Steps:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test

If that doesn't work, you can create the file manually:

mikesart@mikesart64:~/data/src/blah/build64$ cat /etc/apt/sources.list.d/toolchain.list
# https://launchpad.net/~ubuntu-toolchain-r/+archive/test
deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu precise main   
deb-src http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu precise main

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1E9377A2BA9EF27F
sudo apt-get update
sudo apt-get install gcc-4.8 g++-4.8

I then added gcc 4.8 to my alternatives list.

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50
sudo update-alternatives --install /usr/bin/cpp cpp-bin /usr/bin/cpp-4.8 50

Here are some clang / gcc commands to view various options, include paths, etc.

# Show default options and commands, plus include paths
mikesart@mikesart64:~/data/src/llvm.svn/build$ clang -v -fsyntax-only -x c++ /dev/null 2>&1
clang version 3.3 (tags/RELEASE_33/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
 "/home/mikesart/data/src/clang3.3/bin/clang" -cc1 -triple x86_64-unknown-linux-gnu -fsyntax-only -disable-free -disable-llvm-verifier -main-file-name null -mrelocation-model static -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -target-linker-version 2.20.1 -v -resource-dir /home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/x86_64-linux-gnu -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/x86_64-linux-gnu/c++/4.8 -internal-isystem /usr/local/include -internal-isystem /home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdeprecated-macro -fdebug-compilation-dir /home/mikesart/data/src/llvm.svn/build -ferror-limit 19 -fmessage-length 181 -mstackrealign -fobjc-runtime=gcc -fobjc-default-synthesize-properties -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -backend-option -vectorize-loops -x c++ /dev/null
clang -cc1 version 3.3 based upon LLVM 3.3 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/x86_64-linux-gnu"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/backward
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/x86_64-linux-gnu/c++/4.8
 /usr/local/include
 /home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

# Print the paths used for finding libraries and programs
mikesart@mikesart64:~/data/src/llvm.svn/build$ clang -print-search-dirs | tr : '\n'
programs
 =/home/mikesart/bin
/home/mikesart/data/src/clang3.3/bin
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../x86_64-linux-gnu/bin
libraries
 =/home/mikesart/data/src/clang3.3/bin/../lib/clang/3.3
/usr/lib/gcc/x86_64-linux-gnu/4.8
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu
/lib/x86_64-linux-gnu
/lib/../lib64
/usr/lib/x86_64-linux-gnu
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../..
/lib
/usr/lib

# list all preprocessor definitions
clang -dM -E - < /dev/null
#define _LP64 1
#define __ATOMIC_ACQUIRE 2
#define __ATOMIC_ACQ_REL 4
#define __ATOMIC_CONSUME 1
#define __ATOMIC_RELAXED 0
#define __ATOMIC_RELEASE 3
#define __ATOMIC_SEQ_CST 5

#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__
...

Link for gcc options which control kind of output:
http://gcc.gnu.org/onlinedocs/gcc-4.3.5/gcc/Overall-Options.html

This one really useful: "If the -Q option appears on the command line before the --help= option, then the descriptive text displayed by --help= is changed. Instead of describing the displayed options, an indication is given as to whether the option is enabled, disabled or set to a specific value (assuming that the compiler knows this at the point where the --help= option is used)."

# See what gcc enables with native flag (sse, avx, etc)
# (Clang 3.3 doesn't appear to support the --help=XX stuff)
gcc -march=native -Q --help=target -v

--help=XX supports the following:

  optimizers: display all optimization options supported by the compiler.

  warnings: display all options controlling warning messages produced by the compiler.

  target: display target-specific options.

  params: display values recognized by the --param option.

  common: display options that are common to all languages.

  language: display options supported for language, where language = c++, etc.

You can add undocumented to list all undocumented target-specific switches as well. Ie:

/usr/bin/gcc-4.8 -march=native -Q --help=target,undocumented -v

/usr/bin/gcc-4.8 -march=native -Q --help=c++,undocumented -v

Simple SSE/AVX/MMX sample source code...

For testing a bunch of register stuff in LLDB. Shoved it up here also:

https://gist.github.com/mikesart/6832418#file-gistfile1-txt

// Output from my cmake VERBOSE=1 command for building:
// c++ -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -march=native -g -O0 -std=c++0x -g -o sse.cpp.o -c sse.cpp
// c++ -march=native -g -O0 -std=c++0x -g sse.cpp.o -o sse -rdynamic -ldl -lpthread

// SSE
//
#include <stdio.h>
#include <stdlib.h>

// #include <mmintrin.h> // MMX
// #include <xmmintrin.h> // SSE
// #include <emmintrin.h> // SSE2
// #include <pmmintrin.h> // SSE3
// #include <tmmintrin.h> // SSSE3
// #include <nmmintrin.h> // SSE4.1
// #include <ammintrin.h> // SSE4.2
// #include <wmmintrin.h> // AES/PCMUL
// #include <immintrin.h> // AVX
#include <x86intrin.h>      // Pulls in all of the above based on compiler switches (-march)

// AVX, SSE intrinsics, etc.:
// http://chessprogramming.wikispaces.com/AVX

// Intrinsics for Advanced Vector Extensions:
// http://software.intel.com/sites/products/documentation/hpc/composerxe/en-us/2011Update/cpp/lin/intref_cls/common/intref_bk_advectorext.htm

// Intrinsics for Advanced Vector Extensions 2:
// http://software.intel.com/sites/products/documentation/hpc/composerxe/en-us/2011Update/cpp/lin/intref_cls/common/intref_bk_advectorext2.htm

#ifndef __AVX__
#error AVX not defined
#endif

int main( int argc, char *argv[] )
{
    float a = 16.0f;
    float b = 9.0f;

    __m128 SSE0 = _mm_setzero_ps();
    __m128 SSEa = _mm_set_ps1(a);   // _mm_load1_ps(&a);
    __m128 SSEb = _mm_set_ps1(b);   // _mm_load1_ps(&b);
    __m128 SSEv = _mm_add_ps(SSEa, SSEb);

    __m256 AVX0 = _mm256_setzero_ps();
    __m256 AVXa = _mm256_set1_ps(a);
    __m256 AVXb = _mm256_set1_ps(b);
    __m256 AVXv = _mm256_add_ps(AVXa, AVXb);

    __m64 MMX0 = _mm_setzero_si64();
    __m64 MMXa = _mm_setr_pi32(16, 16);
    __m64 MMXb = _mm_setr_pi32(9, 9);
    __m64 MMXv = _mm_add_pi32(MMXa, MMXb);

    float temp[4] __attribute__((aligned(16)));
    _mm_store_ps(&temp[0], SSEv);
    printf("tempsse is %.2f %.2f %.2f %.2f\n", temp[0], temp[1], temp[2], temp[3]);

    float temp2[8] __attribute((aligned(32)));
    _mm256_store_ps(&temp2[0], AVXv);
    printf("tempavx is %.2f %.2f %.2f %.2f %.2f %.2f %.2f %.2f\n",
        temp2[0], temp2[1], temp2[2], temp2[3],
        temp2[4], temp2[5], temp2[6], temp2[7]);

    printf("%d\n", _mm_cvtsi64_si32(MMXv));

    return 0;
}

More on Linux Threads

Got Linux thread names working in LLDB. "thread list" will now display the proper thread name and will be updated after calling pthread_setname_np(), etc. Still need thread-events, but that's a bit lower priority right now.

Couple of interesting notes & questions.

1. I initially implemented this by reading the "/proc/[pid]/task/[tid]/comm" file. Matt Kopec pointed out this could be read from "/proc/[pid]/comm" as well, even though "/proc/[tid]" isn't visible using ls in the terminal. This directory existing makes sense as threads are just light-weight processes, I just had never thought or read about it anywhere before. (Although to be fair, Pierre-Loup said he mentioned it to me at some point.)

2. For the curious, "/proc/self" has process granularity. Ie, I read "/proc/self/comm" from a background thread and it was the name of the process.

3. The "man proc" page for "/proc/[pid]/task" has this warning:

In a multithreaded process, the contents of the /proc/[pid]/task directory are not available if the main thread has already terminated (typically by calling pthread_exit(3)).

If anyone knows a system where this is true, I'd love to hear about it.

4. Gdb uses this libthread_db library to get notifications about new threads, and it looks like this is quite the doozy to set up and get running. Some great ( and only other than source? :) info on that here:

http://timetobleed.com/notes-about-an-odd-esoteric-yet-incredibly-useful-library-libthread_db/


LLDB doesn't use libthread_db though - it uses signals. Source code can be found in ProcessMonitor.cpp if you search for the "case (SIGTRAP | (PTRACE_EVENT_CLONE << 8))" statement in ProcessMonitor::MonitorSIGTRAP().

https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/Process/Linux/ProcessMonitor.cpp

My question would be: why on earth go through all the trouble to use libthread_db if signals will work just as well?

There is an intriguing note in the libthread_db post where he mentions accessing thread local data:

Now you can use the library

At this point, you’ve done enough setup to be able to dlsym search for and call various functions to iterate over the threads in a remote process, to be notified asynchronously when threads are created or destroyed, and to access thread local data if you want to.

Now that could be incredibly useful... but from what I can tell, gdb doesn't use this feature. Getting to tls data in gdb (unless I've missed something) is a bit of a pain in the backside.

I'm going to put these on the backburner for now and start trying to track down some stack tracing bugs. Which means diving in and trying to understand CIE and FDEs: http://www.airs.com/blog/archives/460

Good times!

LLDB Project Notes

This has been moved to: https://bitbucket.org/mikesart/lldb_branch/wiki/LLDB%20Project%20Notes

#
# Useful links
#

Subversion Commit Access: http://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access

lldb build page: http://lldb.llvm.org/build.html

lldb Linux buglist: http://llvm.org/bugs/buglist.cgi?cmdtype=runnamed&namedcmd=lldb-linux&list_id=40756

lldb-dev archives: http://lists.cs.uiuc.edu/pipermail/lldb-dev/
lldb-commits archives: http://lists.cs.uiuc.edu/pipermail/lldb-commits/

LLDB Reference Documentation: http://lldb.llvm.org/docs.html
GDB commands in LLDB: http://lldb.llvm.org/lldb-gdb.html

Code reviews with Phabricator: http://llvm.org/docs/Phabricator.html
 - This phabricator thing produces easy to read diffs so I can submit patches like this:
     http://lists.cs.uiuc.edu/pipermail/lldb-dev/2013-July/002027.html
 - "arc diff" and "arc submit" are the two main commands.
   Warning: they can be a bit of a pain when working with multiple patches.

Rad's working lldb branch: https://bitbucket.org/mikesart/lldb_branch
 - has a couple fixes not merged into lldb svn branch yet (largest is libedit 3.1).

Rad's bugs/work list: https://bitbucket.org/mikesart/lldb_branch/issues?status=new&status=open

I've set up two lldb enlistments:
 - Our working mercurial branch: ~/data/src/lldb.hg
     (tools/lldb is from our Mercurial branch)
 - Official Subversion branch: ~/data/src/lldb.svn
     (tools/lldb is from lld Subversion branch)

Each branch looks like this on my machine:

  lldb.hg
  |
  `-- build
      |
      llvm
      |
      `-- tools
          |
          +-- clang
          |
          `-- lldb

I test individual patches, submit, test official tree from lldb.svn, and work out of lldb.hg.

#
# Tools
#

cmake version 2.8.10.2
ninja 1.3.3
Mercurial Distributed SCM (version 2.6)
svn, version 1.7.9 (r1462340), compiled Apr 6 2013, 21:23:46
Clang 3.3 (http://linux-debugger-bits.blogspot.com/2013/07/clang-33-with-64-bit-ubuntu-1204.html)

# Other useful tools:
TortoiseHg Dialogs (version 2.8), Mercurial (version 2.6)
meld 1.7.3
CGDB 20130523 (cgdb built from http://cgdb.github.io/)
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04

#
# internal patch to get lldb build warnings down (until these can be fixed).
#

Index: ../../cmake/modules/HandleLLVMOptions.cmake
===================================================================
--- ../../cmake/modules/HandleLLVMOptions.cmake (revision 186469)
+++ ../../cmake/modules/HandleLLVMOptions.cmake (working copy)
@@ -206,6 +206,12 @@
   if (LLVM_ENABLE_WARNINGS)
     append("-Wall -W -Wno-unused-parameter -Wwrite-strings" CMAKE_C_FLAGS CMAKE_CXX_FLAGS)

+       # option(RAD_EXTRA_CFLAGS "View the TERM environment var" OFF)
+    if(RAD_EXTRA_CFLAGS)
+        message(RAD_EXTRA_CFLAGS " environment variable is ${RAD_EXTRA_CFLAGS}")
+        append(${RAD_EXTRA_CFLAGS} CMAKE_C_FLAGS CMAKE_CXX_FLAGS)
+    endif(RAD_EXTRA_CFLAGS)
+
     # Turn off missing field initializer warnings for gcc to avoid noise from
     # false positives with empty {}. Turn them on otherwise (they're off by
     # default for clang).

#
# My bash aliases for working with lldb project
#
# Can run following from lldb build directories:
#   ninja lldb
#   ninja check-lldb
# To run individual tests, something like this from lldb/test directory:
#   ./dotest.py -C clang -v -t -f PlatformCommandTestCase.test_process_list

# cd lldb_src_hg, etc. will work if "shopt -s cdable_vars" is set.
export lldb_src_hg=~/data/src/llvm.hg/llvm/tools/lldb
export lldb_src_svn=~/data/src/llvm.svn/llvm/tools/lldb
export lldb_build_hg=~/data/src/llvm.hg/build
export lldb_build_svn=~/data/src/llvm.svn/build

path_append ()  { path_remove $1; export PATH="$PATH:$1"; }
path_prepend () { path_remove $1; export PATH="$1:$PATH"; }
path_remove ()  { export PATH=`echo -n $PATH | awk -v RS=: -v ORS=: '$0 != "'$1'"' | sed 's/:$//'`; }

lldb_setenv_svn()
{
    path_remove "/home/mikesart/data/src/llvm.hg/build/bin"
    path_prepend "/home/mikesart/data/src/llvm.svn/build/bin"
}
lldb_setenv_hg()
{
    path_remove "/home/mikesart/data/src/llvm.svn/build/bin"
    path_prepend "/home/mikesart/data/src/llvm.hg/build/bin"
}
lldb_cmake_debug()
{
    # run from lldb build directory to create ninja build files
    CC=clang cmake -DRAD_EXTRA_CFLAGS="-Wno-c99-extensions -Wno-sign-compare -Wno-four-char-constants -Wno-extended-offsetof -Wno-unused-function" -DCMAKE_CXX_FLAGS="-fcolor-diagnostics" -DCMAKE_BUILD_TYPE=Debug -C ../llvm -G Ninja
}

#
# Faster debugging with gdb...
#

Gdb with lldb is super slow loading the symbols. Connecting can take ~15 seconds. I run this alias after building (takes about 19 seconds) and loading symbols with gdb drops to less than 1 second.

mikesart@mikesart-rad:~/data/src/llvm.hg/llvm/tools/lldb/test$ type lldb_gdb_add_index
lldb_gdb_add_index is a function
lldb_gdb_add_index ()
{
    echo gdb-add-index $(readlink -f $(dirname $(which lldb))/../lib/liblldb.so);
    time gdb-add-index $(readlink -f $(dirname $(which lldb))/../lib/liblldb.so)
}

mikesart@mikesart-rad:~/data/src/llvm.hg/llvm/tools/lldb/test$ cat ~/bin/gdb-add-index
#! /bin/sh

# Add a .gdb_index section to a file.

# Copyright (C) 2010 Free Software Foundation, Inc.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

file="$1"
dir="${file%/*}"

# We don't care if gdb gives an error.
/home/mikesart/data/src/gdb-7.6/gdb/gdb --data-directory=/home/mikesart/data/src/gdb-7.6/gdb/data-directory -nx --batch-silent -ex "file $file" -ex "save gdb-index $dir"

if test -f "${file}.gdb-index"; then
   objcopy --add-section .gdb_index="${file}.gdb-index" --set-section-flags .gdb_index=readonly "$file" "$file"
   rm -f "${file}.gdb-index"
fi

exit 0

Linux pthread test app with lldb

Testing some multithreaded debugging with lldb. We've got four issues to start looking at so far...

1. Running "gdb -- blah", "b nanosleep", "r", will result in breaking on the nanosleep call.
Running "lldb -- blah", "b nanosleep", "r", will result in breaking in the 'ret' instruction of the nanosleep call. So the break doesn't happen until after the sleep period.

2. Doing a "gdb -- blah", "b main", "r", "b 71", "c", "info threads" will result in this on gdb:

(gdb) info threads
  Id   Target Id         Frame
  3    Thread 0x7ffff65e2700 (LWP 5353) "thread_1" 0x00007ffff6ea384d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  2    Thread 0x7ffff6de3700 (LWP 5350) "thread_0" 0x00007ffff6ea384d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
* 1    Thread 0x7ffff7fd1740 (LWP 5220) "mainthrd" main (argc=1, argv=0x7fffffffdd08) at /home/mikesart/data/src/blah_pthreads/blah.cpp:71

This on lldb:

(lldb) thread list
Process 5520 stopped
* thread #1: tid = 0x1590, 0x000000000040113f blah`main(argc=1, argv=0x00007fff4ef17f28) + 495 at blah.cpp:71, name = 'blah, stop reason = breakpoint 2.1
  thread #2: tid = 0x15ff, 0x00007f92502d584d libc.so.6`__nanosleep + 45 at syscall-template.S:82, name = 'mainthrd
  thread #3: tid = 0x1600, 0x00007f92502d584d libc.so.6`__nanosleep + 45 at syscall-template.S:82, name = 'mainthrd

The TIDs are hex and the thread names are wrong.

3. On gdb, "thread apply all" gives us this:

(gdb) thread apply all bt
Thread 3 (Thread 0x7ffff65e2700 (LWP 5353)):
#0  0x00007ffff6ea384d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff6ea36ec in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
#2  0x0000000000400f28 in thread_proc (arg=0x1) at /home/mikesart/data/src/blah_pthreads/blah.cpp:37
#3  0x00007ffff79c0e9a in start_thread (arg=0x7ffff65e2700) at pthread_create.c:308
#4  0x00007ffff6ed7ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7ffff6de3700 (LWP 5350)):
#0  0x00007ffff6ea384d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff6ea36ec in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
#2  0x0000000000400f28 in thread_proc (arg=0x0) at /home/mikesart/data/src/blah_pthreads/blah.cpp:37
#3  0x00007ffff79c0e9a in start_thread (arg=0x7ffff6de3700) at pthread_create.c:308
#4  0x00007ffff6ed7ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7ffff7fd1740 (LWP 5220)):
#0  main (argc=1, argv=0x7fffffffdd08) at /home/mikesart/data/src/blah_pthreads/blah.cpp:71

On lldb, this (no backtraces for sleeping threads):

(lldb) bt all
* thread #1: tid = 0x1590, 0x000000000040113f blah`main(argc=1, argv=0x00007fff4ef17f28) + 495 at blah.cpp:78, name = 'blah, stop reason = breakpoint 2.1
    frame #0: 0x000000000040113f blah`main(argc=1, argv=0x00007fff4ef17f28) + 495 at blah.cpp:71
    frame #1: 0x00007f925023776d libc.so.6`__libc_start_main(main=0x0000000000400f50, argc=1, ubp_av=0x00007fff4ef17f28, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fff4ef17f18) + 237 at libc-start.c:226
    frame #2: 0x0000000000400b99 blah`_start + 41
  thread #2: tid = 0x15ff, 0x00007f92502d584d libc.so.6`__nanosleep + 45 at syscall-template.S:82, name = 'mainthrd
    frame #0: 0x00007f92502d584d libc.so.6`__nanosleep + 45 at syscall-template.S:82
  thread #3: tid = 0x1600, 0x00007f92502d584d libc.so.6`__nanosleep + 45 at syscall-template.S:82, name = 'mainthrd
    frame #0: 0x00007f92502d584d libc.so.6`__nanosleep + 45 at syscall-template.S:82

4. gdb has these excellent thread-events (controllable via set print thread-events). Nothing like this in lldb yet.
      [New Thread 0x7ffff6de3700 (LWP 19270)]
      [Thread 0x7ffff6de3700 (LWP 19270) exited]

Here is the test source.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 #include <stdio.h>
   2 #include <string.h>
   3 #include <pthread.h>
   4 #include <stdlib.h>
   5 #include <unistd.h>
   6 #include <sys/syscall.h>
   7 
   8 __thread pid_t g_tls = -1;
   9 __thread char g_threadname[32];
  10 
  11 pid_t gettid()
  12 {
  13     return (pid_t)syscall(SYS_gettid);
  14 }
  15 
  16 void logf(const char *format, ...)
  17 {
  18     va_list args;
  19     char buf[1024];
  20 
  21     snprintf(buf, sizeof(buf), "'%s' [#%d LWP:%d 0x%lx] %s", g_threadname, g_tls, gettid(), pthread_self(), format);
  22 
  23     va_start (args, format);
  24     vprintf (buf, args);
  25     va_end (args);
  26 }
  27 
  28 void *thread_proc(void *arg)
  29 {
  30     g_tls = (int)(intptr_t)arg;
  31 
  32     logf("pthread_setname_np('%s')\n", g_threadname);
  33     snprintf(g_threadname, sizeof(g_threadname), "thread_%d", g_tls);
  34     pthread_setname_np(pthread_self(), g_threadname);
  35 
  36     logf("sleep(5)\n");
  37     sleep(5);
  38 
  39     pid_t tid = gettid();
  40     logf("pthread_exit(%d)\n", tid);
  41     pthread_exit((void *)(intptr_t)tid);
  42     return 0;
  43 }
  44 
  45 int main(int argc, char *argv[])
  46 {
  47     pthread_t threadids[256];
  48     static const size_t max_threads = sizeof(threadids) / sizeof(threadids[0]);
  49 
  50     size_t num_threads = (argc > 1) ? atoi(argv[1]) : 2;
  51     if (num_threads < 2)
  52         num_threads = 2;
  53     else if (num_threads > max_threads)
  54         num_threads = max_threads;
  55 
  56     snprintf(g_threadname, sizeof(g_threadname), "mainthrd");
  57     pthread_setname_np(pthread_self(), g_threadname);
  58 
  59     printf("num_threads:%zu\n", num_threads);
  60 
  61     for(size_t i = 0; i < num_threads; i++)
  62     {
  63         int err = pthread_create(&(threadids[i]), NULL, &thread_proc, (void *)(intptr_t)i);
  64         logf("pthread_create:%d (%s) pthread_t:%lx\n", err, strerror(err), threadids[i]);
  65     }
  66 
  67     sleep(1);
  68 
  69     for(size_t i = 0; i < num_threads; i++)
  70     {
  71         logf("Waiting for thread #%zu\n", i);
  72 
  73         void *status = NULL;
  74         int rc = pthread_join(threadids[i], &status);
  75         logf("Thread #%zu rc:%d status:%d\n", i, rc, (int)(intptr_t)status);
  76     }
  77 
  78     printf("done.\n");
  79     return 0;
  80 } 

gdb catchpoints

Not sure how the hell didn't know about this until now, but these gdb catchpoints are pretty nifty.

http://sourceware.org/gdb/onlinedocs/gdb/Set-Catchpoints.html

You can do something like "catch load libbfd.so" and you'll hit a breakpoint in dl-debug.c when that .so is loaded.

(gdb) catch load libbfd.so
Catchpoint 1 (load)
(gdb) r
Starting program: /home/mikesart/data/src/blah_dyldrendezvous_crash/build/blah
hello world

Catchpoint 1
  Inferior loaded /usr/lib/libbfd.so
    /lib/x86_64-linux-gnu/libz.so.1
__GI__dl_debug_state () at dl-debug.c:77
(gdb) bt
#0  __GI__dl_debug_state () at dl-debug.c:77
#1  in dl_open_worker
#2  _dl_catch_erroroperate
#3  in _dl_open
#4  in dlopen_doit
#5  in _dl_catch_error
#6  in _dlerror_run
#7  in __dlopen
#8  in main

There are catchpoints for calls to exec, fork, syscalls, and signals that could all be useful as well.

I don't see anything like this in lldb - adding to the task list...

Clang 3.3 with 64-bit Ubuntu 12.04

Just documenting how I switch between various compilers on Linux with update-alternatives. I use update-alternatives, a tool to "maintain symbolic links determining default commands."

1. Grab appropriate clang 3.3 binaries from: http://llvm.org/releases/download.html

2. Unpack somewhere. (I unpacked to my ~/data directory using atool -x).

2. Install binaries with update-alternatives. For gcc 4.6 and clang 3.3, I did the following:

# install gcc 4.6
# "update-alternatives --install" takes a link, name, path and priority.
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 100
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.6 100
sudo update-alternatives --install /usr/bin/cpp cpp-bin /usr/bin/cpp-4.6 100

# install clang 3.3
# "update-alternatives --install" takes a link, name, path and priority.
sudo update-alternatives --install /usr/bin/gcc gcc ~/data/clang3.3/bin/clang 50
sudo update-alternatives --install /usr/bin/g++ g++ ~/data/clang3.3/bin/clang++ 50
sudo update-alternatives --install /usr/bin/cpp cpp-bin /usr/bin/cpp-4.6 100

3. I then use the following bash functions to switch compilers.

compiler_clang-3.3()

{

    sudo update-alternatives --set gcc ~/data/clang3.3/bin/clang

    sudo update-alternatives --set g++ ~/data/clang3.3/bin/clang++

    sudo update-alternatives --set cpp-bin /usr/bin/cpp-4.6

    update-alternatives --display gcc

    update-alternatives --display g++

    update-alternatives --display cpp-bin

}

compiler_gcc-4.6()

{

    sudo update-alternatives --auto gcc

    sudo update-alternatives --auto g++

    sudo update-alternatives --auto cpp-bin

    update-alternatives --display gcc

    update-alternatives --display g++

    update-alternatives --display cpp-bin

}

4. I also add clang and clang++ explicitly to my path in case something uses them explicitly.

mikesart@mikesart-rad:~/data/src/blah/build$ ll ~/bin/clang*

lrwxrwxrwx 1 mikesart mikesart 50 Jun 12 14:12 ~/bin/clang -> ~/data/clang3.3/bin/clang

lrwxrwxrwx 1 mikesart mikesart 52 Jun 12 14:12 ~/bin/clang++ -> ~/data/clang3.3/bin/clang++

5 - 100. MAJOR WARNING: Make sure you switch back to your default compiler before installing or updating pretty much anything. Various components like the NVIDIA driver will fail to build with anything but the system default, and then you're in for some good times.

State of LLDB on Linux

I've had this question pop up a few times now. Can be summed up as "why is Rad/Valve working on LLDB on Linux - isn't it already done?" Some people get a bit aggressive about it:

Roberts then followed up with another Tweet, "To be more clear: RAD is looking for developers to write debuggers. We are working on lldb on Linux currently..."

Seriously? The giant's share of the work is coming from Apple. RAD is either exaggerating their intent, talking out their ass, and or both. What little they are working on is testing and bug tracking to make sure LLDB is not broken on Linux. LLDB has been working on Linux for nearly a year.

Sometimes I wish I could Gong Show portions of the Internets.

Many good folks have done a lot of wonderful work and made a lot of progress the past year on Linux LLDB. This includes four developers we've recently met at Intel, as well as several from Apple, FreeBSD, and others - but there is still a _lot_ to do before we can use LLDB to debug L4D2 and TF2. Or even LLDB. Or even a 32-bit version of printf("talking out our asses!");

When we started working on this a couple months ago, you couldn't list currently running Linux processes or attach by process name. Major features like threading and split symbol support have just recently been added. One month ago you could not debug multi-threaded applications or step into system libraries with symbols or source.

As of right now, you still can't load core files and debugging 32-bit Linux applications pretty much just doesn't work. We just started looking at the i386 test suite today.

DWARF4 symbols (the default for gcc 4.8) do not work.

Expressions can be shaky and backtraces could use some sweet, sweet lovin'.

I'm tracking down an assert and crash loading the nss shared library right now. (See last blog post).

Several tests in the test suite currently fail on 64-bit Linux.

33 of 262 tests fail on 32-bit Linux.

Linux LLDB has a tendency to hang at times, and I'm currently seeing some crazy long target load times.

There is no remote debugging on Linux.

For more, take a look at the current Linux lldb bug database:

http://llvm.org/bugs/buglist.cgi?resolution=---&op_sys=Linux&query_format=advanced&component=All%20Bugs&product=lldb

There was a good blog post about LLDB 3.3 recently on llvm.org that is well worth reading.

http://blog.llvm.org/2013/06/lldb-33-and-beyond.html

There is a lot to like about LLDB, and we have really enjoyed the community and working on it. There is also much to do before we can ditch gdb and use it fulltime. If any of this work sounds interesting, please jump in and help!

r_debug

LLDB is asserting and dying in DYLDRendezvous::UpdateSOEntries() and I got a core dump for it and the target it was debugging.

I load the core dump for the target - it's crashing in _dl_debug_state() line in _dl_map_object_from_fd():

 1040|    /* Notify the debugger we have added some objects. We need to  
 1041|     call _dl_debug_initialize in a static program in case dynamic  
 1042|     linking has not been used before. */  
 1043|    r->r_state = RT_ADD;  
 1044+>   _dl_debug_state ();  
 1045|    make_consistent = true;  

_dl_debug_state() is an empty function though.

 70| /* This function exists solely to have a breakpoint set on it by the  
 71|  debugger. The debugger is supposed to find this function's address by  
 72|  examining the r_brk member of struct r_debug, but GDB 4.15 in fact looks  
 73|  for this particular symbol name in the PT_INTERP file. */  
 74| void  
 75| _dl_debug_state (void)  
 76| {  
 77+>}  

Why on earth would it crash there?

(gdb) disassemble _dl_debug_state

Dump of assembler code for function __GI__dl_debug_state:

   0x00007fb123d9cb30 <+0>:     int3

=> 0x00007fb123d9cb31 <+1>:     ret

End of assembler dump.

Oh.

LLDB must have shoved the int3 in there, then it died and didn't remove the mess and the target went down. This is pretty slick though - this is how the debugger is notified that a new shared object is being loaded. Lots of details in elf/link.h that I'm going to start reading.

http://sourceware.org/ml/gdb/2000-q1/msg00193.html

Sadly, loading the core file for lldb doesn't go so well...

BFD: Warning: /var/crash/core.internal-state.6.23639.mikesart-rad.1373391421 is truncated: expected core file size >= 339070976, found: 105598976.

Mixed mode disassembly

I was testing on 64-bit Linux with LLDB and this simple program, and ran into this bit of interesting gdb behavior I had never noticed before.

  1| #include <stdio.h>  
  2| #include <stdlib.h>  
  3|  
  4| int main( int argc, char *argv[] )  
  5| {  
  6|   int blah2[8192];  
  7|   for(size_t i = 0; i < 8192; ++i)  
  8|   {  
  9|     blah2[i] = rand();  
 10|   }  
 11| }  

Set a breakpoint on line 7 with LLDB, and get two locations:

 (lldb) breakpoint set -l 7  
 Breakpoint 2: 2 locations.  

With gdb, it sets one breakpoint:

  (gdb) b 7   
  Breakpoint 1 at 0x4007c9: file ~/data/src/blah2/blah.cpp, line 7.   

I spew out the disassembly with mixed source. LLDB and gdb look quite different. Took a second to figure out what's going on... gdb is moving assembly instructions to match them up with line numbers. It moved the four bold instructions up before the rand() call. Crazy!

 (gdb) disassemble /m main  
 Dump of assembler code for function main(int, char**):  
 5    {  
   0x00000000004007b0 <+0>:   push  rbp  
   0x00000000004007b1 <+1>:   mov  rbp,rsp  
   0x00000000004007b4: sub  rsp,0x8020  
   0x00000000004007bb: mov  DWORD PTR [rbp-0x4],0x0  
   0x00000000004007c2: mov  DWORD PTR [rbp-0x8],edi  
   0x00000000004007c5: mov  QWORD PTR [rbp-0x10],rsi  
 6      int blah2[8192];  
 7      for(size_t i = 0; i < 8192; ++i)  
 => 0x00000000004007c9: mov  QWORD PTR [rbp-0x8018],0x0  
   0x00000000004007d4: cmp  QWORD PTR [rbp-0x8018],0x2000  
   0x00000000004007df: jae  0x400811  
   0x00000000004007f8: mov  rax,QWORD PTR [rbp-0x8018]  
   0x00000000004007ff: add  rax,0x1  
   0x0000000000400805: mov  QWORD PTR [rbp-0x8018],rax  
   0x000000000040080c: jmp  0x4007d4  
 8      {  
 9        blah2[i] = rand();  
   0x00000000004007e5: call  0x400690 <rand@plt>  
   0x00000000004007ea: mov  rcx,QWORD PTR [rbp-0x8018]  
   0x00000000004007f1: mov  DWORD PTR [rbp+rcx*4-0x8010],eax  
 10     }  
 11   }  
   0x0000000000400811: mov  eax,DWORD PTR [rbp-0x4]  
   0x0000000000400814: add  rsp,0x8020  
   0x000000000040081b: pop  rbp  
   0x000000000040081c: ret  

This is the straightforward disassemble call:

 (gdb) disassemble main  
 Dump of assembler code for function main(int, char**):  
   0x00000000004007b0 <+0>:   push  rbp  
   0x00000000004007b1 <+1>:   mov  rbp,rsp  
   0x00000000004007b4: sub  rsp,0x8020  
   0x00000000004007bb: mov  DWORD PTR [rbp-0x4],0x0  
   0x00000000004007c2: mov  DWORD PTR [rbp-0x8],edi  
   0x00000000004007c5: mov  QWORD PTR [rbp-0x10],rsi  
 => 0x00000000004007c9: mov  QWORD PTR [rbp-0x8018],0x0  
   0x00000000004007d4: cmp  QWORD PTR [rbp-0x8018],0x2000  
   0x00000000004007df: jae  0x400811  
   0x00000000004007e5: call  0x400690 <rand@plt>  
   0x00000000004007ea: mov  rcx,QWORD PTR [rbp-0x8018]  
   0x00000000004007f1: mov  DWORD PTR [rbp+rcx*4-0x8010],eax  
   0x00000000004007f8: mov  rax,QWORD PTR [rbp-0x8018]  
   0x00000000004007ff: add  rax,0x1  
   0x0000000000400805: mov  QWORD PTR [rbp-0x8018],rax  
   0x000000000040080c: jmp  0x4007d4  
   0x0000000000400811: mov  eax,DWORD PTR [rbp-0x4]  
   0x0000000000400814: add  rsp,0x8020  
   0x000000000040081b: pop  rbp  
   0x000000000040081c: ret  
 End of assembler dump.  

LLDB looks like the below. The two bold instructions are where the breakpoint locations are. I think I'm going to keep the current LLDB behavior on this one. Although I am going to add the ability to lowercase my registers and get hex constants...

 (lldb) disassemble -m -n main  
 blah`main at blah.cpp:5  
   4  int main( int argc, char *argv[] )  
   5  {  
   6    int blah2[8192];  
   0x4007b0: push  RBP  
   0x4007b1: mov  RBP, RSP  
   0x4007b4: sub  RSP, 32800  
   0x4007bb: mov  DWORD PTR [RBP - 4], 0  
   0x4007c2: mov  DWORD PTR [RBP - 8], EDI  
   0x4007c5: mov  QWORD PTR [RBP - 16], RSI  
 blah`main + 25 at blah.cpp:7  
   6    int blah2[8192];  
   7    for(size_t i = 0; i < 8192; ++i)  
   8    {  
   0x4007c9: mov  QWORD PTR [RBP - 32792], 0  
   0x4007d4: cmp  QWORD PTR [RBP - 32792], 8192  
   0x4007df: jae  0x400811         ; main + 97 at blah.cpp:11  
 blah`main + 53 at blah.cpp:9  
   8    {  
   9      blah2[i] = rand();  
   10    }  
   0x4007e5: call  0x400690         ; symbol stub for: rand  
   0x4007ea: mov  RCX, QWORD PTR [RBP - 32792]  
   0x4007f1: mov  DWORD PTR [RBP + 4*RCX - 32784], EAX  
 blah`main + 72 at blah.cpp:7  
   6    int blah2[8192];  
   7    for(size_t i = 0; i < 8192; ++i)  
   8    {  
   0x4007f8: mov  RAX, QWORD PTR [RBP - 32792]  
   0x4007ff: add  RAX, 1  
   0x400805: mov  QWORD PTR [RBP - 32792], RAX  
   0x40080c: jmpq  0x4007d4         ; main + 36 at blah.cpp:7  
 blah`main + 97 at blah.cpp:11  
   10    }  
   11  }  
   0x400811: mov  EAX, DWORD PTR [RBP - 4]  
   0x400814: add  RSP, 32800  
   0x40081b: pop  RBP  
   0x40081c: ret