1.. highlight:: none
  2
  3Debugging kernel and modules via gdb
  4====================================
  5
  6The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
  7interfaces allow to debug the Linux kernel and its modules during runtime
  8using gdb. Gdb comes with a powerful scripting interface for python. The
  9kernel provides a collection of helper scripts that can simplify typical
 10kernel debugging steps. This is a short tutorial about how to enable and use
 11them. It focuses on QEMU/KVM virtual machines as target, but the examples can
 12be transferred to the other gdb stubs as well.
 13
 14
 15Requirements
 16------------
 17
 18- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
 19  for distributions)
 20
 21
 22Setup
 23-----
 24
 25- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
 26  www.qemu.org for more details). For cross-development,
 27  https://landley.net/aboriginal/bin keeps a pool of machine images and
 28  toolchains that can be helpful to start from.
 29
 30- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
 31  CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
 32  CONFIG_FRAME_POINTER, keep it enabled.
 33
 34- Install that kernel on the guest, turn off KASLR if necessary by adding
 35  "nokaslr" to the kernel command line.
 36  Alternatively, QEMU allows to boot the kernel directly using -kernel,
 37  -append, -initrd command line switches. This is generally only useful if
 38  you do not depend on modules. See QEMU documentation for more details on
 39  this mode. In this case, you should build the kernel with
 40  CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
 41
 42- Build the gdb scripts (required on kernels v5.1 and above)::
 43
 44    make scripts_gdb
 45
 46- Enable the gdb stub of QEMU/KVM, either
 47
 48    - at VM startup time by appending "-s" to the QEMU command line
 49
 50  or
 51
 52    - during runtime by issuing "gdbserver" from the QEMU monitor
 53      console
 54
 55- cd /path/to/linux-build
 56
 57- Start gdb: gdb vmlinux
 58
 59  Note: Some distros may restrict auto-loading of gdb scripts to known safe
 60  directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
 61
 62    add-auto-load-safe-path /path/to/linux-build
 63
 64  to ~/.gdbinit. See gdb help for more details.
 65
 66- Attach to the booted guest::
 67
 68    (gdb) target remote :1234
 69
 70
 71Examples of using the Linux-provided gdb helpers
 72------------------------------------------------
 73
 74- Load module (and main kernel) symbols::
 75
 76    (gdb) lx-symbols
 77    loading vmlinux
 78    scanning for modules in /home/user/linux/build
 79    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
 80    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
 81    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
 82    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
 83    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
 84    ...
 85    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
 86
 87- Set a breakpoint on some not yet loaded module function, e.g.::
 88
 89    (gdb) b btrfs_init_sysfs
 90    Function "btrfs_init_sysfs" not defined.
 91    Make breakpoint pending on future shared library load? (y or [n]) y
 92    Breakpoint 1 (btrfs_init_sysfs) pending.
 93
 94- Continue the target::
 95
 96    (gdb) c
 97
 98- Load the module on the target and watch the symbols being loaded as well as
 99  the breakpoint hit::
100
101    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
102    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
103    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
104    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
105
106    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
107    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
108
109- Dump the log buffer of the target kernel::
110
111    (gdb) lx-dmesg
112    [     0.000000] Initializing cgroup subsys cpuset
113    [     0.000000] Initializing cgroup subsys cpu
114    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
115    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
116    [     0.000000] e820: BIOS-provided physical RAM map:
117    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
118    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
119    ....
120
121- Examine fields of the current task struct(supported by x86 and arm64 only)::
122
123    (gdb) p $lx_current().pid
124    $1 = 4998
125    (gdb) p $lx_current().comm
126    $2 = "modprobe\000\000\000\000\000\000\000"
127
128- Make use of the per-cpu function for the current or a specified CPU::
129
130    (gdb) p $lx_per_cpu("runqueues").nr_running
131    $3 = 1
132    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
133    $4 = 0
134
135- Dig into hrtimers using the container_of helper::
136
137    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
138    (gdb) p *$container_of($next, "struct hrtimer", "node")
139    $5 = {
140      node = {
141        node = {
142          __rb_parent_color = 18446612133355256072,
143          rb_right = 0x0 <irq_stack_union>,
144          rb_left = 0x0 <irq_stack_union>
145        },
146        expires = {
147          tv64 = 1835268000000
148        }
149      },
150      _softexpires = {
151        tv64 = 1835268000000
152      },
153      function = 0xffffffff81078232 <tick_sched_timer>,
154      base = 0xffff88003fd0d6f0,
155      state = 1,
156      start_pid = 0,
157      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
158      start_comm = "swapper/2\000\000\000\000\000\000"
159    }
160
161
162List of commands and functions
163------------------------------
164
165The number of commands and convenience functions may evolve over the time,
166this is just a snapshot of the initial version::
167
168 (gdb) apropos lx
169 function lx_current -- Return current task
170 function lx_module -- Find module by name and return the module variable
171 function lx_per_cpu -- Return per-cpu variable
172 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
173 function lx_thread_info -- Calculate Linux thread_info from task variable
174 lx-dmesg -- Print Linux kernel log buffer
175 lx-lsmod -- List currently loaded modules
176 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
177
178Detailed help can be obtained via "help <command-name>" for commands and "help
179function <function-name>" for convenience functions.