Loading...
1.. highlight:: none
2
3Debugging kernel and modules via gdb
4====================================
5
6The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
7interfaces allow to debug the Linux kernel and its modules during runtime
8using gdb. Gdb comes with a powerful scripting interface for python. The
9kernel provides a collection of helper scripts that can simplify typical
10kernel debugging steps. This is a short tutorial about how to enable and use
11them. It focuses on QEMU/KVM virtual machines as target, but the examples can
12be transferred to the other gdb stubs as well.
13
14
15Requirements
16------------
17
18- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
19 for distributions)
20
21
22Setup
23-----
24
25- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
26 www.qemu.org for more details). For cross-development,
27 https://landley.net/aboriginal/bin keeps a pool of machine images and
28 toolchains that can be helpful to start from.
29
30- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
31 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
32 CONFIG_FRAME_POINTER, keep it enabled.
33
34- Install that kernel on the guest, turn off KASLR if necessary by adding
35 "nokaslr" to the kernel command line.
36 Alternatively, QEMU allows to boot the kernel directly using -kernel,
37 -append, -initrd command line switches. This is generally only useful if
38 you do not depend on modules. See QEMU documentation for more details on
39 this mode. In this case, you should build the kernel with
40 CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
41
42- Enable the gdb stub of QEMU/KVM, either
43
44 - at VM startup time by appending "-s" to the QEMU command line
45
46 or
47
48 - during runtime by issuing "gdbserver" from the QEMU monitor
49 console
50
51- cd /path/to/linux-build
52
53- Start gdb: gdb vmlinux
54
55 Note: Some distros may restrict auto-loading of gdb scripts to known safe
56 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
57
58 add-auto-load-safe-path /path/to/linux-build
59
60 to ~/.gdbinit. See gdb help for more details.
61
62- Attach to the booted guest::
63
64 (gdb) target remote :1234
65
66
67Examples of using the Linux-provided gdb helpers
68------------------------------------------------
69
70- Load module (and main kernel) symbols::
71
72 (gdb) lx-symbols
73 loading vmlinux
74 scanning for modules in /home/user/linux/build
75 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
76 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
77 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
78 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
79 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
80 ...
81 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
82
83- Set a breakpoint on some not yet loaded module function, e.g.::
84
85 (gdb) b btrfs_init_sysfs
86 Function "btrfs_init_sysfs" not defined.
87 Make breakpoint pending on future shared library load? (y or [n]) y
88 Breakpoint 1 (btrfs_init_sysfs) pending.
89
90- Continue the target::
91
92 (gdb) c
93
94- Load the module on the target and watch the symbols being loaded as well as
95 the breakpoint hit::
96
97 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
98 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
99 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
100 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
101
102 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
103 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
104
105- Dump the log buffer of the target kernel::
106
107 (gdb) lx-dmesg
108 [ 0.000000] Initializing cgroup subsys cpuset
109 [ 0.000000] Initializing cgroup subsys cpu
110 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
111 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
112 [ 0.000000] e820: BIOS-provided physical RAM map:
113 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
114 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
115 ....
116
117- Examine fields of the current task struct(supported by x86 and arm64 only)::
118
119 (gdb) p $lx_current().pid
120 $1 = 4998
121 (gdb) p $lx_current().comm
122 $2 = "modprobe\000\000\000\000\000\000\000"
123
124- Make use of the per-cpu function for the current or a specified CPU::
125
126 (gdb) p $lx_per_cpu("runqueues").nr_running
127 $3 = 1
128 (gdb) p $lx_per_cpu("runqueues", 2).nr_running
129 $4 = 0
130
131- Dig into hrtimers using the container_of helper::
132
133 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
134 (gdb) p *$container_of($next, "struct hrtimer", "node")
135 $5 = {
136 node = {
137 node = {
138 __rb_parent_color = 18446612133355256072,
139 rb_right = 0x0 <irq_stack_union>,
140 rb_left = 0x0 <irq_stack_union>
141 },
142 expires = {
143 tv64 = 1835268000000
144 }
145 },
146 _softexpires = {
147 tv64 = 1835268000000
148 },
149 function = 0xffffffff81078232 <tick_sched_timer>,
150 base = 0xffff88003fd0d6f0,
151 state = 1,
152 start_pid = 0,
153 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
154 start_comm = "swapper/2\000\000\000\000\000\000"
155 }
156
157
158List of commands and functions
159------------------------------
160
161The number of commands and convenience functions may evolve over the time,
162this is just a snapshot of the initial version::
163
164 (gdb) apropos lx
165 function lx_current -- Return current task
166 function lx_module -- Find module by name and return the module variable
167 function lx_per_cpu -- Return per-cpu variable
168 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
169 function lx_thread_info -- Calculate Linux thread_info from task variable
170 lx-dmesg -- Print Linux kernel log buffer
171 lx-lsmod -- List currently loaded modules
172 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
173
174Detailed help can be obtained via "help <command-name>" for commands and "help
175function <function-name>" for convenience functions.
1.. highlight:: none
2
3Debugging kernel and modules via gdb
4====================================
5
6The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
7interfaces allow to debug the Linux kernel and its modules during runtime
8using gdb. Gdb comes with a powerful scripting interface for python. The
9kernel provides a collection of helper scripts that can simplify typical
10kernel debugging steps. This is a short tutorial about how to enable and use
11them. It focuses on QEMU/KVM virtual machines as target, but the examples can
12be transferred to the other gdb stubs as well.
13
14
15Requirements
16------------
17
18- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
19 for distributions)
20
21
22Setup
23-----
24
25- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
26 www.qemu.org for more details). For cross-development,
27 http://landley.net/aboriginal/bin keeps a pool of machine images and
28 toolchains that can be helpful to start from.
29
30- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
31 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
32 CONFIG_FRAME_POINTER, keep it enabled.
33
34- Install that kernel on the guest.
35 Alternatively, QEMU allows to boot the kernel directly using -kernel,
36 -append, -initrd command line switches. This is generally only useful if
37 you do not depend on modules. See QEMU documentation for more details on
38 this mode.
39
40- Enable the gdb stub of QEMU/KVM, either
41
42 - at VM startup time by appending "-s" to the QEMU command line
43
44 or
45
46 - during runtime by issuing "gdbserver" from the QEMU monitor
47 console
48
49- cd /path/to/linux-build
50
51- Start gdb: gdb vmlinux
52
53 Note: Some distros may restrict auto-loading of gdb scripts to known safe
54 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
55
56 add-auto-load-safe-path /path/to/linux-build
57
58 to ~/.gdbinit. See gdb help for more details.
59
60- Attach to the booted guest::
61
62 (gdb) target remote :1234
63
64
65Examples of using the Linux-provided gdb helpers
66------------------------------------------------
67
68- Load module (and main kernel) symbols::
69
70 (gdb) lx-symbols
71 loading vmlinux
72 scanning for modules in /home/user/linux/build
73 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
74 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
75 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
76 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
77 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
78 ...
79 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
80
81- Set a breakpoint on some not yet loaded module function, e.g.::
82
83 (gdb) b btrfs_init_sysfs
84 Function "btrfs_init_sysfs" not defined.
85 Make breakpoint pending on future shared library load? (y or [n]) y
86 Breakpoint 1 (btrfs_init_sysfs) pending.
87
88- Continue the target::
89
90 (gdb) c
91
92- Load the module on the target and watch the symbols being loaded as well as
93 the breakpoint hit::
94
95 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
96 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
97 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
98 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
99
100 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
101 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
102
103- Dump the log buffer of the target kernel::
104
105 (gdb) lx-dmesg
106 [ 0.000000] Initializing cgroup subsys cpuset
107 [ 0.000000] Initializing cgroup subsys cpu
108 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
109 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
110 [ 0.000000] e820: BIOS-provided physical RAM map:
111 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
112 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
113 ....
114
115- Examine fields of the current task struct::
116
117 (gdb) p $lx_current().pid
118 $1 = 4998
119 (gdb) p $lx_current().comm
120 $2 = "modprobe\000\000\000\000\000\000\000"
121
122- Make use of the per-cpu function for the current or a specified CPU::
123
124 (gdb) p $lx_per_cpu("runqueues").nr_running
125 $3 = 1
126 (gdb) p $lx_per_cpu("runqueues", 2).nr_running
127 $4 = 0
128
129- Dig into hrtimers using the container_of helper::
130
131 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
132 (gdb) p *$container_of($next, "struct hrtimer", "node")
133 $5 = {
134 node = {
135 node = {
136 __rb_parent_color = 18446612133355256072,
137 rb_right = 0x0 <irq_stack_union>,
138 rb_left = 0x0 <irq_stack_union>
139 },
140 expires = {
141 tv64 = 1835268000000
142 }
143 },
144 _softexpires = {
145 tv64 = 1835268000000
146 },
147 function = 0xffffffff81078232 <tick_sched_timer>,
148 base = 0xffff88003fd0d6f0,
149 state = 1,
150 start_pid = 0,
151 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
152 start_comm = "swapper/2\000\000\000\000\000\000"
153 }
154
155
156List of commands and functions
157------------------------------
158
159The number of commands and convenience functions may evolve over the time,
160this is just a snapshot of the initial version::
161
162 (gdb) apropos lx
163 function lx_current -- Return current task
164 function lx_module -- Find module by name and return the module variable
165 function lx_per_cpu -- Return per-cpu variable
166 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
167 function lx_thread_info -- Calculate Linux thread_info from task variable
168 lx-dmesg -- Print Linux kernel log buffer
169 lx-lsmod -- List currently loaded modules
170 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
171
172Detailed help can be obtained via "help <command-name>" for commands and "help
173function <function-name>" for convenience functions.