Loading...
1===========================================
2Fault injection capabilities infrastructure
3===========================================
4
5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
6
7
8Available fault injection capabilities
9--------------------------------------
10
11- failslab
12
13 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
14
15- fail_page_alloc
16
17 injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
18
19- fail_usercopy
20
21 injects failures in user memory access functions. (copy_from_user(), get_user(), ...)
22
23- fail_futex
24
25 injects futex deadlock and uaddr fault errors.
26
27- fail_sunrpc
28
29 injects kernel RPC client and server failures.
30
31- fail_make_request
32
33 injects disk IO errors on devices permitted by setting
34 /sys/block/<device>/make-it-fail or
35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
36
37- fail_mmc_request
38
39 injects MMC data errors on devices permitted by setting
40 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
41
42- fail_function
43
44 injects error return on specific functions, which are marked by
45 ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
46 under /sys/kernel/debug/fail_function. No boot option supported.
47
48- fail_skb_realloc
49
50 inject skb (socket buffer) reallocation events into the network path. The
51 primary goal is to identify and prevent issues related to pointer
52 mismanagement in the network subsystem. By forcing skb reallocation at
53 strategic points, this feature creates scenarios where existing pointers to
54 skb headers become invalid.
55
56 When the fault is injected and the reallocation is triggered, cached pointers
57 to skb headers and data no longer reference valid memory locations. This
58 deliberate invalidation helps expose code paths where proper pointer updating
59 is neglected after a reallocation event.
60
61 By creating these controlled fault scenarios, the system can catch instances
62 where stale pointers are used, potentially leading to memory corruption or
63 system instability.
64
65 To select the interface to act on, write the network name to
66 /sys/kernel/debug/fail_skb_realloc/devname.
67 If this field is left empty (which is the default value), skb reallocation
68 will be forced on all network interfaces.
69
70 The effectiveness of this fault detection is enhanced when KASAN is
71 enabled, as it helps identify invalid memory references and use-after-free
72 (UAF) issues.
73
74- NVMe fault injection
75
76 inject NVMe status code and retry flag on devices permitted by setting
77 debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
78 status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
79 retry flag can be set via the debugfs.
80
81- Null test block driver fault injection
82
83 inject IO timeouts by setting config items under
84 /sys/kernel/config/nullb/<disk>/timeout_inject,
85 inject requeue requests by setting config items under
86 /sys/kernel/config/nullb/<disk>/requeue_inject, and
87 inject init_hctx() errors by setting config items under
88 /sys/kernel/config/nullb/<disk>/init_hctx_fault_inject.
89
90Configure fault-injection capabilities behavior
91-----------------------------------------------
92
93debugfs entries
94^^^^^^^^^^^^^^^
95
96fault-inject-debugfs kernel module provides some debugfs entries for runtime
97configuration of fault-injection capabilities.
98
99- /sys/kernel/debug/fail*/probability:
100
101 likelihood of failure injection, in percent.
102
103 Format: <percent>
104
105 Note that one-failure-per-hundred is a very high error rate
106 for some testcases. Consider setting probability=100 and configure
107 /sys/kernel/debug/fail*/interval for such testcases.
108
109- /sys/kernel/debug/fail*/interval:
110
111 specifies the interval between failures, for calls to
112 should_fail() that pass all the other tests.
113
114 Note that if you enable this, by setting interval>1, you will
115 probably want to set probability=100.
116
117- /sys/kernel/debug/fail*/times:
118
119 specifies how many times failures may happen at most. A value of -1
120 means "no limit".
121
122- /sys/kernel/debug/fail*/space:
123
124 specifies an initial resource "budget", decremented by "size"
125 on each call to should_fail(,size). Failure injection is
126 suppressed until "space" reaches zero.
127
128- /sys/kernel/debug/fail*/verbose
129
130 Format: { 0 | 1 | 2 }
131
132 specifies the verbosity of the messages when failure is
133 injected. '0' means no messages; '1' will print only a single
134 log line per failure; '2' will print a call trace too -- useful
135 to debug the problems revealed by fault injection.
136
137- /sys/kernel/debug/fail*/task-filter:
138
139 Format: { 'Y' | 'N' }
140
141 A value of 'N' disables filtering by process (default).
142 Any positive value limits failures to only processes indicated by
143 /proc/<pid>/make-it-fail==1.
144
145- /sys/kernel/debug/fail*/require-start,
146 /sys/kernel/debug/fail*/require-end,
147 /sys/kernel/debug/fail*/reject-start,
148 /sys/kernel/debug/fail*/reject-end:
149
150 specifies the range of virtual addresses tested during
151 stacktrace walking. Failure is injected only if some caller
152 in the walked stacktrace lies within the required range, and
153 none lies within the rejected range.
154 Default required range is [0,ULONG_MAX) (whole of virtual address space).
155 Default rejected range is [0,0).
156
157- /sys/kernel/debug/fail*/stacktrace-depth:
158
159 specifies the maximum stacktrace depth walked during search
160 for a caller within [require-start,require-end) OR
161 [reject-start,reject-end).
162
163- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
164
165 Format: { 'Y' | 'N' }
166
167 default is 'Y', setting it to 'N' will also inject failures into
168 highmem/user allocations (__GFP_HIGHMEM allocations).
169
170- /sys/kernel/debug/failslab/cache-filter
171 Format: { 'Y' | 'N' }
172
173 default is 'N', setting it to 'Y' will only inject failures when
174 objects are requests from certain caches.
175
176 Select the cache by writing '1' to /sys/kernel/slab/<cache>/failslab:
177
178- /sys/kernel/debug/failslab/ignore-gfp-wait:
179- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
180
181 Format: { 'Y' | 'N' }
182
183 default is 'Y', setting it to 'N' will also inject failures
184 into allocations that can sleep (__GFP_DIRECT_RECLAIM allocations).
185
186- /sys/kernel/debug/fail_page_alloc/min-order:
187
188 specifies the minimum page allocation order to be injected
189 failures.
190
191- /sys/kernel/debug/fail_futex/ignore-private:
192
193 Format: { 'Y' | 'N' }
194
195 default is 'N', setting it to 'Y' will disable failure injections
196 when dealing with private (address space) futexes.
197
198- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
199
200 Format: { 'Y' | 'N' }
201
202 default is 'N', setting it to 'Y' will disable disconnect
203 injection on the RPC client.
204
205- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
206
207 Format: { 'Y' | 'N' }
208
209 default is 'N', setting it to 'Y' will disable disconnect
210 injection on the RPC server.
211
212- /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
213
214 Format: { 'Y' | 'N' }
215
216 default is 'N', setting it to 'Y' will disable cache wait
217 injection on the RPC server.
218
219- /sys/kernel/debug/fail_function/inject:
220
221 Format: { 'function-name' | '!function-name' | '' }
222
223 specifies the target function of error injection by name.
224 If the function name leads '!' prefix, given function is
225 removed from injection list. If nothing specified ('')
226 injection list is cleared.
227
228- /sys/kernel/debug/fail_function/injectable:
229
230 (read only) shows error injectable functions and what type of
231 error values can be specified. The error type will be one of
232 below;
233 - NULL: retval must be 0.
234 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
235 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
236
237- /sys/kernel/debug/fail_function/<function-name>/retval:
238
239 specifies the "error" return value to inject to the given function.
240 This will be created when the user specifies a new injection entry.
241 Note that this file only accepts unsigned values. So, if you want to
242 use a negative errno, you better use 'printf' instead of 'echo', e.g.:
243 $ printf %#x -12 > retval
244
245- /sys/kernel/debug/fail_skb_realloc/devname:
246
247 Specifies the network interface on which to force SKB reallocation. If
248 left empty, SKB reallocation will be applied to all network interfaces.
249
250 Example usage::
251
252 # Force skb reallocation on eth0
253 echo "eth0" > /sys/kernel/debug/fail_skb_realloc/devname
254
255 # Clear the selection and force skb reallocation on all interfaces
256 echo "" > /sys/kernel/debug/fail_skb_realloc/devname
257
258Boot option
259^^^^^^^^^^^
260
261In order to inject faults while debugfs is not available (early boot time),
262use the boot option::
263
264 failslab=
265 fail_page_alloc=
266 fail_usercopy=
267 fail_make_request=
268 fail_futex=
269 fail_skb_realloc=
270 mmc_core.fail_request=<interval>,<probability>,<space>,<times>
271
272proc entries
273^^^^^^^^^^^^
274
275- /proc/<pid>/fail-nth,
276 /proc/self/task/<tid>/fail-nth:
277
278 Write to this file of integer N makes N-th call in the task fail.
279 Read from this file returns a integer value. A value of '0' indicates
280 that the fault setup with a previous write to this file was injected.
281 A positive integer N indicates that the fault wasn't yet injected.
282 Note that this file enables all types of faults (slab, futex, etc).
283 This setting takes precedence over all other generic debugfs settings
284 like probability, interval, times, etc. But per-capability settings
285 (e.g. fail_futex/ignore-private) take precedence over it.
286
287 This feature is intended for systematic testing of faults in a single
288 system call. See an example below.
289
290
291Error Injectable Functions
292--------------------------
293
294This part is for the kernel developers considering to add a function to
295ALLOW_ERROR_INJECTION() macro.
296
297Requirements for the Error Injectable Functions
298^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
299
300Since the function-level error injection forcibly changes the code path
301and returns an error even if the input and conditions are proper, this can
302cause unexpected kernel crash if you allow error injection on the function
303which is NOT error injectable. Thus, you (and reviewers) must ensure;
304
305- The function returns an error code if it fails, and the callers must check
306 it correctly (need to recover from it).
307
308- The function does not execute any code which can change any state before
309 the first error return. The state includes global or local, or input
310 variable. For example, clear output address storage (e.g. `*ret = NULL`),
311 increments/decrements counter, set a flag, preempt/irq disable or get
312 a lock (if those are recovered before returning error, that will be OK.)
313
314The first requirement is important, and it will result in that the release
315(free objects) functions are usually harder to inject errors than allocate
316functions. If errors of such release functions are not correctly handled
317it will cause a memory leak easily (the caller will confuse that the object
318has been released or corrupted.)
319
320The second one is for the caller which expects the function should always
321does something. Thus if the function error injection skips whole of the
322function, the expectation is betrayed and causes an unexpected error.
323
324Type of the Error Injectable Functions
325^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
326
327Each error injectable functions will have the error type specified by the
328ALLOW_ERROR_INJECTION() macro. You have to choose it carefully if you add
329a new error injectable function. If the wrong error type is chosen, the
330kernel may crash because it may not be able to handle the error.
331There are 4 types of errors defined in include/asm-generic/error-injection.h
332
333EI_ETYPE_NULL
334 This function will return `NULL` if it fails. e.g. return an allocated
335 object address.
336
337EI_ETYPE_ERRNO
338 This function will return an `-errno` error code if it fails. e.g. return
339 -EINVAL if the input is wrong. This will include the functions which will
340 return an address which encodes `-errno` by ERR_PTR() macro.
341
342EI_ETYPE_ERRNO_NULL
343 This function will return an `-errno` or `NULL` if it fails. If the caller
344 of this function checks the return value with IS_ERR_OR_NULL() macro, this
345 type will be appropriate.
346
347EI_ETYPE_TRUE
348 This function will return `true` (non-zero positive value) if it fails.
349
350If you specifies a wrong type, for example, EI_TYPE_ERRNO for the function
351which returns an allocated object, it may cause a problem because the returned
352value is not an object address and the caller can not access to the address.
353
354
355How to add new fault injection capability
356-----------------------------------------
357
358- #include <linux/fault-inject.h>
359
360- define the fault attributes
361
362 DECLARE_FAULT_ATTR(name);
363
364 Please see the definition of struct fault_attr in fault-inject.h
365 for details.
366
367- provide a way to configure fault attributes
368
369- boot option
370
371 If you need to enable the fault injection capability from boot time, you can
372 provide boot option to configure it. There is a helper function for it:
373
374 setup_fault_attr(attr, str);
375
376- debugfs entries
377
378 failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way.
379 Helper functions:
380
381 fault_create_debugfs_attr(name, parent, attr);
382
383- module parameters
384
385 If the scope of the fault injection capability is limited to a
386 single kernel module, it is better to provide module parameters to
387 configure the fault attributes.
388
389- add a hook to insert failures
390
391 Upon should_fail() returning true, client code should inject a failure:
392
393 should_fail(attr, size);
394
395Application Examples
396--------------------
397
398- Inject slab allocation failures into module init/exit code::
399
400 #!/bin/bash
401
402 FAILTYPE=failslab
403 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
404 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
405 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
406 echo -1 > /sys/kernel/debug/$FAILTYPE/times
407 echo 0 > /sys/kernel/debug/$FAILTYPE/space
408 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
409 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
410
411 faulty_system()
412 {
413 bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
414 }
415
416 if [ $# -eq 0 ]
417 then
418 echo "Usage: $0 modulename [ modulename ... ]"
419 exit 1
420 fi
421
422 for m in $*
423 do
424 echo inserting $m...
425 faulty_system modprobe $m
426
427 echo removing $m...
428 faulty_system modprobe -r $m
429 done
430
431------------------------------------------------------------------------------
432
433- Inject page allocation failures only for a specific module::
434
435 #!/bin/bash
436
437 FAILTYPE=fail_page_alloc
438 module=$1
439
440 if [ -z $module ]
441 then
442 echo "Usage: $0 <modulename>"
443 exit 1
444 fi
445
446 modprobe $module
447
448 if [ ! -d /sys/module/$module/sections ]
449 then
450 echo Module $module is not loaded
451 exit 1
452 fi
453
454 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
455 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
456
457 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
458 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
459 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
460 echo -1 > /sys/kernel/debug/$FAILTYPE/times
461 echo 0 > /sys/kernel/debug/$FAILTYPE/space
462 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
463 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
464 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
465 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
466
467 trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
468
469 echo "Injecting errors into the module $module... (interrupt to stop)"
470 sleep 1000000
471
472------------------------------------------------------------------------------
473
474- Inject open_ctree error while btrfs mount::
475
476 #!/bin/bash
477
478 rm -f testfile.img
479 dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
480 DEVICE=$(losetup --show -f testfile.img)
481 mkfs.btrfs -f $DEVICE
482 mkdir -p tmpmnt
483
484 FAILTYPE=fail_function
485 FAILFUNC=open_ctree
486 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
487 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
488 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
489 echo 100 > /sys/kernel/debug/$FAILTYPE/probability
490 echo 0 > /sys/kernel/debug/$FAILTYPE/interval
491 echo -1 > /sys/kernel/debug/$FAILTYPE/times
492 echo 0 > /sys/kernel/debug/$FAILTYPE/space
493 echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
494
495 mount -t btrfs $DEVICE tmpmnt
496 if [ $? -ne 0 ]
497 then
498 echo "SUCCESS!"
499 else
500 echo "FAILED!"
501 umount tmpmnt
502 fi
503
504 echo > /sys/kernel/debug/$FAILTYPE/inject
505
506 rmdir tmpmnt
507 losetup -d $DEVICE
508 rm testfile.img
509
510------------------------------------------------------------------------------
511
512- Inject only skbuff allocation failures ::
513
514 # mark skbuff_head_cache as faulty
515 echo 1 > /sys/kernel/slab/skbuff_head_cache/failslab
516 # Turn on cache filter (off by default)
517 echo 1 > /sys/kernel/debug/failslab/cache-filter
518 # Turn on fault injection
519 echo 1 > /sys/kernel/debug/failslab/times
520 echo 1 > /sys/kernel/debug/failslab/probability
521
522
523Tool to run command with failslab or fail_page_alloc
524----------------------------------------------------
525In order to make it easier to accomplish the tasks mentioned above, we can use
526tools/testing/fault-injection/failcmd.sh. Please run a command
527"./tools/testing/fault-injection/failcmd.sh --help" for more information and
528see the following examples.
529
530Examples:
531
532Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
533allocation failure::
534
535 # ./tools/testing/fault-injection/failcmd.sh \
536 -- make -C tools/testing/selftests/ run_tests
537
538Same as above except to specify 100 times failures at most instead of one time
539at most by default::
540
541 # ./tools/testing/fault-injection/failcmd.sh --times=100 \
542 -- make -C tools/testing/selftests/ run_tests
543
544Same as above except to inject page allocation failure instead of slab
545allocation failure::
546
547 # env FAILCMD_TYPE=fail_page_alloc \
548 ./tools/testing/fault-injection/failcmd.sh --times=100 \
549 -- make -C tools/testing/selftests/ run_tests
550
551Systematic faults using fail-nth
552---------------------------------
553
554The following code systematically faults 0-th, 1-st, 2-nd and so on
555capabilities in the socketpair() system call::
556
557 #include <sys/types.h>
558 #include <sys/stat.h>
559 #include <sys/socket.h>
560 #include <sys/syscall.h>
561 #include <fcntl.h>
562 #include <unistd.h>
563 #include <string.h>
564 #include <stdlib.h>
565 #include <stdio.h>
566 #include <errno.h>
567
568 int main()
569 {
570 int i, err, res, fail_nth, fds[2];
571 char buf[128];
572
573 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
574 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
575 fail_nth = open(buf, O_RDWR);
576 for (i = 1;; i++) {
577 sprintf(buf, "%d", i);
578 write(fail_nth, buf, strlen(buf));
579 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
580 err = errno;
581 pread(fail_nth, buf, sizeof(buf), 0);
582 if (res == 0) {
583 close(fds[0]);
584 close(fds[1]);
585 }
586 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
587 res, err);
588 if (atoi(buf))
589 break;
590 }
591 return 0;
592 }
593
594An example output::
595
596 1-th fault Y: res=-1/23
597 2-th fault Y: res=-1/23
598 3-th fault Y: res=-1/12
599 4-th fault Y: res=-1/12
600 5-th fault Y: res=-1/23
601 6-th fault Y: res=-1/23
602 7-th fault Y: res=-1/23
603 8-th fault Y: res=-1/12
604 9-th fault Y: res=-1/12
605 10-th fault Y: res=-1/12
606 11-th fault Y: res=-1/12
607 12-th fault Y: res=-1/12
608 13-th fault Y: res=-1/12
609 14-th fault Y: res=-1/12
610 15-th fault Y: res=-1/12
611 16-th fault N: res=0/12
1===========================================
2Fault injection capabilities infrastructure
3===========================================
4
5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
6
7
8Available fault injection capabilities
9--------------------------------------
10
11- failslab
12
13 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
14
15- fail_page_alloc
16
17 injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
18
19- fail_usercopy
20
21 injects failures in user memory access functions. (copy_from_user(), get_user(), ...)
22
23- fail_futex
24
25 injects futex deadlock and uaddr fault errors.
26
27- fail_sunrpc
28
29 injects kernel RPC client and server failures.
30
31- fail_make_request
32
33 injects disk IO errors on devices permitted by setting
34 /sys/block/<device>/make-it-fail or
35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
36
37- fail_mmc_request
38
39 injects MMC data errors on devices permitted by setting
40 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
41
42- fail_function
43
44 injects error return on specific functions, which are marked by
45 ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
46 under /sys/kernel/debug/fail_function. No boot option supported.
47
48- NVMe fault injection
49
50 inject NVMe status code and retry flag on devices permitted by setting
51 debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
52 status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
53 retry flag can be set via the debugfs.
54
55
56Configure fault-injection capabilities behavior
57-----------------------------------------------
58
59debugfs entries
60^^^^^^^^^^^^^^^
61
62fault-inject-debugfs kernel module provides some debugfs entries for runtime
63configuration of fault-injection capabilities.
64
65- /sys/kernel/debug/fail*/probability:
66
67 likelihood of failure injection, in percent.
68
69 Format: <percent>
70
71 Note that one-failure-per-hundred is a very high error rate
72 for some testcases. Consider setting probability=100 and configure
73 /sys/kernel/debug/fail*/interval for such testcases.
74
75- /sys/kernel/debug/fail*/interval:
76
77 specifies the interval between failures, for calls to
78 should_fail() that pass all the other tests.
79
80 Note that if you enable this, by setting interval>1, you will
81 probably want to set probability=100.
82
83- /sys/kernel/debug/fail*/times:
84
85 specifies how many times failures may happen at most. A value of -1
86 means "no limit".
87
88- /sys/kernel/debug/fail*/space:
89
90 specifies an initial resource "budget", decremented by "size"
91 on each call to should_fail(,size). Failure injection is
92 suppressed until "space" reaches zero.
93
94- /sys/kernel/debug/fail*/verbose
95
96 Format: { 0 | 1 | 2 }
97
98 specifies the verbosity of the messages when failure is
99 injected. '0' means no messages; '1' will print only a single
100 log line per failure; '2' will print a call trace too -- useful
101 to debug the problems revealed by fault injection.
102
103- /sys/kernel/debug/fail*/task-filter:
104
105 Format: { 'Y' | 'N' }
106
107 A value of 'N' disables filtering by process (default).
108 Any positive value limits failures to only processes indicated by
109 /proc/<pid>/make-it-fail==1.
110
111- /sys/kernel/debug/fail*/require-start,
112 /sys/kernel/debug/fail*/require-end,
113 /sys/kernel/debug/fail*/reject-start,
114 /sys/kernel/debug/fail*/reject-end:
115
116 specifies the range of virtual addresses tested during
117 stacktrace walking. Failure is injected only if some caller
118 in the walked stacktrace lies within the required range, and
119 none lies within the rejected range.
120 Default required range is [0,ULONG_MAX) (whole of virtual address space).
121 Default rejected range is [0,0).
122
123- /sys/kernel/debug/fail*/stacktrace-depth:
124
125 specifies the maximum stacktrace depth walked during search
126 for a caller within [require-start,require-end) OR
127 [reject-start,reject-end).
128
129- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
130
131 Format: { 'Y' | 'N' }
132
133 default is 'Y', setting it to 'N' will also inject failures into
134 highmem/user allocations (__GFP_HIGHMEM allocations).
135
136- /sys/kernel/debug/failslab/ignore-gfp-wait:
137- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
138
139 Format: { 'Y' | 'N' }
140
141 default is 'Y', setting it to 'N' will also inject failures
142 into allocations that can sleep (__GFP_DIRECT_RECLAIM allocations).
143
144- /sys/kernel/debug/fail_page_alloc/min-order:
145
146 specifies the minimum page allocation order to be injected
147 failures.
148
149- /sys/kernel/debug/fail_futex/ignore-private:
150
151 Format: { 'Y' | 'N' }
152
153 default is 'N', setting it to 'Y' will disable failure injections
154 when dealing with private (address space) futexes.
155
156- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
157
158 Format: { 'Y' | 'N' }
159
160 default is 'N', setting it to 'Y' will disable disconnect
161 injection on the RPC client.
162
163- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
164
165 Format: { 'Y' | 'N' }
166
167 default is 'N', setting it to 'Y' will disable disconnect
168 injection on the RPC server.
169
170- /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
171
172 Format: { 'Y' | 'N' }
173
174 default is 'N', setting it to 'Y' will disable cache wait
175 injection on the RPC server.
176
177- /sys/kernel/debug/fail_function/inject:
178
179 Format: { 'function-name' | '!function-name' | '' }
180
181 specifies the target function of error injection by name.
182 If the function name leads '!' prefix, given function is
183 removed from injection list. If nothing specified ('')
184 injection list is cleared.
185
186- /sys/kernel/debug/fail_function/injectable:
187
188 (read only) shows error injectable functions and what type of
189 error values can be specified. The error type will be one of
190 below;
191 - NULL: retval must be 0.
192 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
193 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
194
195- /sys/kernel/debug/fail_function/<function-name>/retval:
196
197 specifies the "error" return value to inject to the given function.
198 This will be created when the user specifies a new injection entry.
199 Note that this file only accepts unsigned values. So, if you want to
200 use a negative errno, you better use 'printf' instead of 'echo', e.g.:
201 $ printf %#x -12 > retval
202
203Boot option
204^^^^^^^^^^^
205
206In order to inject faults while debugfs is not available (early boot time),
207use the boot option::
208
209 failslab=
210 fail_page_alloc=
211 fail_usercopy=
212 fail_make_request=
213 fail_futex=
214 mmc_core.fail_request=<interval>,<probability>,<space>,<times>
215
216proc entries
217^^^^^^^^^^^^
218
219- /proc/<pid>/fail-nth,
220 /proc/self/task/<tid>/fail-nth:
221
222 Write to this file of integer N makes N-th call in the task fail.
223 Read from this file returns a integer value. A value of '0' indicates
224 that the fault setup with a previous write to this file was injected.
225 A positive integer N indicates that the fault wasn't yet injected.
226 Note that this file enables all types of faults (slab, futex, etc).
227 This setting takes precedence over all other generic debugfs settings
228 like probability, interval, times, etc. But per-capability settings
229 (e.g. fail_futex/ignore-private) take precedence over it.
230
231 This feature is intended for systematic testing of faults in a single
232 system call. See an example below.
233
234How to add new fault injection capability
235-----------------------------------------
236
237- #include <linux/fault-inject.h>
238
239- define the fault attributes
240
241 DECLARE_FAULT_ATTR(name);
242
243 Please see the definition of struct fault_attr in fault-inject.h
244 for details.
245
246- provide a way to configure fault attributes
247
248- boot option
249
250 If you need to enable the fault injection capability from boot time, you can
251 provide boot option to configure it. There is a helper function for it:
252
253 setup_fault_attr(attr, str);
254
255- debugfs entries
256
257 failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way.
258 Helper functions:
259
260 fault_create_debugfs_attr(name, parent, attr);
261
262- module parameters
263
264 If the scope of the fault injection capability is limited to a
265 single kernel module, it is better to provide module parameters to
266 configure the fault attributes.
267
268- add a hook to insert failures
269
270 Upon should_fail() returning true, client code should inject a failure:
271
272 should_fail(attr, size);
273
274Application Examples
275--------------------
276
277- Inject slab allocation failures into module init/exit code::
278
279 #!/bin/bash
280
281 FAILTYPE=failslab
282 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
283 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
284 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
285 echo -1 > /sys/kernel/debug/$FAILTYPE/times
286 echo 0 > /sys/kernel/debug/$FAILTYPE/space
287 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
288 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
289
290 faulty_system()
291 {
292 bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
293 }
294
295 if [ $# -eq 0 ]
296 then
297 echo "Usage: $0 modulename [ modulename ... ]"
298 exit 1
299 fi
300
301 for m in $*
302 do
303 echo inserting $m...
304 faulty_system modprobe $m
305
306 echo removing $m...
307 faulty_system modprobe -r $m
308 done
309
310------------------------------------------------------------------------------
311
312- Inject page allocation failures only for a specific module::
313
314 #!/bin/bash
315
316 FAILTYPE=fail_page_alloc
317 module=$1
318
319 if [ -z $module ]
320 then
321 echo "Usage: $0 <modulename>"
322 exit 1
323 fi
324
325 modprobe $module
326
327 if [ ! -d /sys/module/$module/sections ]
328 then
329 echo Module $module is not loaded
330 exit 1
331 fi
332
333 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
334 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
335
336 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
337 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
338 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
339 echo -1 > /sys/kernel/debug/$FAILTYPE/times
340 echo 0 > /sys/kernel/debug/$FAILTYPE/space
341 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
342 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
343 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
344 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
345
346 trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
347
348 echo "Injecting errors into the module $module... (interrupt to stop)"
349 sleep 1000000
350
351------------------------------------------------------------------------------
352
353- Inject open_ctree error while btrfs mount::
354
355 #!/bin/bash
356
357 rm -f testfile.img
358 dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
359 DEVICE=$(losetup --show -f testfile.img)
360 mkfs.btrfs -f $DEVICE
361 mkdir -p tmpmnt
362
363 FAILTYPE=fail_function
364 FAILFUNC=open_ctree
365 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
366 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
367 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
368 echo 100 > /sys/kernel/debug/$FAILTYPE/probability
369 echo 0 > /sys/kernel/debug/$FAILTYPE/interval
370 echo -1 > /sys/kernel/debug/$FAILTYPE/times
371 echo 0 > /sys/kernel/debug/$FAILTYPE/space
372 echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
373
374 mount -t btrfs $DEVICE tmpmnt
375 if [ $? -ne 0 ]
376 then
377 echo "SUCCESS!"
378 else
379 echo "FAILED!"
380 umount tmpmnt
381 fi
382
383 echo > /sys/kernel/debug/$FAILTYPE/inject
384
385 rmdir tmpmnt
386 losetup -d $DEVICE
387 rm testfile.img
388
389
390Tool to run command with failslab or fail_page_alloc
391----------------------------------------------------
392In order to make it easier to accomplish the tasks mentioned above, we can use
393tools/testing/fault-injection/failcmd.sh. Please run a command
394"./tools/testing/fault-injection/failcmd.sh --help" for more information and
395see the following examples.
396
397Examples:
398
399Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
400allocation failure::
401
402 # ./tools/testing/fault-injection/failcmd.sh \
403 -- make -C tools/testing/selftests/ run_tests
404
405Same as above except to specify 100 times failures at most instead of one time
406at most by default::
407
408 # ./tools/testing/fault-injection/failcmd.sh --times=100 \
409 -- make -C tools/testing/selftests/ run_tests
410
411Same as above except to inject page allocation failure instead of slab
412allocation failure::
413
414 # env FAILCMD_TYPE=fail_page_alloc \
415 ./tools/testing/fault-injection/failcmd.sh --times=100 \
416 -- make -C tools/testing/selftests/ run_tests
417
418Systematic faults using fail-nth
419---------------------------------
420
421The following code systematically faults 0-th, 1-st, 2-nd and so on
422capabilities in the socketpair() system call::
423
424 #include <sys/types.h>
425 #include <sys/stat.h>
426 #include <sys/socket.h>
427 #include <sys/syscall.h>
428 #include <fcntl.h>
429 #include <unistd.h>
430 #include <string.h>
431 #include <stdlib.h>
432 #include <stdio.h>
433 #include <errno.h>
434
435 int main()
436 {
437 int i, err, res, fail_nth, fds[2];
438 char buf[128];
439
440 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
441 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
442 fail_nth = open(buf, O_RDWR);
443 for (i = 1;; i++) {
444 sprintf(buf, "%d", i);
445 write(fail_nth, buf, strlen(buf));
446 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
447 err = errno;
448 pread(fail_nth, buf, sizeof(buf), 0);
449 if (res == 0) {
450 close(fds[0]);
451 close(fds[1]);
452 }
453 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
454 res, err);
455 if (atoi(buf))
456 break;
457 }
458 return 0;
459 }
460
461An example output::
462
463 1-th fault Y: res=-1/23
464 2-th fault Y: res=-1/23
465 3-th fault Y: res=-1/12
466 4-th fault Y: res=-1/12
467 5-th fault Y: res=-1/23
468 6-th fault Y: res=-1/23
469 7-th fault Y: res=-1/23
470 8-th fault Y: res=-1/12
471 9-th fault Y: res=-1/12
472 10-th fault Y: res=-1/12
473 11-th fault Y: res=-1/12
474 12-th fault Y: res=-1/12
475 13-th fault Y: res=-1/12
476 14-th fault Y: res=-1/12
477 15-th fault Y: res=-1/12
478 16-th fault N: res=0/12