Loading...
1========
2hwpoison
3========
4
5What is hwpoison?
6=================
7
8Upcoming Intel CPUs have support for recovering from some memory errors
9(``MCA recovery``). This requires the OS to declare a page "poisoned",
10kill the processes associated with it and avoid using it in the future.
11
12This patchkit implements the necessary infrastructure in the VM.
13
14To quote the overview comment::
15
16 High level machine check handler. Handles pages reported by the
17 hardware as being corrupted usually due to a 2bit ECC memory or cache
18 failure.
19
20 This focusses on pages detected as corrupted in the background.
21 When the current CPU tries to consume corruption the currently
22 running process can just be killed directly instead. This implies
23 that if the error cannot be handled for some reason it's safe to
24 just ignore it because no corruption has been consumed yet. Instead
25 when that happens another machine check will happen.
26
27 Handles page cache pages in various states. The tricky part
28 here is that we can access any page asynchronous to other VM
29 users, because memory failures could happen anytime and anywhere,
30 possibly violating some of their assumptions. This is why this code
31 has to be extremely careful. Generally it tries to use normal locking
32 rules, as in get the standard locks, even if that means the
33 error handling takes potentially a long time.
34
35 Some of the operations here are somewhat inefficient and have non
36 linear algorithmic complexity, because the data structures have not
37 been optimized for this case. This is in particular the case
38 for the mapping from a vma to a process. Since this case is expected
39 to be rare we hope we can get away with this.
40
41The code consists of a the high level handler in mm/memory-failure.c,
42a new page poison bit and various checks in the VM to handle poisoned
43pages.
44
45The main target right now is KVM guests, but it works for all kinds
46of applications. KVM support requires a recent qemu-kvm release.
47
48For the KVM use there was need for a new signal type so that
49KVM can inject the machine check into the guest with the proper
50address. This in theory allows other applications to handle
51memory failures too. The expectation is that most applications
52won't do that, but some very specialized ones might.
53
54Failure recovery modes
55======================
56
57There are two (actually three) modes memory failure recovery can be in:
58
59vm.memory_failure_recovery sysctl set to zero:
60 All memory failures cause a panic. Do not attempt recovery.
61
62early kill
63 (can be controlled globally and per process)
64 Send SIGBUS to the application as soon as the error is detected
65 This allows applications who can process memory errors in a gentle
66 way (e.g. drop affected object)
67 This is the mode used by KVM qemu.
68
69late kill
70 Send SIGBUS when the application runs into the corrupted page.
71 This is best for memory error unaware applications and default
72 Note some pages are always handled as late kill.
73
74User control
75============
76
77vm.memory_failure_recovery
78 See sysctl.txt
79
80vm.memory_failure_early_kill
81 Enable early kill mode globally
82
83PR_MCE_KILL
84 Set early/late kill mode/revert to system default
85
86 arg1: PR_MCE_KILL_CLEAR:
87 Revert to system default
88 arg1: PR_MCE_KILL_SET:
89 arg2 defines thread specific mode
90
91 PR_MCE_KILL_EARLY:
92 Early kill
93 PR_MCE_KILL_LATE:
94 Late kill
95 PR_MCE_KILL_DEFAULT
96 Use system global default
97
98 Note that if you want to have a dedicated thread which handles
99 the SIGBUS(BUS_MCEERR_AO) on behalf of the process, you should
100 call prctl(PR_MCE_KILL_EARLY) on the designated thread. Otherwise,
101 the SIGBUS is sent to the main thread.
102
103PR_MCE_KILL_GET
104 return current mode
105
106Testing
107=======
108
109* madvise(MADV_HWPOISON, ....) (as root) - Poison a page in the
110 process for testing
111
112* hwpoison-inject module through debugfs ``/sys/kernel/debug/hwpoison/``
113
114 corrupt-pfn
115 Inject hwpoison fault at PFN echoed into this file. This does
116 some early filtering to avoid corrupted unintended pages in test suites.
117
118 unpoison-pfn
119 Software-unpoison page at PFN echoed into this file. This way
120 a page can be reused again. This only works for Linux
121 injected failures, not for real memory failures. Once any hardware
122 memory failure happens, this feature is disabled.
123
124 Note these injection interfaces are not stable and might change between
125 kernel versions
126
127 corrupt-filter-dev-major, corrupt-filter-dev-minor
128 Only handle memory failures to pages associated with the file
129 system defined by block device major/minor. -1U is the
130 wildcard value. This should be only used for testing with
131 artificial injection.
132
133 corrupt-filter-memcg
134 Limit injection to pages owned by memgroup. Specified by inode
135 number of the memcg.
136
137 Example::
138
139 mkdir /sys/fs/cgroup/mem/hwpoison
140
141 usemem -m 100 -s 1000 &
142 echo `jobs -p` > /sys/fs/cgroup/mem/hwpoison/tasks
143
144 memcg_ino=$(ls -id /sys/fs/cgroup/mem/hwpoison | cut -f1 -d' ')
145 echo $memcg_ino > /debug/hwpoison/corrupt-filter-memcg
146
147 page-types -p `pidof init` --hwpoison # shall do nothing
148 page-types -p `pidof usemem` --hwpoison # poison its pages
149
150 corrupt-filter-flags-mask, corrupt-filter-flags-value
151 When specified, only poison pages if ((page_flags & mask) ==
152 value). This allows stress testing of many kinds of
153 pages. The page_flags are the same as in /proc/kpageflags. The
154 flag bits are defined in include/linux/kernel-page-flags.h and
155 documented in Documentation/admin-guide/mm/pagemap.rst
156
157* Architecture specific MCE injector
158
159 x86 has mce-inject, mce-test
160
161 Some portable hwpoison test programs in mce-test, see below.
162
163References
164==========
165
166http://halobates.de/mce-lc09-2.pdf
167 Overview presentation from LinuxCon 09
168
169git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git
170 Test suite (hwpoison specific portable tests in tsrc)
171
172git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
173 x86 specific injector
174
175
176Limitations
177===========
178- Not all page types are supported and never will. Most kernel internal
179 objects cannot be recovered, only LRU pages for now.
180
181---
182Andi Kleen, Oct 2009
1.. hwpoison:
2
3========
4hwpoison
5========
6
7What is hwpoison?
8=================
9
10Upcoming Intel CPUs have support for recovering from some memory errors
11(``MCA recovery``). This requires the OS to declare a page "poisoned",
12kill the processes associated with it and avoid using it in the future.
13
14This patchkit implements the necessary infrastructure in the VM.
15
16To quote the overview comment::
17
18 High level machine check handler. Handles pages reported by the
19 hardware as being corrupted usually due to a 2bit ECC memory or cache
20 failure.
21
22 This focusses on pages detected as corrupted in the background.
23 When the current CPU tries to consume corruption the currently
24 running process can just be killed directly instead. This implies
25 that if the error cannot be handled for some reason it's safe to
26 just ignore it because no corruption has been consumed yet. Instead
27 when that happens another machine check will happen.
28
29 Handles page cache pages in various states. The tricky part
30 here is that we can access any page asynchronous to other VM
31 users, because memory failures could happen anytime and anywhere,
32 possibly violating some of their assumptions. This is why this code
33 has to be extremely careful. Generally it tries to use normal locking
34 rules, as in get the standard locks, even if that means the
35 error handling takes potentially a long time.
36
37 Some of the operations here are somewhat inefficient and have non
38 linear algorithmic complexity, because the data structures have not
39 been optimized for this case. This is in particular the case
40 for the mapping from a vma to a process. Since this case is expected
41 to be rare we hope we can get away with this.
42
43The code consists of a the high level handler in mm/memory-failure.c,
44a new page poison bit and various checks in the VM to handle poisoned
45pages.
46
47The main target right now is KVM guests, but it works for all kinds
48of applications. KVM support requires a recent qemu-kvm release.
49
50For the KVM use there was need for a new signal type so that
51KVM can inject the machine check into the guest with the proper
52address. This in theory allows other applications to handle
53memory failures too. The expection is that near all applications
54won't do that, but some very specialized ones might.
55
56Failure recovery modes
57======================
58
59There are two (actually three) modes memory failure recovery can be in:
60
61vm.memory_failure_recovery sysctl set to zero:
62 All memory failures cause a panic. Do not attempt recovery.
63
64early kill
65 (can be controlled globally and per process)
66 Send SIGBUS to the application as soon as the error is detected
67 This allows applications who can process memory errors in a gentle
68 way (e.g. drop affected object)
69 This is the mode used by KVM qemu.
70
71late kill
72 Send SIGBUS when the application runs into the corrupted page.
73 This is best for memory error unaware applications and default
74 Note some pages are always handled as late kill.
75
76User control
77============
78
79vm.memory_failure_recovery
80 See sysctl.txt
81
82vm.memory_failure_early_kill
83 Enable early kill mode globally
84
85PR_MCE_KILL
86 Set early/late kill mode/revert to system default
87
88 arg1: PR_MCE_KILL_CLEAR:
89 Revert to system default
90 arg1: PR_MCE_KILL_SET:
91 arg2 defines thread specific mode
92
93 PR_MCE_KILL_EARLY:
94 Early kill
95 PR_MCE_KILL_LATE:
96 Late kill
97 PR_MCE_KILL_DEFAULT
98 Use system global default
99
100 Note that if you want to have a dedicated thread which handles
101 the SIGBUS(BUS_MCEERR_AO) on behalf of the process, you should
102 call prctl(PR_MCE_KILL_EARLY) on the designated thread. Otherwise,
103 the SIGBUS is sent to the main thread.
104
105PR_MCE_KILL_GET
106 return current mode
107
108Testing
109=======
110
111* madvise(MADV_HWPOISON, ....) (as root) - Poison a page in the
112 process for testing
113
114* hwpoison-inject module through debugfs ``/sys/kernel/debug/hwpoison/``
115
116 corrupt-pfn
117 Inject hwpoison fault at PFN echoed into this file. This does
118 some early filtering to avoid corrupted unintended pages in test suites.
119
120 unpoison-pfn
121 Software-unpoison page at PFN echoed into this file. This way
122 a page can be reused again. This only works for Linux
123 injected failures, not for real memory failures. Once any hardware
124 memory failure happens, this feature is disabled.
125
126 Note these injection interfaces are not stable and might change between
127 kernel versions
128
129 corrupt-filter-dev-major, corrupt-filter-dev-minor
130 Only handle memory failures to pages associated with the file
131 system defined by block device major/minor. -1U is the
132 wildcard value. This should be only used for testing with
133 artificial injection.
134
135 corrupt-filter-memcg
136 Limit injection to pages owned by memgroup. Specified by inode
137 number of the memcg.
138
139 Example::
140
141 mkdir /sys/fs/cgroup/mem/hwpoison
142
143 usemem -m 100 -s 1000 &
144 echo `jobs -p` > /sys/fs/cgroup/mem/hwpoison/tasks
145
146 memcg_ino=$(ls -id /sys/fs/cgroup/mem/hwpoison | cut -f1 -d' ')
147 echo $memcg_ino > /debug/hwpoison/corrupt-filter-memcg
148
149 page-types -p `pidof init` --hwpoison # shall do nothing
150 page-types -p `pidof usemem` --hwpoison # poison its pages
151
152 corrupt-filter-flags-mask, corrupt-filter-flags-value
153 When specified, only poison pages if ((page_flags & mask) ==
154 value). This allows stress testing of many kinds of
155 pages. The page_flags are the same as in /proc/kpageflags. The
156 flag bits are defined in include/linux/kernel-page-flags.h and
157 documented in Documentation/admin-guide/mm/pagemap.rst
158
159* Architecture specific MCE injector
160
161 x86 has mce-inject, mce-test
162
163 Some portable hwpoison test programs in mce-test, see below.
164
165References
166==========
167
168http://halobates.de/mce-lc09-2.pdf
169 Overview presentation from LinuxCon 09
170
171git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git
172 Test suite (hwpoison specific portable tests in tsrc)
173
174git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
175 x86 specific injector
176
177
178Limitations
179===========
180- Not all page types are supported and never will. Most kernel internal
181 objects cannot be recovered, only LRU pages for now.
182
183---
184Andi Kleen, Oct 2009