Loading...
1=========================
2CPU hotplug in the Kernel
3=========================
4
5:Date: September, 2021
6:Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
7 Rusty Russell <rusty@rustcorp.com.au>,
8 Srivatsa Vaddagiri <vatsa@in.ibm.com>,
9 Ashok Raj <ashok.raj@intel.com>,
10 Joel Schopp <jschopp@austin.ibm.com>,
11 Thomas Gleixner <tglx@linutronix.de>
12
13Introduction
14============
15
16Modern advances in system architectures have introduced advanced error
17reporting and correction capabilities in processors. There are couple OEMS that
18support NUMA hardware which are hot pluggable as well, where physical node
19insertion and removal require support for CPU hotplug.
20
21Such advances require CPUs available to a kernel to be removed either for
22provisioning reasons, or for RAS purposes to keep an offending CPU off
23system execution path. Hence the need for CPU hotplug support in the
24Linux kernel.
25
26A more novel use of CPU-hotplug support is its use today in suspend resume
27support for SMP. Dual-core and HT support makes even a laptop run SMP kernels
28which didn't support these methods.
29
30
31Command Line Switches
32=====================
33``maxcpus=n``
34 Restrict boot time CPUs to *n*. Say if you have four CPUs, using
35 ``maxcpus=2`` will only boot two. You can choose to bring the
36 other CPUs later online.
37
38``nr_cpus=n``
39 Restrict the total amount of CPUs the kernel will support. If the number
40 supplied here is lower than the number of physically available CPUs, then
41 those CPUs can not be brought online later.
42
43``additional_cpus=n``
44 Use this to limit hotpluggable CPUs. This option sets
45 ``cpu_possible_mask = cpu_present_mask + additional_cpus``
46
47 This option is limited to the IA64 architecture.
48
49``possible_cpus=n``
50 This option sets ``possible_cpus`` bits in ``cpu_possible_mask``.
51
52 This option is limited to the X86 and S390 architecture.
53
54``cpu0_hotplug``
55 Allow to shutdown CPU0.
56
57 This option is limited to the X86 architecture.
58
59CPU maps
60========
61
62``cpu_possible_mask``
63 Bitmap of possible CPUs that can ever be available in the
64 system. This is used to allocate some boot time memory for per_cpu variables
65 that aren't designed to grow/shrink as CPUs are made available or removed.
66 Once set during boot time discovery phase, the map is static, i.e no bits
67 are added or removed anytime. Trimming it accurately for your system needs
68 upfront can save some boot time memory.
69
70``cpu_online_mask``
71 Bitmap of all CPUs currently online. Its set in ``__cpu_up()``
72 after a CPU is available for kernel scheduling and ready to receive
73 interrupts from devices. Its cleared when a CPU is brought down using
74 ``__cpu_disable()``, before which all OS services including interrupts are
75 migrated to another target CPU.
76
77``cpu_present_mask``
78 Bitmap of CPUs currently present in the system. Not all
79 of them may be online. When physical hotplug is processed by the relevant
80 subsystem (e.g ACPI) can change and new bit either be added or removed
81 from the map depending on the event is hot-add/hot-remove. There are currently
82 no locking rules as of now. Typical usage is to init topology during boot,
83 at which time hotplug is disabled.
84
85You really don't need to manipulate any of the system CPU maps. They should
86be read-only for most use. When setting up per-cpu resources almost always use
87``cpu_possible_mask`` or ``for_each_possible_cpu()`` to iterate. To macro
88``for_each_cpu()`` can be used to iterate over a custom CPU mask.
89
90Never use anything other than ``cpumask_t`` to represent bitmap of CPUs.
91
92
93Using CPU hotplug
94=================
95
96The kernel option *CONFIG_HOTPLUG_CPU* needs to be enabled. It is currently
97available on multiple architectures including ARM, MIPS, PowerPC and X86. The
98configuration is done via the sysfs interface::
99
100 $ ls -lh /sys/devices/system/cpu
101 total 0
102 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu0
103 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu1
104 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu2
105 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu3
106 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu4
107 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu5
108 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu6
109 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu7
110 drwxr-xr-x 2 root root 0 Dec 21 16:33 hotplug
111 -r--r--r-- 1 root root 4.0K Dec 21 16:33 offline
112 -r--r--r-- 1 root root 4.0K Dec 21 16:33 online
113 -r--r--r-- 1 root root 4.0K Dec 21 16:33 possible
114 -r--r--r-- 1 root root 4.0K Dec 21 16:33 present
115
116The files *offline*, *online*, *possible*, *present* represent the CPU masks.
117Each CPU folder contains an *online* file which controls the logical on (1) and
118off (0) state. To logically shutdown CPU4::
119
120 $ echo 0 > /sys/devices/system/cpu/cpu4/online
121 smpboot: CPU 4 is now offline
122
123Once the CPU is shutdown, it will be removed from */proc/interrupts*,
124*/proc/cpuinfo* and should also not be shown visible by the *top* command. To
125bring CPU4 back online::
126
127 $ echo 1 > /sys/devices/system/cpu/cpu4/online
128 smpboot: Booting Node 0 Processor 4 APIC 0x1
129
130The CPU is usable again. This should work on all CPUs. CPU0 is often special
131and excluded from CPU hotplug. On X86 the kernel option
132*CONFIG_BOOTPARAM_HOTPLUG_CPU0* has to be enabled in order to be able to
133shutdown CPU0. Alternatively the kernel command option *cpu0_hotplug* can be
134used. Some known dependencies of CPU0:
135
136* Resume from hibernate/suspend. Hibernate/suspend will fail if CPU0 is offline.
137* PIC interrupts. CPU0 can't be removed if a PIC interrupt is detected.
138
139Please let Fenghua Yu <fenghua.yu@intel.com> know if you find any dependencies
140on CPU0.
141
142The CPU hotplug coordination
143============================
144
145The offline case
146----------------
147
148Once a CPU has been logically shutdown the teardown callbacks of registered
149hotplug states will be invoked, starting with ``CPUHP_ONLINE`` and terminating
150at state ``CPUHP_OFFLINE``. This includes:
151
152* If tasks are frozen due to a suspend operation then *cpuhp_tasks_frozen*
153 will be set to true.
154* All processes are migrated away from this outgoing CPU to new CPUs.
155 The new CPU is chosen from each process' current cpuset, which may be
156 a subset of all online CPUs.
157* All interrupts targeted to this CPU are migrated to a new CPU
158* timers are also migrated to a new CPU
159* Once all services are migrated, kernel calls an arch specific routine
160 ``__cpu_disable()`` to perform arch specific cleanup.
161
162
163The CPU hotplug API
164===================
165
166CPU hotplug state machine
167-------------------------
168
169CPU hotplug uses a trivial state machine with a linear state space from
170CPUHP_OFFLINE to CPUHP_ONLINE. Each state has a startup and a teardown
171callback.
172
173When a CPU is onlined, the startup callbacks are invoked sequentially until
174the state CPUHP_ONLINE is reached. They can also be invoked when the
175callbacks of a state are set up or an instance is added to a multi-instance
176state.
177
178When a CPU is offlined the teardown callbacks are invoked in the reverse
179order sequentially until the state CPUHP_OFFLINE is reached. They can also
180be invoked when the callbacks of a state are removed or an instance is
181removed from a multi-instance state.
182
183If a usage site requires only a callback in one direction of the hotplug
184operations (CPU online or CPU offline) then the other not-required callback
185can be set to NULL when the state is set up.
186
187The state space is divided into three sections:
188
189* The PREPARE section
190
191 The PREPARE section covers the state space from CPUHP_OFFLINE to
192 CPUHP_BRINGUP_CPU.
193
194 The startup callbacks in this section are invoked before the CPU is
195 started during a CPU online operation. The teardown callbacks are invoked
196 after the CPU has become dysfunctional during a CPU offline operation.
197
198 The callbacks are invoked on a control CPU as they can't obviously run on
199 the hotplugged CPU which is either not yet started or has become
200 dysfunctional already.
201
202 The startup callbacks are used to setup resources which are required to
203 bring a CPU successfully online. The teardown callbacks are used to free
204 resources or to move pending work to an online CPU after the hotplugged
205 CPU became dysfunctional.
206
207 The startup callbacks are allowed to fail. If a callback fails, the CPU
208 online operation is aborted and the CPU is brought down to the previous
209 state (usually CPUHP_OFFLINE) again.
210
211 The teardown callbacks in this section are not allowed to fail.
212
213* The STARTING section
214
215 The STARTING section covers the state space between CPUHP_BRINGUP_CPU + 1
216 and CPUHP_AP_ONLINE.
217
218 The startup callbacks in this section are invoked on the hotplugged CPU
219 with interrupts disabled during a CPU online operation in the early CPU
220 setup code. The teardown callbacks are invoked with interrupts disabled
221 on the hotplugged CPU during a CPU offline operation shortly before the
222 CPU is completely shut down.
223
224 The callbacks in this section are not allowed to fail.
225
226 The callbacks are used for low level hardware initialization/shutdown and
227 for core subsystems.
228
229* The ONLINE section
230
231 The ONLINE section covers the state space between CPUHP_AP_ONLINE + 1 and
232 CPUHP_ONLINE.
233
234 The startup callbacks in this section are invoked on the hotplugged CPU
235 during a CPU online operation. The teardown callbacks are invoked on the
236 hotplugged CPU during a CPU offline operation.
237
238 The callbacks are invoked in the context of the per CPU hotplug thread,
239 which is pinned on the hotplugged CPU. The callbacks are invoked with
240 interrupts and preemption enabled.
241
242 The callbacks are allowed to fail. When a callback fails the hotplug
243 operation is aborted and the CPU is brought back to the previous state.
244
245CPU online/offline operations
246-----------------------------
247
248A successful online operation looks like this::
249
250 [CPUHP_OFFLINE]
251 [CPUHP_OFFLINE + 1]->startup() -> success
252 [CPUHP_OFFLINE + 2]->startup() -> success
253 [CPUHP_OFFLINE + 3] -> skipped because startup == NULL
254 ...
255 [CPUHP_BRINGUP_CPU]->startup() -> success
256 === End of PREPARE section
257 [CPUHP_BRINGUP_CPU + 1]->startup() -> success
258 ...
259 [CPUHP_AP_ONLINE]->startup() -> success
260 === End of STARTUP section
261 [CPUHP_AP_ONLINE + 1]->startup() -> success
262 ...
263 [CPUHP_ONLINE - 1]->startup() -> success
264 [CPUHP_ONLINE]
265
266A successful offline operation looks like this::
267
268 [CPUHP_ONLINE]
269 [CPUHP_ONLINE - 1]->teardown() -> success
270 ...
271 [CPUHP_AP_ONLINE + 1]->teardown() -> success
272 === Start of STARTUP section
273 [CPUHP_AP_ONLINE]->teardown() -> success
274 ...
275 [CPUHP_BRINGUP_ONLINE - 1]->teardown()
276 ...
277 === Start of PREPARE section
278 [CPUHP_BRINGUP_CPU]->teardown()
279 [CPUHP_OFFLINE + 3]->teardown()
280 [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL
281 [CPUHP_OFFLINE + 1]->teardown()
282 [CPUHP_OFFLINE]
283
284A failed online operation looks like this::
285
286 [CPUHP_OFFLINE]
287 [CPUHP_OFFLINE + 1]->startup() -> success
288 [CPUHP_OFFLINE + 2]->startup() -> success
289 [CPUHP_OFFLINE + 3] -> skipped because startup == NULL
290 ...
291 [CPUHP_BRINGUP_CPU]->startup() -> success
292 === End of PREPARE section
293 [CPUHP_BRINGUP_CPU + 1]->startup() -> success
294 ...
295 [CPUHP_AP_ONLINE]->startup() -> success
296 === End of STARTUP section
297 [CPUHP_AP_ONLINE + 1]->startup() -> success
298 ---
299 [CPUHP_AP_ONLINE + N]->startup() -> fail
300 [CPUHP_AP_ONLINE + (N - 1)]->teardown()
301 ...
302 [CPUHP_AP_ONLINE + 1]->teardown()
303 === Start of STARTUP section
304 [CPUHP_AP_ONLINE]->teardown()
305 ...
306 [CPUHP_BRINGUP_ONLINE - 1]->teardown()
307 ...
308 === Start of PREPARE section
309 [CPUHP_BRINGUP_CPU]->teardown()
310 [CPUHP_OFFLINE + 3]->teardown()
311 [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL
312 [CPUHP_OFFLINE + 1]->teardown()
313 [CPUHP_OFFLINE]
314
315A failed offline operation looks like this::
316
317 [CPUHP_ONLINE]
318 [CPUHP_ONLINE - 1]->teardown() -> success
319 ...
320 [CPUHP_ONLINE - N]->teardown() -> fail
321 [CPUHP_ONLINE - (N - 1)]->startup()
322 ...
323 [CPUHP_ONLINE - 1]->startup()
324 [CPUHP_ONLINE]
325
326Recursive failures cannot be handled sensibly. Look at the following
327example of a recursive fail due to a failed offline operation: ::
328
329 [CPUHP_ONLINE]
330 [CPUHP_ONLINE - 1]->teardown() -> success
331 ...
332 [CPUHP_ONLINE - N]->teardown() -> fail
333 [CPUHP_ONLINE - (N - 1)]->startup() -> success
334 [CPUHP_ONLINE - (N - 2)]->startup() -> fail
335
336The CPU hotplug state machine stops right here and does not try to go back
337down again because that would likely result in an endless loop::
338
339 [CPUHP_ONLINE - (N - 1)]->teardown() -> success
340 [CPUHP_ONLINE - N]->teardown() -> fail
341 [CPUHP_ONLINE - (N - 1)]->startup() -> success
342 [CPUHP_ONLINE - (N - 2)]->startup() -> fail
343 [CPUHP_ONLINE - (N - 1)]->teardown() -> success
344 [CPUHP_ONLINE - N]->teardown() -> fail
345
346Lather, rinse and repeat. In this case the CPU left in state::
347
348 [CPUHP_ONLINE - (N - 1)]
349
350which at least lets the system make progress and gives the user a chance to
351debug or even resolve the situation.
352
353Allocating a state
354------------------
355
356There are two ways to allocate a CPU hotplug state:
357
358* Static allocation
359
360 Static allocation has to be used when the subsystem or driver has
361 ordering requirements versus other CPU hotplug states. E.g. the PERF core
362 startup callback has to be invoked before the PERF driver startup
363 callbacks during a CPU online operation. During a CPU offline operation
364 the driver teardown callbacks have to be invoked before the core teardown
365 callback. The statically allocated states are described by constants in
366 the cpuhp_state enum which can be found in include/linux/cpuhotplug.h.
367
368 Insert the state into the enum at the proper place so the ordering
369 requirements are fulfilled. The state constant has to be used for state
370 setup and removal.
371
372 Static allocation is also required when the state callbacks are not set
373 up at runtime and are part of the initializer of the CPU hotplug state
374 array in kernel/cpu.c.
375
376* Dynamic allocation
377
378 When there are no ordering requirements for the state callbacks then
379 dynamic allocation is the preferred method. The state number is allocated
380 by the setup function and returned to the caller on success.
381
382 Only the PREPARE and ONLINE sections provide a dynamic allocation
383 range. The STARTING section does not as most of the callbacks in that
384 section have explicit ordering requirements.
385
386Setup of a CPU hotplug state
387----------------------------
388
389The core code provides the following functions to setup a state:
390
391* cpuhp_setup_state(state, name, startup, teardown)
392* cpuhp_setup_state_nocalls(state, name, startup, teardown)
393* cpuhp_setup_state_cpuslocked(state, name, startup, teardown)
394* cpuhp_setup_state_nocalls_cpuslocked(state, name, startup, teardown)
395
396For cases where a driver or a subsystem has multiple instances and the same
397CPU hotplug state callbacks need to be invoked for each instance, the CPU
398hotplug core provides multi-instance support. The advantage over driver
399specific instance lists is that the instance related functions are fully
400serialized against CPU hotplug operations and provide the automatic
401invocations of the state callbacks on add and removal. To set up such a
402multi-instance state the following function is available:
403
404* cpuhp_setup_state_multi(state, name, startup, teardown)
405
406The @state argument is either a statically allocated state or one of the
407constants for dynamically allocated states - CPUHP_PREPARE_DYN,
408CPUHP_ONLINE_DYN - depending on the state section (PREPARE, ONLINE) for
409which a dynamic state should be allocated.
410
411The @name argument is used for sysfs output and for instrumentation. The
412naming convention is "subsys:mode" or "subsys/driver:mode",
413e.g. "perf:mode" or "perf/x86:mode". The common mode names are:
414
415======== =======================================================
416prepare For states in the PREPARE section
417
418dead For states in the PREPARE section which do not provide
419 a startup callback
420
421starting For states in the STARTING section
422
423dying For states in the STARTING section which do not provide
424 a startup callback
425
426online For states in the ONLINE section
427
428offline For states in the ONLINE section which do not provide
429 a startup callback
430======== =======================================================
431
432As the @name argument is only used for sysfs and instrumentation other mode
433descriptors can be used as well if they describe the nature of the state
434better than the common ones.
435
436Examples for @name arguments: "perf/online", "perf/x86:prepare",
437"RCU/tree:dying", "sched/waitempty"
438
439The @startup argument is a function pointer to the callback which should be
440invoked during a CPU online operation. If the usage site does not require a
441startup callback set the pointer to NULL.
442
443The @teardown argument is a function pointer to the callback which should
444be invoked during a CPU offline operation. If the usage site does not
445require a teardown callback set the pointer to NULL.
446
447The functions differ in the way how the installed callbacks are treated:
448
449 * cpuhp_setup_state_nocalls(), cpuhp_setup_state_nocalls_cpuslocked()
450 and cpuhp_setup_state_multi() only install the callbacks
451
452 * cpuhp_setup_state() and cpuhp_setup_state_cpuslocked() install the
453 callbacks and invoke the @startup callback (if not NULL) for all online
454 CPUs which have currently a state greater than the newly installed
455 state. Depending on the state section the callback is either invoked on
456 the current CPU (PREPARE section) or on each online CPU (ONLINE
457 section) in the context of the CPU's hotplug thread.
458
459 If a callback fails for CPU N then the teardown callback for CPU
460 0 .. N-1 is invoked to rollback the operation. The state setup fails,
461 the callbacks for the state are not installed and in case of dynamic
462 allocation the allocated state is freed.
463
464The state setup and the callback invocations are serialized against CPU
465hotplug operations. If the setup function has to be called from a CPU
466hotplug read locked region, then the _cpuslocked() variants have to be
467used. These functions cannot be used from within CPU hotplug callbacks.
468
469The function return values:
470 ======== ===================================================================
471 0 Statically allocated state was successfully set up
472
473 >0 Dynamically allocated state was successfully set up.
474
475 The returned number is the state number which was allocated. If
476 the state callbacks have to be removed later, e.g. module
477 removal, then this number has to be saved by the caller and used
478 as @state argument for the state remove function. For
479 multi-instance states the dynamically allocated state number is
480 also required as @state argument for the instance add/remove
481 operations.
482
483 <0 Operation failed
484 ======== ===================================================================
485
486Removal of a CPU hotplug state
487------------------------------
488
489To remove a previously set up state, the following functions are provided:
490
491* cpuhp_remove_state(state)
492* cpuhp_remove_state_nocalls(state)
493* cpuhp_remove_state_nocalls_cpuslocked(state)
494* cpuhp_remove_multi_state(state)
495
496The @state argument is either a statically allocated state or the state
497number which was allocated in the dynamic range by cpuhp_setup_state*(). If
498the state is in the dynamic range, then the state number is freed and
499available for dynamic allocation again.
500
501The functions differ in the way how the installed callbacks are treated:
502
503 * cpuhp_remove_state_nocalls(), cpuhp_remove_state_nocalls_cpuslocked()
504 and cpuhp_remove_multi_state() only remove the callbacks.
505
506 * cpuhp_remove_state() removes the callbacks and invokes the teardown
507 callback (if not NULL) for all online CPUs which have currently a state
508 greater than the removed state. Depending on the state section the
509 callback is either invoked on the current CPU (PREPARE section) or on
510 each online CPU (ONLINE section) in the context of the CPU's hotplug
511 thread.
512
513 In order to complete the removal, the teardown callback should not fail.
514
515The state removal and the callback invocations are serialized against CPU
516hotplug operations. If the remove function has to be called from a CPU
517hotplug read locked region, then the _cpuslocked() variants have to be
518used. These functions cannot be used from within CPU hotplug callbacks.
519
520If a multi-instance state is removed then the caller has to remove all
521instances first.
522
523Multi-Instance state instance management
524----------------------------------------
525
526Once the multi-instance state is set up, instances can be added to the
527state:
528
529 * cpuhp_state_add_instance(state, node)
530 * cpuhp_state_add_instance_nocalls(state, node)
531
532The @state argument is either a statically allocated state or the state
533number which was allocated in the dynamic range by cpuhp_setup_state_multi().
534
535The @node argument is a pointer to an hlist_node which is embedded in the
536instance's data structure. The pointer is handed to the multi-instance
537state callbacks and can be used by the callback to retrieve the instance
538via container_of().
539
540The functions differ in the way how the installed callbacks are treated:
541
542 * cpuhp_state_add_instance_nocalls() and only adds the instance to the
543 multi-instance state's node list.
544
545 * cpuhp_state_add_instance() adds the instance and invokes the startup
546 callback (if not NULL) associated with @state for all online CPUs which
547 have currently a state greater than @state. The callback is only
548 invoked for the to be added instance. Depending on the state section
549 the callback is either invoked on the current CPU (PREPARE section) or
550 on each online CPU (ONLINE section) in the context of the CPU's hotplug
551 thread.
552
553 If a callback fails for CPU N then the teardown callback for CPU
554 0 .. N-1 is invoked to rollback the operation, the function fails and
555 the instance is not added to the node list of the multi-instance state.
556
557To remove an instance from the state's node list these functions are
558available:
559
560 * cpuhp_state_remove_instance(state, node)
561 * cpuhp_state_remove_instance_nocalls(state, node)
562
563The arguments are the same as for the cpuhp_state_add_instance*()
564variants above.
565
566The functions differ in the way how the installed callbacks are treated:
567
568 * cpuhp_state_remove_instance_nocalls() only removes the instance from the
569 state's node list.
570
571 * cpuhp_state_remove_instance() removes the instance and invokes the
572 teardown callback (if not NULL) associated with @state for all online
573 CPUs which have currently a state greater than @state. The callback is
574 only invoked for the to be removed instance. Depending on the state
575 section the callback is either invoked on the current CPU (PREPARE
576 section) or on each online CPU (ONLINE section) in the context of the
577 CPU's hotplug thread.
578
579 In order to complete the removal, the teardown callback should not fail.
580
581The node list add/remove operations and the callback invocations are
582serialized against CPU hotplug operations. These functions cannot be used
583from within CPU hotplug callbacks and CPU hotplug read locked regions.
584
585Examples
586--------
587
588Setup and teardown a statically allocated state in the STARTING section for
589notifications on online and offline operations::
590
591 ret = cpuhp_setup_state(CPUHP_SUBSYS_STARTING, "subsys:starting", subsys_cpu_starting, subsys_cpu_dying);
592 if (ret < 0)
593 return ret;
594 ....
595 cpuhp_remove_state(CPUHP_SUBSYS_STARTING);
596
597Setup and teardown a dynamically allocated state in the ONLINE section
598for notifications on offline operations::
599
600 state = cpuhp_setup_state(CPUHP_ONLINE_DYN, "subsys:offline", NULL, subsys_cpu_offline);
601 if (state < 0)
602 return state;
603 ....
604 cpuhp_remove_state(state);
605
606Setup and teardown a dynamically allocated state in the ONLINE section
607for notifications on online operations without invoking the callbacks::
608
609 state = cpuhp_setup_state_nocalls(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, NULL);
610 if (state < 0)
611 return state;
612 ....
613 cpuhp_remove_state_nocalls(state);
614
615Setup, use and teardown a dynamically allocated multi-instance state in the
616ONLINE section for notifications on online and offline operation::
617
618 state = cpuhp_setup_state_multi(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, subsys_cpu_offline);
619 if (state < 0)
620 return state;
621 ....
622 ret = cpuhp_state_add_instance(state, &inst1->node);
623 if (ret)
624 return ret;
625 ....
626 ret = cpuhp_state_add_instance(state, &inst2->node);
627 if (ret)
628 return ret;
629 ....
630 cpuhp_remove_instance(state, &inst1->node);
631 ....
632 cpuhp_remove_instance(state, &inst2->node);
633 ....
634 remove_multi_state(state);
635
636
637Testing of hotplug states
638=========================
639
640One way to verify whether a custom state is working as expected or not is to
641shutdown a CPU and then put it online again. It is also possible to put the CPU
642to certain state (for instance *CPUHP_AP_ONLINE*) and then go back to
643*CPUHP_ONLINE*. This would simulate an error one state after *CPUHP_AP_ONLINE*
644which would lead to rollback to the online state.
645
646All registered states are enumerated in ``/sys/devices/system/cpu/hotplug/states`` ::
647
648 $ tail /sys/devices/system/cpu/hotplug/states
649 138: mm/vmscan:online
650 139: mm/vmstat:online
651 140: lib/percpu_cnt:online
652 141: acpi/cpu-drv:online
653 142: base/cacheinfo:online
654 143: virtio/net:online
655 144: x86/mce:online
656 145: printk:online
657 168: sched:active
658 169: online
659
660To rollback CPU4 to ``lib/percpu_cnt:online`` and back online just issue::
661
662 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
663 169
664 $ echo 140 > /sys/devices/system/cpu/cpu4/hotplug/target
665 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
666 140
667
668It is important to note that the teardown callback of state 140 have been
669invoked. And now get back online::
670
671 $ echo 169 > /sys/devices/system/cpu/cpu4/hotplug/target
672 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
673 169
674
675With trace events enabled, the individual steps are visible, too::
676
677 # TASK-PID CPU# TIMESTAMP FUNCTION
678 # | | | | |
679 bash-394 [001] 22.976: cpuhp_enter: cpu: 0004 target: 140 step: 169 (cpuhp_kick_ap_work)
680 cpuhp/4-31 [004] 22.977: cpuhp_enter: cpu: 0004 target: 140 step: 168 (sched_cpu_deactivate)
681 cpuhp/4-31 [004] 22.990: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
682 cpuhp/4-31 [004] 22.991: cpuhp_enter: cpu: 0004 target: 140 step: 144 (mce_cpu_pre_down)
683 cpuhp/4-31 [004] 22.992: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
684 cpuhp/4-31 [004] 22.993: cpuhp_multi_enter: cpu: 0004 target: 140 step: 143 (virtnet_cpu_down_prep)
685 cpuhp/4-31 [004] 22.994: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
686 cpuhp/4-31 [004] 22.995: cpuhp_enter: cpu: 0004 target: 140 step: 142 (cacheinfo_cpu_pre_down)
687 cpuhp/4-31 [004] 22.996: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
688 bash-394 [001] 22.997: cpuhp_exit: cpu: 0004 state: 140 step: 169 ret: 0
689 bash-394 [005] 95.540: cpuhp_enter: cpu: 0004 target: 169 step: 140 (cpuhp_kick_ap_work)
690 cpuhp/4-31 [004] 95.541: cpuhp_enter: cpu: 0004 target: 169 step: 141 (acpi_soft_cpu_online)
691 cpuhp/4-31 [004] 95.542: cpuhp_exit: cpu: 0004 state: 141 step: 141 ret: 0
692 cpuhp/4-31 [004] 95.543: cpuhp_enter: cpu: 0004 target: 169 step: 142 (cacheinfo_cpu_online)
693 cpuhp/4-31 [004] 95.544: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
694 cpuhp/4-31 [004] 95.545: cpuhp_multi_enter: cpu: 0004 target: 169 step: 143 (virtnet_cpu_online)
695 cpuhp/4-31 [004] 95.546: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
696 cpuhp/4-31 [004] 95.547: cpuhp_enter: cpu: 0004 target: 169 step: 144 (mce_cpu_online)
697 cpuhp/4-31 [004] 95.548: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
698 cpuhp/4-31 [004] 95.549: cpuhp_enter: cpu: 0004 target: 169 step: 145 (console_cpu_notify)
699 cpuhp/4-31 [004] 95.550: cpuhp_exit: cpu: 0004 state: 145 step: 145 ret: 0
700 cpuhp/4-31 [004] 95.551: cpuhp_enter: cpu: 0004 target: 169 step: 168 (sched_cpu_activate)
701 cpuhp/4-31 [004] 95.552: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
702 bash-394 [005] 95.553: cpuhp_exit: cpu: 0004 state: 169 step: 140 ret: 0
703
704As it an be seen, CPU4 went down until timestamp 22.996 and then back up until
70595.552. All invoked callbacks including their return codes are visible in the
706trace.
707
708Architecture's requirements
709===========================
710
711The following functions and configurations are required:
712
713``CONFIG_HOTPLUG_CPU``
714 This entry needs to be enabled in Kconfig
715
716``__cpu_up()``
717 Arch interface to bring up a CPU
718
719``__cpu_disable()``
720 Arch interface to shutdown a CPU, no more interrupts can be handled by the
721 kernel after the routine returns. This includes the shutdown of the timer.
722
723``__cpu_die()``
724 This actually supposed to ensure death of the CPU. Actually look at some
725 example code in other arch that implement CPU hotplug. The processor is taken
726 down from the ``idle()`` loop for that specific architecture. ``__cpu_die()``
727 typically waits for some per_cpu state to be set, to ensure the processor dead
728 routine is called to be sure positively.
729
730User Space Notification
731=======================
732
733After CPU successfully onlined or offline udev events are sent. A udev rule like::
734
735 SUBSYSTEM=="cpu", DRIVERS=="processor", DEVPATH=="/devices/system/cpu/*", RUN+="the_hotplug_receiver.sh"
736
737will receive all events. A script like::
738
739 #!/bin/sh
740
741 if [ "${ACTION}" = "offline" ]
742 then
743 echo "CPU ${DEVPATH##*/} offline"
744
745 elif [ "${ACTION}" = "online" ]
746 then
747 echo "CPU ${DEVPATH##*/} online"
748
749 fi
750
751can process the event further.
752
753Kernel Inline Documentations Reference
754======================================
755
756.. kernel-doc:: include/linux/cpuhotplug.h
1=========================
2CPU hotplug in the Kernel
3=========================
4
5:Date: December, 2016
6:Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
7 Rusty Russell <rusty@rustcorp.com.au>,
8 Srivatsa Vaddagiri <vatsa@in.ibm.com>,
9 Ashok Raj <ashok.raj@intel.com>,
10 Joel Schopp <jschopp@austin.ibm.com>
11
12Introduction
13============
14
15Modern advances in system architectures have introduced advanced error
16reporting and correction capabilities in processors. There are couple OEMS that
17support NUMA hardware which are hot pluggable as well, where physical node
18insertion and removal require support for CPU hotplug.
19
20Such advances require CPUs available to a kernel to be removed either for
21provisioning reasons, or for RAS purposes to keep an offending CPU off
22system execution path. Hence the need for CPU hotplug support in the
23Linux kernel.
24
25A more novel use of CPU-hotplug support is its use today in suspend resume
26support for SMP. Dual-core and HT support makes even a laptop run SMP kernels
27which didn't support these methods.
28
29
30Command Line Switches
31=====================
32``maxcpus=n``
33 Restrict boot time CPUs to *n*. Say if you have fourV CPUs, using
34 ``maxcpus=2`` will only boot two. You can choose to bring the
35 other CPUs later online.
36
37``nr_cpus=n``
38 Restrict the total amount CPUs the kernel will support. If the number
39 supplied here is lower than the number of physically available CPUs than
40 those CPUs can not be brought online later.
41
42``additional_cpus=n``
43 Use this to limit hotpluggable CPUs. This option sets
44 ``cpu_possible_mask = cpu_present_mask + additional_cpus``
45
46 This option is limited to the IA64 architecture.
47
48``possible_cpus=n``
49 This option sets ``possible_cpus`` bits in ``cpu_possible_mask``.
50
51 This option is limited to the X86 and S390 architecture.
52
53``cede_offline={"off","on"}``
54 Use this option to disable/enable putting offlined processors to an extended
55 ``H_CEDE`` state on supported pseries platforms. If nothing is specified,
56 ``cede_offline`` is set to "on".
57
58 This option is limited to the PowerPC architecture.
59
60``cpu0_hotplug``
61 Allow to shutdown CPU0.
62
63 This option is limited to the X86 architecture.
64
65CPU maps
66========
67
68``cpu_possible_mask``
69 Bitmap of possible CPUs that can ever be available in the
70 system. This is used to allocate some boot time memory for per_cpu variables
71 that aren't designed to grow/shrink as CPUs are made available or removed.
72 Once set during boot time discovery phase, the map is static, i.e no bits
73 are added or removed anytime. Trimming it accurately for your system needs
74 upfront can save some boot time memory.
75
76``cpu_online_mask``
77 Bitmap of all CPUs currently online. Its set in ``__cpu_up()``
78 after a CPU is available for kernel scheduling and ready to receive
79 interrupts from devices. Its cleared when a CPU is brought down using
80 ``__cpu_disable()``, before which all OS services including interrupts are
81 migrated to another target CPU.
82
83``cpu_present_mask``
84 Bitmap of CPUs currently present in the system. Not all
85 of them may be online. When physical hotplug is processed by the relevant
86 subsystem (e.g ACPI) can change and new bit either be added or removed
87 from the map depending on the event is hot-add/hot-remove. There are currently
88 no locking rules as of now. Typical usage is to init topology during boot,
89 at which time hotplug is disabled.
90
91You really don't need to manipulate any of the system CPU maps. They should
92be read-only for most use. When setting up per-cpu resources almost always use
93``cpu_possible_mask`` or ``for_each_possible_cpu()`` to iterate. To macro
94``for_each_cpu()`` can be used to iterate over a custom CPU mask.
95
96Never use anything other than ``cpumask_t`` to represent bitmap of CPUs.
97
98
99Using CPU hotplug
100=================
101The kernel option *CONFIG_HOTPLUG_CPU* needs to be enabled. It is currently
102available on multiple architectures including ARM, MIPS, PowerPC and X86. The
103configuration is done via the sysfs interface: ::
104
105 $ ls -lh /sys/devices/system/cpu
106 total 0
107 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu0
108 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu1
109 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu2
110 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu3
111 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu4
112 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu5
113 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu6
114 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu7
115 drwxr-xr-x 2 root root 0 Dec 21 16:33 hotplug
116 -r--r--r-- 1 root root 4.0K Dec 21 16:33 offline
117 -r--r--r-- 1 root root 4.0K Dec 21 16:33 online
118 -r--r--r-- 1 root root 4.0K Dec 21 16:33 possible
119 -r--r--r-- 1 root root 4.0K Dec 21 16:33 present
120
121The files *offline*, *online*, *possible*, *present* represent the CPU masks.
122Each CPU folder contains an *online* file which controls the logical on (1) and
123off (0) state. To logically shutdown CPU4: ::
124
125 $ echo 0 > /sys/devices/system/cpu/cpu4/online
126 smpboot: CPU 4 is now offline
127
128Once the CPU is shutdown, it will be removed from */proc/interrupts*,
129*/proc/cpuinfo* and should also not be shown visible by the *top* command. To
130bring CPU4 back online: ::
131
132 $ echo 1 > /sys/devices/system/cpu/cpu4/online
133 smpboot: Booting Node 0 Processor 4 APIC 0x1
134
135The CPU is usable again. This should work on all CPUs. CPU0 is often special
136and excluded from CPU hotplug. On X86 the kernel option
137*CONFIG_BOOTPARAM_HOTPLUG_CPU0* has to be enabled in order to be able to
138shutdown CPU0. Alternatively the kernel command option *cpu0_hotplug* can be
139used. Some known dependencies of CPU0:
140
141* Resume from hibernate/suspend. Hibernate/suspend will fail if CPU0 is offline.
142* PIC interrupts. CPU0 can't be removed if a PIC interrupt is detected.
143
144Please let Fenghua Yu <fenghua.yu@intel.com> know if you find any dependencies
145on CPU0.
146
147The CPU hotplug coordination
148============================
149
150The offline case
151----------------
152Once a CPU has been logically shutdown the teardown callbacks of registered
153hotplug states will be invoked, starting with ``CPUHP_ONLINE`` and terminating
154at state ``CPUHP_OFFLINE``. This includes:
155
156* If tasks are frozen due to a suspend operation then *cpuhp_tasks_frozen*
157 will be set to true.
158* All processes are migrated away from this outgoing CPU to new CPUs.
159 The new CPU is chosen from each process' current cpuset, which may be
160 a subset of all online CPUs.
161* All interrupts targeted to this CPU are migrated to a new CPU
162* timers are also migrated to a new CPU
163* Once all services are migrated, kernel calls an arch specific routine
164 ``__cpu_disable()`` to perform arch specific cleanup.
165
166Using the hotplug API
167---------------------
168It is possible to receive notifications once a CPU is offline or onlined. This
169might be important to certain drivers which need to perform some kind of setup
170or clean up functions based on the number of available CPUs: ::
171
172 #include <linux/cpuhotplug.h>
173
174 ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "X/Y:online",
175 Y_online, Y_prepare_down);
176
177*X* is the subsystem and *Y* the particular driver. The *Y_online* callback
178will be invoked during registration on all online CPUs. If an error
179occurs during the online callback the *Y_prepare_down* callback will be
180invoked on all CPUs on which the online callback was previously invoked.
181After registration completed, the *Y_online* callback will be invoked
182once a CPU is brought online and *Y_prepare_down* will be invoked when a
183CPU is shutdown. All resources which were previously allocated in
184*Y_online* should be released in *Y_prepare_down*.
185The return value *ret* is negative if an error occurred during the
186registration process. Otherwise a positive value is returned which
187contains the allocated hotplug for dynamically allocated states
188(*CPUHP_AP_ONLINE_DYN*). It will return zero for predefined states.
189
190The callback can be remove by invoking ``cpuhp_remove_state()``. In case of a
191dynamically allocated state (*CPUHP_AP_ONLINE_DYN*) use the returned state.
192During the removal of a hotplug state the teardown callback will be invoked.
193
194Multiple instances
195~~~~~~~~~~~~~~~~~~
196If a driver has multiple instances and each instance needs to perform the
197callback independently then it is likely that a ''multi-state'' should be used.
198First a multi-state state needs to be registered: ::
199
200 ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "X/Y:online,
201 Y_online, Y_prepare_down);
202 Y_hp_online = ret;
203
204The ``cpuhp_setup_state_multi()`` behaves similar to ``cpuhp_setup_state()``
205except it prepares the callbacks for a multi state and does not invoke
206the callbacks. This is a one time setup.
207Once a new instance is allocated, you need to register this new instance: ::
208
209 ret = cpuhp_state_add_instance(Y_hp_online, &d->node);
210
211This function will add this instance to your previously allocated
212*Y_hp_online* state and invoke the previously registered callback
213(*Y_online*) on all online CPUs. The *node* element is a ``struct
214hlist_node`` member of your per-instance data structure.
215
216On removal of the instance: ::
217 cpuhp_state_remove_instance(Y_hp_online, &d->node)
218
219should be invoked which will invoke the teardown callback on all online
220CPUs.
221
222Manual setup
223~~~~~~~~~~~~
224Usually it is handy to invoke setup and teardown callbacks on registration or
225removal of a state because usually the operation needs to performed once a CPU
226goes online (offline) and during initial setup (shutdown) of the driver. However
227each registration and removal function is also available with a ``_nocalls``
228suffix which does not invoke the provided callbacks if the invocation of the
229callbacks is not desired. During the manual setup (or teardown) the functions
230``get_online_cpus()`` and ``put_online_cpus()`` should be used to inhibit CPU
231hotplug operations.
232
233
234The ordering of the events
235--------------------------
236The hotplug states are defined in ``include/linux/cpuhotplug.h``:
237
238* The states *CPUHP_OFFLINE* … *CPUHP_AP_OFFLINE* are invoked before the
239 CPU is up.
240* The states *CPUHP_AP_OFFLINE* … *CPUHP_AP_ONLINE* are invoked
241 just the after the CPU has been brought up. The interrupts are off and
242 the scheduler is not yet active on this CPU. Starting with *CPUHP_AP_OFFLINE*
243 the callbacks are invoked on the target CPU.
244* The states between *CPUHP_AP_ONLINE_DYN* and *CPUHP_AP_ONLINE_DYN_END* are
245 reserved for the dynamic allocation.
246* The states are invoked in the reverse order on CPU shutdown starting with
247 *CPUHP_ONLINE* and stopping at *CPUHP_OFFLINE*. Here the callbacks are
248 invoked on the CPU that will be shutdown until *CPUHP_AP_OFFLINE*.
249
250A dynamically allocated state via *CPUHP_AP_ONLINE_DYN* is often enough.
251However if an earlier invocation during the bring up or shutdown is required
252then an explicit state should be acquired. An explicit state might also be
253required if the hotplug event requires specific ordering in respect to
254another hotplug event.
255
256Testing of hotplug states
257=========================
258One way to verify whether a custom state is working as expected or not is to
259shutdown a CPU and then put it online again. It is also possible to put the CPU
260to certain state (for instance *CPUHP_AP_ONLINE*) and then go back to
261*CPUHP_ONLINE*. This would simulate an error one state after *CPUHP_AP_ONLINE*
262which would lead to rollback to the online state.
263
264All registered states are enumerated in ``/sys/devices/system/cpu/hotplug/states``: ::
265
266 $ tail /sys/devices/system/cpu/hotplug/states
267 138: mm/vmscan:online
268 139: mm/vmstat:online
269 140: lib/percpu_cnt:online
270 141: acpi/cpu-drv:online
271 142: base/cacheinfo:online
272 143: virtio/net:online
273 144: x86/mce:online
274 145: printk:online
275 168: sched:active
276 169: online
277
278To rollback CPU4 to ``lib/percpu_cnt:online`` and back online just issue: ::
279
280 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
281 169
282 $ echo 140 > /sys/devices/system/cpu/cpu4/hotplug/target
283 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
284 140
285
286It is important to note that the teardown callbac of state 140 have been
287invoked. And now get back online: ::
288
289 $ echo 169 > /sys/devices/system/cpu/cpu4/hotplug/target
290 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
291 169
292
293With trace events enabled, the individual steps are visible, too: ::
294
295 # TASK-PID CPU# TIMESTAMP FUNCTION
296 # | | | | |
297 bash-394 [001] 22.976: cpuhp_enter: cpu: 0004 target: 140 step: 169 (cpuhp_kick_ap_work)
298 cpuhp/4-31 [004] 22.977: cpuhp_enter: cpu: 0004 target: 140 step: 168 (sched_cpu_deactivate)
299 cpuhp/4-31 [004] 22.990: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
300 cpuhp/4-31 [004] 22.991: cpuhp_enter: cpu: 0004 target: 140 step: 144 (mce_cpu_pre_down)
301 cpuhp/4-31 [004] 22.992: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
302 cpuhp/4-31 [004] 22.993: cpuhp_multi_enter: cpu: 0004 target: 140 step: 143 (virtnet_cpu_down_prep)
303 cpuhp/4-31 [004] 22.994: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
304 cpuhp/4-31 [004] 22.995: cpuhp_enter: cpu: 0004 target: 140 step: 142 (cacheinfo_cpu_pre_down)
305 cpuhp/4-31 [004] 22.996: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
306 bash-394 [001] 22.997: cpuhp_exit: cpu: 0004 state: 140 step: 169 ret: 0
307 bash-394 [005] 95.540: cpuhp_enter: cpu: 0004 target: 169 step: 140 (cpuhp_kick_ap_work)
308 cpuhp/4-31 [004] 95.541: cpuhp_enter: cpu: 0004 target: 169 step: 141 (acpi_soft_cpu_online)
309 cpuhp/4-31 [004] 95.542: cpuhp_exit: cpu: 0004 state: 141 step: 141 ret: 0
310 cpuhp/4-31 [004] 95.543: cpuhp_enter: cpu: 0004 target: 169 step: 142 (cacheinfo_cpu_online)
311 cpuhp/4-31 [004] 95.544: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
312 cpuhp/4-31 [004] 95.545: cpuhp_multi_enter: cpu: 0004 target: 169 step: 143 (virtnet_cpu_online)
313 cpuhp/4-31 [004] 95.546: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
314 cpuhp/4-31 [004] 95.547: cpuhp_enter: cpu: 0004 target: 169 step: 144 (mce_cpu_online)
315 cpuhp/4-31 [004] 95.548: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
316 cpuhp/4-31 [004] 95.549: cpuhp_enter: cpu: 0004 target: 169 step: 145 (console_cpu_notify)
317 cpuhp/4-31 [004] 95.550: cpuhp_exit: cpu: 0004 state: 145 step: 145 ret: 0
318 cpuhp/4-31 [004] 95.551: cpuhp_enter: cpu: 0004 target: 169 step: 168 (sched_cpu_activate)
319 cpuhp/4-31 [004] 95.552: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
320 bash-394 [005] 95.553: cpuhp_exit: cpu: 0004 state: 169 step: 140 ret: 0
321
322As it an be seen, CPU4 went down until timestamp 22.996 and then back up until
32395.552. All invoked callbacks including their return codes are visible in the
324trace.
325
326Architecture's requirements
327===========================
328The following functions and configurations are required:
329
330``CONFIG_HOTPLUG_CPU``
331 This entry needs to be enabled in Kconfig
332
333``__cpu_up()``
334 Arch interface to bring up a CPU
335
336``__cpu_disable()``
337 Arch interface to shutdown a CPU, no more interrupts can be handled by the
338 kernel after the routine returns. This includes the shutdown of the timer.
339
340``__cpu_die()``
341 This actually supposed to ensure death of the CPU. Actually look at some
342 example code in other arch that implement CPU hotplug. The processor is taken
343 down from the ``idle()`` loop for that specific architecture. ``__cpu_die()``
344 typically waits for some per_cpu state to be set, to ensure the processor dead
345 routine is called to be sure positively.
346
347User Space Notification
348=======================
349After CPU successfully onlined or offline udev events are sent. A udev rule like: ::
350
351 SUBSYSTEM=="cpu", DRIVERS=="processor", DEVPATH=="/devices/system/cpu/*", RUN+="the_hotplug_receiver.sh"
352
353will receive all events. A script like: ::
354
355 #!/bin/sh
356
357 if [ "${ACTION}" = "offline" ]
358 then
359 echo "CPU ${DEVPATH##*/} offline"
360
361 elif [ "${ACTION}" = "online" ]
362 then
363 echo "CPU ${DEVPATH##*/} online"
364
365 fi
366
367can process the event further.
368
369Kernel Inline Documentations Reference
370======================================
371
372.. kernel-doc:: include/linux/cpuhotplug.h