Loading...
1=================
2Freezing of tasks
3=================
4
5(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
6
7I. What is the freezing of tasks?
8=================================
9
10The freezing of tasks is a mechanism by which user space processes and some
11kernel threads are controlled during hibernation or system-wide suspend (on some
12architectures).
13
14II. How does it work?
15=====================
16
17There is one per-task flag (PF_NOFREEZE) and three per-task states
18(TASK_FROZEN, TASK_FREEZABLE and __TASK_FREEZABLE_UNSAFE) used for that.
19The tasks that have PF_NOFREEZE unset (all user space tasks and some kernel
20threads) are regarded as 'freezable' and treated in a special way before the
21system enters a sleep state as well as before a hibernation image is created
22(hibernation is directly covered by what follows, but the description applies
23to system-wide suspend too).
24
25Namely, as the first step of the hibernation procedure the function
26freeze_processes() (defined in kernel/power/process.c) is called. A system-wide
27static key freezer_active (as opposed to a per-task flag or state) is used to
28indicate whether the system is to undergo a freezing operation. And
29freeze_processes() sets this static key. After this, it executes
30try_to_freeze_tasks() that sends a fake signal to all user space processes, and
31wakes up all the kernel threads. All freezable tasks must react to that by
32calling try_to_freeze(), which results in a call to __refrigerator() (defined
33in kernel/freezer.c), which changes the task's state to TASK_FROZEN, and makes
34it loop until it is woken by an explicit TASK_FROZEN wakeup. Then, that task
35is regarded as 'frozen' and so the set of functions handling this mechanism is
36referred to as 'the freezer' (these functions are defined in
37kernel/power/process.c, kernel/freezer.c & include/linux/freezer.h). User space
38tasks are generally frozen before kernel threads.
39
40__refrigerator() must not be called directly. Instead, use the
41try_to_freeze() function (defined in include/linux/freezer.h), that checks
42if the task is to be frozen and makes the task enter __refrigerator().
43
44For user space processes try_to_freeze() is called automatically from the
45signal-handling code, but the freezable kernel threads need to call it
46explicitly in suitable places or use the wait_event_freezable() or
47wait_event_freezable_timeout() macros (defined in include/linux/wait.h)
48that put the task to sleep (TASK_INTERRUPTIBLE) or freeze it (TASK_FROZEN) if
49freezer_active is set. The main loop of a freezable kernel thread may look
50like the following one::
51
52 set_freezable();
53
54 while (true) {
55 struct task_struct *tsk = NULL;
56
57 wait_event_freezable(oom_reaper_wait, oom_reaper_list != NULL);
58 spin_lock_irq(&oom_reaper_lock);
59 if (oom_reaper_list != NULL) {
60 tsk = oom_reaper_list;
61 oom_reaper_list = tsk->oom_reaper_list;
62 }
63 spin_unlock_irq(&oom_reaper_lock);
64
65 if (tsk)
66 oom_reap_task(tsk);
67 }
68
69(from mm/oom_kill.c::oom_reaper()).
70
71If a freezable kernel thread is not put to the frozen state after the freezer
72has initiated a freezing operation, the freezing of tasks will fail and the
73entire system-wide transition will be cancelled. For this reason, freezable
74kernel threads must call try_to_freeze() somewhere or use one of the
75wait_event_freezable() and wait_event_freezable_timeout() macros.
76
77After the system memory state has been restored from a hibernation image and
78devices have been reinitialized, the function thaw_processes() is called in
79order to wake up each frozen task. Then, the tasks that have been frozen leave
80__refrigerator() and continue running.
81
82
83Rationale behind the functions dealing with freezing and thawing of tasks
84-------------------------------------------------------------------------
85
86freeze_processes():
87 - freezes only userspace tasks
88
89freeze_kernel_threads():
90 - freezes all tasks (including kernel threads) because we can't freeze
91 kernel threads without freezing userspace tasks
92
93thaw_kernel_threads():
94 - thaws only kernel threads; this is particularly useful if we need to do
95 anything special in between thawing of kernel threads and thawing of
96 userspace tasks, or if we want to postpone the thawing of userspace tasks
97
98thaw_processes():
99 - thaws all tasks (including kernel threads) because we can't thaw userspace
100 tasks without thawing kernel threads
101
102
103III. Which kernel threads are freezable?
104========================================
105
106Kernel threads are not freezable by default. However, a kernel thread may clear
107PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
108directly is not allowed). From this point it is regarded as freezable
109and must call try_to_freeze() or variants of wait_event_freezable() in a
110suitable place.
111
112IV. Why do we do that?
113======================
114
115Generally speaking, there is a couple of reasons to use the freezing of tasks:
116
1171. The principal reason is to prevent filesystems from being damaged after
118 hibernation. At the moment we have no simple means of checkpointing
119 filesystems, so if there are any modifications made to filesystem data and/or
120 metadata on disks, we cannot bring them back to the state from before the
121 modifications. At the same time each hibernation image contains some
122 filesystem-related information that must be consistent with the state of the
123 on-disk data and metadata after the system memory state has been restored
124 from the image (otherwise the filesystems will be damaged in a nasty way,
125 usually making them almost impossible to repair). We therefore freeze
126 tasks that might cause the on-disk filesystems' data and metadata to be
127 modified after the hibernation image has been created and before the
128 system is finally powered off. The majority of these are user space
129 processes, but if any of the kernel threads may cause something like this
130 to happen, they have to be freezable.
131
1322. Next, to create the hibernation image we need to free a sufficient amount of
133 memory (approximately 50% of available RAM) and we need to do that before
134 devices are deactivated, because we generally need them for swapping out.
135 Then, after the memory for the image has been freed, we don't want tasks
136 to allocate additional memory and we prevent them from doing that by
137 freezing them earlier. [Of course, this also means that device drivers
138 should not allocate substantial amounts of memory from their .suspend()
139 callbacks before hibernation, but this is a separate issue.]
140
1413. The third reason is to prevent user space processes and some kernel threads
142 from interfering with the suspending and resuming of devices. A user space
143 process running on a second CPU while we are suspending devices may, for
144 example, be troublesome and without the freezing of tasks we would need some
145 safeguards against race conditions that might occur in such a case.
146
147Although Linus Torvalds doesn't like the freezing of tasks, he said this in one
148of the discussions on LKML (https://lore.kernel.org/r/alpine.LFD.0.98.0704271801020.9964@woody.linux-foundation.org):
149
150"RJW:> Why we freeze tasks at all or why we freeze kernel threads?
151
152Linus: In many ways, 'at all'.
153
154I **do** realize the IO request queue issues, and that we cannot actually do
155s2ram with some devices in the middle of a DMA. So we want to be able to
156avoid *that*, there's no question about that. And I suspect that stopping
157user threads and then waiting for a sync is practically one of the easier
158ways to do so.
159
160So in practice, the 'at all' may become a 'why freeze kernel threads?' and
161freezing user threads I don't find really objectionable."
162
163Still, there are kernel threads that may want to be freezable. For example, if
164a kernel thread that belongs to a device driver accesses the device directly, it
165in principle needs to know when the device is suspended, so that it doesn't try
166to access it at that time. However, if the kernel thread is freezable, it will
167be frozen before the driver's .suspend() callback is executed and it will be
168thawed after the driver's .resume() callback has run, so it won't be accessing
169the device while it's suspended.
170
1714. Another reason for freezing tasks is to prevent user space processes from
172 realizing that hibernation (or suspend) operation takes place. Ideally, user
173 space processes should not notice that such a system-wide operation has
174 occurred and should continue running without any problems after the restore
175 (or resume from suspend). Unfortunately, in the most general case this
176 is quite difficult to achieve without the freezing of tasks. Consider,
177 for example, a process that depends on all CPUs being online while it's
178 running. Since we need to disable nonboot CPUs during the hibernation,
179 if this process is not frozen, it may notice that the number of CPUs has
180 changed and may start to work incorrectly because of that.
181
182V. Are there any problems related to the freezing of tasks?
183===========================================================
184
185Yes, there are.
186
187First of all, the freezing of kernel threads may be tricky if they depend one
188on another. For example, if kernel thread A waits for a completion (in the
189TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B
190and B is frozen in the meantime, then A will be blocked until B is thawed, which
191may be undesirable. That's why kernel threads are not freezable by default.
192
193Second, there are the following two problems related to the freezing of user
194space processes:
195
1961. Putting processes into an uninterruptible sleep distorts the load average.
1972. Now that we have FUSE, plus the framework for doing device drivers in
198 userspace, it gets even more complicated because some userspace processes are
199 now doing the sorts of things that kernel threads do
200 (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).
201
202The problem 1. seems to be fixable, although it hasn't been fixed so far. The
203other one is more serious, but it seems that we can work around it by using
204hibernation (and suspend) notifiers (in that case, though, we won't be able to
205avoid the realization by the user space processes that the hibernation is taking
206place).
207
208There are also problems that the freezing of tasks tends to expose, although
209they are not directly related to it. For example, if request_firmware() is
210called from a device driver's .resume() routine, it will timeout and eventually
211fail, because the user land process that should respond to the request is frozen
212at this point. So, seemingly, the failure is due to the freezing of tasks.
213Suppose, however, that the firmware file is located on a filesystem accessible
214only through another device that hasn't been resumed yet. In that case,
215request_firmware() will fail regardless of whether or not the freezing of tasks
216is used. Consequently, the problem is not really related to the freezing of
217tasks, since it generally exists anyway.
218
219A driver must have all firmwares it may need in RAM before suspend() is called.
220If keeping them is not practical, for example due to their size, they must be
221requested early enough using the suspend notifier API described in
222Documentation/driver-api/pm/notifiers.rst.
223
224VI. Are there any precautions to be taken to prevent freezing failures?
225=======================================================================
226
227Yes, there are.
228
229First of all, grabbing the 'system_transition_mutex' lock to mutually exclude a
230piece of code from system-wide sleep such as suspend/hibernation is not
231encouraged. If possible, that piece of code must instead hook onto the
232suspend/hibernation notifiers to achieve mutual exclusion. Look at the
233CPU-Hotplug code (kernel/cpu.c) for an example.
234
235However, if that is not feasible, and grabbing 'system_transition_mutex' is
236deemed necessary, it is strongly discouraged to directly call
237mutex_[un]lock(&system_transition_mutex) since that could lead to freezing
238failures, because if the suspend/hibernate code successfully acquired the
239'system_transition_mutex' lock, and hence that other entity failed to acquire
240the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE state. As a
241consequence, the freezer would not be able to freeze that task, leading to
242freezing failure.
243
244However, the [un]lock_system_sleep() APIs are safe to use in this scenario,
245since they ask the freezer to skip freezing this task, since it is anyway
246"frozen enough" as it is blocked on 'system_transition_mutex', which will be
247released only after the entire suspend/hibernation sequence is complete. So, to
248summarize, use [un]lock_system_sleep() instead of directly using
249mutex_[un]lock(&system_transition_mutex). That would prevent freezing failures.
250
251V. Miscellaneous
252================
253
254/sys/power/pm_freeze_timeout controls how long it will cost at most to freeze
255all user space processes or all freezable kernel threads, in unit of
256millisecond. The default value is 20000, with range of unsigned integer.
1=================
2Freezing of tasks
3=================
4
5(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
6
7I. What is the freezing of tasks?
8=================================
9
10The freezing of tasks is a mechanism by which user space processes and some
11kernel threads are controlled during hibernation or system-wide suspend (on some
12architectures).
13
14II. How does it work?
15=====================
16
17There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN
18and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have
19PF_NOFREEZE unset (all user space processes and some kernel threads) are
20regarded as 'freezable' and treated in a special way before the system enters a
21suspend state as well as before a hibernation image is created (in what follows
22we only consider hibernation, but the description also applies to suspend).
23
24Namely, as the first step of the hibernation procedure the function
25freeze_processes() (defined in kernel/power/process.c) is called. A system-wide
26variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate
27whether the system is to undergo a freezing operation. And freeze_processes()
28sets this variable. After this, it executes try_to_freeze_tasks() that sends a
29fake signal to all user space processes, and wakes up all the kernel threads.
30All freezable tasks must react to that by calling try_to_freeze(), which
31results in a call to __refrigerator() (defined in kernel/freezer.c), which sets
32the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes
33it loop until PF_FROZEN is cleared for it. Then, we say that the task is
34'frozen' and therefore the set of functions handling this mechanism is referred
35to as 'the freezer' (these functions are defined in kernel/power/process.c,
36kernel/freezer.c & include/linux/freezer.h). User space processes are generally
37frozen before kernel threads.
38
39__refrigerator() must not be called directly. Instead, use the
40try_to_freeze() function (defined in include/linux/freezer.h), that checks
41if the task is to be frozen and makes the task enter __refrigerator().
42
43For user space processes try_to_freeze() is called automatically from the
44signal-handling code, but the freezable kernel threads need to call it
45explicitly in suitable places or use the wait_event_freezable() or
46wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
47that combine interruptible sleep with checking if the task is to be frozen and
48calling try_to_freeze(). The main loop of a freezable kernel thread may look
49like the following one::
50
51 set_freezable();
52 do {
53 hub_events();
54 wait_event_freezable(khubd_wait,
55 !list_empty(&hub_event_list) ||
56 kthread_should_stop());
57 } while (!kthread_should_stop() || !list_empty(&hub_event_list));
58
59(from drivers/usb/core/hub.c::hub_thread()).
60
61If a freezable kernel thread fails to call try_to_freeze() after the freezer has
62initiated a freezing operation, the freezing of tasks will fail and the entire
63hibernation operation will be cancelled. For this reason, freezable kernel
64threads must call try_to_freeze() somewhere or use one of the
65wait_event_freezable() and wait_event_freezable_timeout() macros.
66
67After the system memory state has been restored from a hibernation image and
68devices have been reinitialized, the function thaw_processes() is called in
69order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that
70have been frozen leave __refrigerator() and continue running.
71
72
73Rationale behind the functions dealing with freezing and thawing of tasks
74-------------------------------------------------------------------------
75
76freeze_processes():
77 - freezes only userspace tasks
78
79freeze_kernel_threads():
80 - freezes all tasks (including kernel threads) because we can't freeze
81 kernel threads without freezing userspace tasks
82
83thaw_kernel_threads():
84 - thaws only kernel threads; this is particularly useful if we need to do
85 anything special in between thawing of kernel threads and thawing of
86 userspace tasks, or if we want to postpone the thawing of userspace tasks
87
88thaw_processes():
89 - thaws all tasks (including kernel threads) because we can't thaw userspace
90 tasks without thawing kernel threads
91
92
93III. Which kernel threads are freezable?
94========================================
95
96Kernel threads are not freezable by default. However, a kernel thread may clear
97PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
98directly is not allowed). From this point it is regarded as freezable
99and must call try_to_freeze() in a suitable place.
100
101IV. Why do we do that?
102======================
103
104Generally speaking, there is a couple of reasons to use the freezing of tasks:
105
1061. The principal reason is to prevent filesystems from being damaged after
107 hibernation. At the moment we have no simple means of checkpointing
108 filesystems, so if there are any modifications made to filesystem data and/or
109 metadata on disks, we cannot bring them back to the state from before the
110 modifications. At the same time each hibernation image contains some
111 filesystem-related information that must be consistent with the state of the
112 on-disk data and metadata after the system memory state has been restored
113 from the image (otherwise the filesystems will be damaged in a nasty way,
114 usually making them almost impossible to repair). We therefore freeze
115 tasks that might cause the on-disk filesystems' data and metadata to be
116 modified after the hibernation image has been created and before the
117 system is finally powered off. The majority of these are user space
118 processes, but if any of the kernel threads may cause something like this
119 to happen, they have to be freezable.
120
1212. Next, to create the hibernation image we need to free a sufficient amount of
122 memory (approximately 50% of available RAM) and we need to do that before
123 devices are deactivated, because we generally need them for swapping out.
124 Then, after the memory for the image has been freed, we don't want tasks
125 to allocate additional memory and we prevent them from doing that by
126 freezing them earlier. [Of course, this also means that device drivers
127 should not allocate substantial amounts of memory from their .suspend()
128 callbacks before hibernation, but this is a separate issue.]
129
1303. The third reason is to prevent user space processes and some kernel threads
131 from interfering with the suspending and resuming of devices. A user space
132 process running on a second CPU while we are suspending devices may, for
133 example, be troublesome and without the freezing of tasks we would need some
134 safeguards against race conditions that might occur in such a case.
135
136Although Linus Torvalds doesn't like the freezing of tasks, he said this in one
137of the discussions on LKML (https://lore.kernel.org/r/alpine.LFD.0.98.0704271801020.9964@woody.linux-foundation.org):
138
139"RJW:> Why we freeze tasks at all or why we freeze kernel threads?
140
141Linus: In many ways, 'at all'.
142
143I **do** realize the IO request queue issues, and that we cannot actually do
144s2ram with some devices in the middle of a DMA. So we want to be able to
145avoid *that*, there's no question about that. And I suspect that stopping
146user threads and then waiting for a sync is practically one of the easier
147ways to do so.
148
149So in practice, the 'at all' may become a 'why freeze kernel threads?' and
150freezing user threads I don't find really objectionable."
151
152Still, there are kernel threads that may want to be freezable. For example, if
153a kernel thread that belongs to a device driver accesses the device directly, it
154in principle needs to know when the device is suspended, so that it doesn't try
155to access it at that time. However, if the kernel thread is freezable, it will
156be frozen before the driver's .suspend() callback is executed and it will be
157thawed after the driver's .resume() callback has run, so it won't be accessing
158the device while it's suspended.
159
1604. Another reason for freezing tasks is to prevent user space processes from
161 realizing that hibernation (or suspend) operation takes place. Ideally, user
162 space processes should not notice that such a system-wide operation has
163 occurred and should continue running without any problems after the restore
164 (or resume from suspend). Unfortunately, in the most general case this
165 is quite difficult to achieve without the freezing of tasks. Consider,
166 for example, a process that depends on all CPUs being online while it's
167 running. Since we need to disable nonboot CPUs during the hibernation,
168 if this process is not frozen, it may notice that the number of CPUs has
169 changed and may start to work incorrectly because of that.
170
171V. Are there any problems related to the freezing of tasks?
172===========================================================
173
174Yes, there are.
175
176First of all, the freezing of kernel threads may be tricky if they depend one
177on another. For example, if kernel thread A waits for a completion (in the
178TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B
179and B is frozen in the meantime, then A will be blocked until B is thawed, which
180may be undesirable. That's why kernel threads are not freezable by default.
181
182Second, there are the following two problems related to the freezing of user
183space processes:
184
1851. Putting processes into an uninterruptible sleep distorts the load average.
1862. Now that we have FUSE, plus the framework for doing device drivers in
187 userspace, it gets even more complicated because some userspace processes are
188 now doing the sorts of things that kernel threads do
189 (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).
190
191The problem 1. seems to be fixable, although it hasn't been fixed so far. The
192other one is more serious, but it seems that we can work around it by using
193hibernation (and suspend) notifiers (in that case, though, we won't be able to
194avoid the realization by the user space processes that the hibernation is taking
195place).
196
197There are also problems that the freezing of tasks tends to expose, although
198they are not directly related to it. For example, if request_firmware() is
199called from a device driver's .resume() routine, it will timeout and eventually
200fail, because the user land process that should respond to the request is frozen
201at this point. So, seemingly, the failure is due to the freezing of tasks.
202Suppose, however, that the firmware file is located on a filesystem accessible
203only through another device that hasn't been resumed yet. In that case,
204request_firmware() will fail regardless of whether or not the freezing of tasks
205is used. Consequently, the problem is not really related to the freezing of
206tasks, since it generally exists anyway.
207
208A driver must have all firmwares it may need in RAM before suspend() is called.
209If keeping them is not practical, for example due to their size, they must be
210requested early enough using the suspend notifier API described in
211Documentation/driver-api/pm/notifiers.rst.
212
213VI. Are there any precautions to be taken to prevent freezing failures?
214=======================================================================
215
216Yes, there are.
217
218First of all, grabbing the 'system_transition_mutex' lock to mutually exclude a
219piece of code from system-wide sleep such as suspend/hibernation is not
220encouraged. If possible, that piece of code must instead hook onto the
221suspend/hibernation notifiers to achieve mutual exclusion. Look at the
222CPU-Hotplug code (kernel/cpu.c) for an example.
223
224However, if that is not feasible, and grabbing 'system_transition_mutex' is
225deemed necessary, it is strongly discouraged to directly call
226mutex_[un]lock(&system_transition_mutex) since that could lead to freezing
227failures, because if the suspend/hibernate code successfully acquired the
228'system_transition_mutex' lock, and hence that other entity failed to acquire
229the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE state. As a
230consequence, the freezer would not be able to freeze that task, leading to
231freezing failure.
232
233However, the [un]lock_system_sleep() APIs are safe to use in this scenario,
234since they ask the freezer to skip freezing this task, since it is anyway
235"frozen enough" as it is blocked on 'system_transition_mutex', which will be
236released only after the entire suspend/hibernation sequence is complete. So, to
237summarize, use [un]lock_system_sleep() instead of directly using
238mutex_[un]lock(&system_transition_mutex). That would prevent freezing failures.
239
240V. Miscellaneous
241================
242
243/sys/power/pm_freeze_timeout controls how long it will cost at most to freeze
244all user space processes or all freezable kernel threads, in unit of
245millisecond. The default value is 20000, with range of unsigned integer.