1MARKING SHARED-MEMORY ACCESSES
  2==============================
  3
  4This document provides guidelines for marking intentionally concurrent
  5normal accesses to shared memory, that is "normal" as in accesses that do
  6not use read-modify-write atomic operations.  It also describes how to
  7document these accesses, both with comments and with special assertions
  8processed by the Kernel Concurrency Sanitizer (KCSAN).  This discussion
  9builds on an earlier LWN article [1] and Linux Foundation mentorship
 10session [2].
 11
 12
 13ACCESS-MARKING OPTIONS
 14======================
 15
 16The Linux kernel provides the following access-marking options:
 17
 181.	Plain C-language accesses (unmarked), for example, "a = b;"
 19
 202.	Data-race marking, for example, "data_race(a = b);"
 21
 223.	READ_ONCE(), for example, "a = READ_ONCE(b);"
 23	The various forms of atomic_read() also fit in here.
 24
 254.	WRITE_ONCE(), for example, "WRITE_ONCE(a, b);"
 26	The various forms of atomic_set() also fit in here.
 27
 285.	__data_racy, for example "int __data_racy a;"
 29
 306.	KCSAN's negative-marking assertions, ASSERT_EXCLUSIVE_ACCESS()
 31	and ASSERT_EXCLUSIVE_WRITER(), are described in the
 32	"ACCESS-DOCUMENTATION OPTIONS" section below.
 33
 34These may be used in combination, as shown in this admittedly improbable
 35example:
 36
 37	WRITE_ONCE(a, b + data_race(c + d) + READ_ONCE(e));
 38
 39Neither plain C-language accesses nor data_race() (#1 and #2 above) place
 40any sort of constraint on the compiler's choice of optimizations [3].
 41In contrast, READ_ONCE() and WRITE_ONCE() (#3 and #4 above) restrict the
 42compiler's use of code-motion and common-subexpression optimizations.
 43Therefore, if a given access is involved in an intentional data race,
 44using READ_ONCE() for loads and WRITE_ONCE() for stores is usually
 45preferable to data_race(), which in turn is usually preferable to plain
 46C-language accesses.  It is permissible to combine #2 and #3, for example,
 47data_race(READ_ONCE(a)), which will both restrict compiler optimizations
 48and disable KCSAN diagnostics.
 49
 50KCSAN will complain about many types of data races involving plain
 51C-language accesses, but marking all accesses involved in a given data
 52race with one of data_race(), READ_ONCE(), or WRITE_ONCE(), will prevent
 53KCSAN from complaining.  Of course, lack of KCSAN complaints does not
 54imply correct code.  Therefore, please take a thoughtful approach
 55when responding to KCSAN complaints.  Churning the code base with
 56ill-considered additions of data_race(), READ_ONCE(), and WRITE_ONCE()
 57is unhelpful.
 58
 59In fact, the following sections describe situations where use of
 60data_race() and even plain C-language accesses is preferable to
 61READ_ONCE() and WRITE_ONCE().
 62
 63
 64Use of the data_race() Macro
 65----------------------------
 66
 67Here are some situations where data_race() should be used instead of
 68READ_ONCE() and WRITE_ONCE():
 69
 701.	Data-racy loads from shared variables whose values are used only
 71	for diagnostic purposes.
 72
 732.	Data-racy reads whose values are checked against marked reload.
 74
 753.	Reads whose values feed into error-tolerant heuristics.
 76
 774.	Writes setting values that feed into error-tolerant heuristics.
 78
 79
 80Data-Racy Reads for Approximate Diagnostics
 81
 82Approximate diagnostics include lockdep reports, monitoring/statistics
 83(including /proc and /sys output), WARN*()/BUG*() checks whose return
 84values are ignored, and other situations where reads from shared variables
 85are not an integral part of the core concurrency design.
 86
 87In fact, use of data_race() instead READ_ONCE() for these diagnostic
 88reads can enable better checking of the remaining accesses implementing
 89the core concurrency design.  For example, suppose that the core design
 90prevents any non-diagnostic reads from shared variable x from running
 91concurrently with updates to x.  Then using plain C-language writes
 92to x allows KCSAN to detect reads from x from within regions of code
 93that fail to exclude the updates.  In this case, it is important to use
 94data_race() for the diagnostic reads because otherwise KCSAN would give
 95false-positive warnings about these diagnostic reads.
 96
 97If it is necessary to both restrict compiler optimizations and disable
 98KCSAN diagnostics, use both data_race() and READ_ONCE(), for example,
 99data_race(READ_ONCE(a)).
100
101In theory, plain C-language loads can also be used for this use case.
102However, in practice this will have the disadvantage of causing KCSAN
103to generate false positives because KCSAN will have no way of knowing
104that the resulting data race was intentional.
105
106
107Data-Racy Reads That Are Checked Against Marked Reload
108
109The values from some reads are not implicitly trusted.  They are instead
110fed into some operation that checks the full value against a later marked
111load from memory, which means that the occasional arbitrarily bogus value
112is not a problem.  For example, if a bogus value is fed into cmpxchg(),
113all that happens is that this cmpxchg() fails, which normally results
114in a retry.  Unless the race condition that resulted in the bogus value
115recurs, this retry will with high probability succeed, so no harm done.
116
117However, please keep in mind that a data_race() load feeding into
118a cmpxchg_relaxed() might still be subject to load fusing on some
119architectures.  Therefore, it is best to capture the return value from
120the failing cmpxchg() for the next iteration of the loop, an approach
121that provides the compiler much less scope for mischievous optimizations.
122Capturing the return value from cmpxchg() also saves a memory reference
123in many cases.
124
125In theory, plain C-language loads can also be used for this use case.
126However, in practice this will have the disadvantage of causing KCSAN
127to generate false positives because KCSAN will have no way of knowing
128that the resulting data race was intentional.
129
130
131Reads Feeding Into Error-Tolerant Heuristics
132
133Values from some reads feed into heuristics that can tolerate occasional
134errors.  Such reads can use data_race(), thus allowing KCSAN to focus on
135the other accesses to the relevant shared variables.  But please note
136that data_race() loads are subject to load fusing, which can result in
137consistent errors, which in turn are quite capable of breaking heuristics.
138Therefore use of data_race() should be limited to cases where some other
139code (such as a barrier() call) will force the occasional reload.
140
141Note that this use case requires that the heuristic be able to handle
142any possible error.  In contrast, if the heuristics might be fatally
143confused by one or more of the possible erroneous values, use READ_ONCE()
144instead of data_race().
145
146In theory, plain C-language loads can also be used for this use case.
147However, in practice this will have the disadvantage of causing KCSAN
148to generate false positives because KCSAN will have no way of knowing
149that the resulting data race was intentional.
150
151
152Writes Setting Values Feeding Into Error-Tolerant Heuristics
153
154The values read into error-tolerant heuristics come from somewhere,
155for example, from sysfs.  This means that some code in sysfs writes
156to this same variable, and these writes can also use data_race().
157After all, if the heuristic can tolerate the occasional bogus value
158due to compiler-mangled reads, it can also tolerate the occasional
159compiler-mangled write, at least assuming that the proper value is in
160place once the write completes.
161
162Plain C-language stores can also be used for this use case.  However,
163in kernels built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n, this
164will have the disadvantage of causing KCSAN to generate false positives
165because KCSAN will have no way of knowing that the resulting data race
166was intentional.
167
168
169Use of Plain C-Language Accesses
170--------------------------------
171
172Here are some example situations where plain C-language accesses should
173used instead of READ_ONCE(), WRITE_ONCE(), and data_race():
174
1751.	Accesses protected by mutual exclusion, including strict locking
176	and sequence locking.
177
1782.	Initialization-time and cleanup-time accesses.	This covers a
179	wide variety of situations, including the uniprocessor phase of
180	system boot, variables to be used by not-yet-spawned kthreads,
181	structures not yet published to reference-counted or RCU-protected
182	data structures, and the cleanup side of any of these situations.
183
1843.	Per-CPU variables that are not accessed from other CPUs.
185
1864.	Private per-task variables, including on-stack variables, some
187	fields in the task_struct structure, and task-private heap data.
188
1895.	Any other loads for which there is not supposed to be a concurrent
190	store to that same variable.
191
1926.	Any other stores for which there should be neither concurrent
193	loads nor concurrent stores to that same variable.
194
195	But note that KCSAN makes two explicit exceptions to this rule
196	by default, refraining from flagging plain C-language stores:
197
198	a.	No matter what.  You can override this default by building
199		with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n.
200
201	b.	When the store writes the value already contained in
202		that variable.	You can override this default by building
203		with CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n.
204
205	c.	When one of the stores is in an interrupt handler and
206		the other in the interrupted code.  You can override this
207		default by building with CONFIG_KCSAN_INTERRUPT_WATCHER=y.
208
209Note that it is important to use plain C-language accesses in these cases,
210because doing otherwise prevents KCSAN from detecting violations of your
211code's synchronization rules.
212
213
214Use of __data_racy
215------------------
216
217Adding the __data_racy type qualifier to the declaration of a variable
218causes KCSAN to treat all accesses to that variable as if they were
219enclosed by data_race().  However, __data_racy does not affect the
220compiler, though one could imagine hardened kernel builds treating the
221__data_racy type qualifier as if it was the volatile keyword.
222
223Note well that __data_racy is subject to the same pointer-declaration
224rules as are other type qualifiers such as const and volatile.
225For example:
226
227	int __data_racy *p; // Pointer to data-racy data.
228	int *__data_racy p; // Data-racy pointer to non-data-racy data.
229
230
231ACCESS-DOCUMENTATION OPTIONS
232============================
233
234It is important to comment marked accesses so that people reading your
235code, yourself included, are reminded of the synchronization design.
236However, it is even more important to comment plain C-language accesses
237that are intentionally involved in data races.  Such comments are
238needed to remind people reading your code, again, yourself included,
239of how the compiler has been prevented from optimizing those accesses
240into concurrency bugs.
241
242It is also possible to tell KCSAN about your synchronization design.
243For example, ASSERT_EXCLUSIVE_ACCESS(foo) tells KCSAN that any
244concurrent access to variable foo by any other CPU is an error, even
245if that concurrent access is marked with READ_ONCE().  In addition,
246ASSERT_EXCLUSIVE_WRITER(foo) tells KCSAN that although it is OK for there
247to be concurrent reads from foo from other CPUs, it is an error for some
248other CPU to be concurrently writing to foo, even if that concurrent
249write is marked with data_race() or WRITE_ONCE().
250
251Note that although KCSAN will call out data races involving either
252ASSERT_EXCLUSIVE_ACCESS() or ASSERT_EXCLUSIVE_WRITER() on the one hand
253and data_race() writes on the other, KCSAN will not report the location
254of these data_race() writes.
255
256
257EXAMPLES
258========
259
260As noted earlier, the goal is to prevent the compiler from destroying
261your concurrent algorithm, to help the human reader, and to inform
262KCSAN of aspects of your concurrency design.  This section looks at a
263few examples showing how this can be done.
264
265
266Lock Protection With Lockless Diagnostic Access
267-----------------------------------------------
268
269For example, suppose a shared variable "foo" is read only while a
270reader-writer spinlock is read-held, written only while that same
271spinlock is write-held, except that it is also read locklessly for
272diagnostic purposes.  The code might look as follows:
273
274	int foo;
275	DEFINE_RWLOCK(foo_rwlock);
276
277	void update_foo(int newval)
278	{
279		write_lock(&foo_rwlock);
280		foo = newval;
281		do_something(newval);
282		write_unlock(&foo_rwlock);
283	}
284
285	int read_foo(void)
286	{
287		int ret;
288
289		read_lock(&foo_rwlock);
290		do_something_else();
291		ret = foo;
292		read_unlock(&foo_rwlock);
293		return ret;
294	}
295
296	void read_foo_diagnostic(void)
297	{
298		pr_info("Current value of foo: %d\n", data_race(foo));
299	}
300
301The reader-writer lock prevents the compiler from introducing concurrency
302bugs into any part of the main algorithm using foo, which means that
303the accesses to foo within both update_foo() and read_foo() can (and
304should) be plain C-language accesses.  One benefit of making them be
305plain C-language accesses is that KCSAN can detect any erroneous lockless
306reads from or updates to foo.  The data_race() in read_foo_diagnostic()
307tells KCSAN that data races are expected, and should be silently
308ignored.  This data_race() also tells the human reading the code that
309read_foo_diagnostic() might sometimes return a bogus value.
310
311If it is necessary to suppress compiler optimization and also detect
312buggy lockless writes, read_foo_diagnostic() can be updated as follows:
313
314	void read_foo_diagnostic(void)
315	{
316		pr_info("Current value of foo: %d\n", data_race(READ_ONCE(foo)));
317	}
318
319Alternatively, given that KCSAN is to ignore all accesses in this function,
320this function can be marked __no_kcsan and the data_race() can be dropped:
321
322	void __no_kcsan read_foo_diagnostic(void)
323	{
324		pr_info("Current value of foo: %d\n", READ_ONCE(foo));
325	}
326
327However, in order for KCSAN to detect buggy lockless writes, your kernel
328must be built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n.  If you
329need KCSAN to detect such a write even if that write did not change
330the value of foo, you also need CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n.
331If you need KCSAN to detect such a write happening in an interrupt handler
332running on the same CPU doing the legitimate lock-protected write, you
333also need CONFIG_KCSAN_INTERRUPT_WATCHER=y.  With some or all of these
334Kconfig options set properly, KCSAN can be quite helpful, although
335it is not necessarily a full replacement for hardware watchpoints.
336On the other hand, neither are hardware watchpoints a full replacement
337for KCSAN because it is not always easy to tell hardware watchpoint to
338conditionally trap on accesses.
339
340
341Lock-Protected Writes With Lockless Reads
342-----------------------------------------
343
344For another example, suppose a shared variable "foo" is updated only
345while holding a spinlock, but is read locklessly.  The code might look
346as follows:
347
348	int foo;
349	DEFINE_SPINLOCK(foo_lock);
350
351	void update_foo(int newval)
352	{
353		spin_lock(&foo_lock);
354		WRITE_ONCE(foo, newval);
355		ASSERT_EXCLUSIVE_WRITER(foo);
356		do_something(newval);
357		spin_unlock(&foo_wlock);
358	}
359
360	int read_foo(void)
361	{
362		do_something_else();
363		return READ_ONCE(foo);
364	}
365
366Because foo is read locklessly, all accesses are marked.  The purpose
367of the ASSERT_EXCLUSIVE_WRITER() is to allow KCSAN to check for a buggy
368concurrent write, whether marked or not.
369
370
371Lock-Protected Writes With Heuristic Lockless Reads
372---------------------------------------------------
373
374For another example, suppose that the code can normally make use of
375a per-data-structure lock, but there are times when a global lock
376is required.  These times are indicated via a global flag.  The code
377might look as follows, and is based loosely on nf_conntrack_lock(),
378nf_conntrack_all_lock(), and nf_conntrack_all_unlock():
379
380	bool global_flag;
381	DEFINE_SPINLOCK(global_lock);
382	struct foo {
383		spinlock_t f_lock;
384		int f_data;
385	};
386
387	/* All foo structures are in the following array. */
388	int nfoo;
389	struct foo *foo_array;
390
391	void do_something_locked(struct foo *fp)
392	{
393		/* This works even if data_race() returns nonsense. */
394		if (!data_race(global_flag)) {
395			spin_lock(&fp->f_lock);
396			if (!smp_load_acquire(&global_flag)) {
397				do_something(fp);
398				spin_unlock(&fp->f_lock);
399				return;
400			}
401			spin_unlock(&fp->f_lock);
402		}
403		spin_lock(&global_lock);
404		/* global_lock held, thus global flag cannot be set. */
405		spin_lock(&fp->f_lock);
406		spin_unlock(&global_lock);
407		/*
408		 * global_flag might be set here, but begin_global()
409		 * will wait for ->f_lock to be released.
410		 */
411		do_something(fp);
412		spin_unlock(&fp->f_lock);
413	}
414
415	void begin_global(void)
416	{
417		int i;
418
419		spin_lock(&global_lock);
420		WRITE_ONCE(global_flag, true);
421		for (i = 0; i < nfoo; i++) {
422			/*
423			 * Wait for pre-existing local locks.  One at
424			 * a time to avoid lockdep limitations.
425			 */
426			spin_lock(&fp->f_lock);
427			spin_unlock(&fp->f_lock);
428		}
429	}
430
431	void end_global(void)
432	{
433		smp_store_release(&global_flag, false);
434		spin_unlock(&global_lock);
435	}
436
437All code paths leading from the do_something_locked() function's first
438read from global_flag acquire a lock, so endless load fusing cannot
439happen.
440
441If the value read from global_flag is true, then global_flag is
442rechecked while holding ->f_lock, which, if global_flag is now false,
443prevents begin_global() from completing.  It is therefore safe to invoke
444do_something().
445
446Otherwise, if either value read from global_flag is true, then after
447global_lock is acquired global_flag must be false.  The acquisition of
448->f_lock will prevent any call to begin_global() from returning, which
449means that it is safe to release global_lock and invoke do_something().
450
451For this to work, only those foo structures in foo_array[] may be passed
452to do_something_locked().  The reason for this is that the synchronization
453with begin_global() relies on momentarily holding the lock of each and
454every foo structure.
455
456The smp_load_acquire() and smp_store_release() are required because
457changes to a foo structure between calls to begin_global() and
458end_global() are carried out without holding that structure's ->f_lock.
459The smp_load_acquire() and smp_store_release() ensure that the next
460invocation of do_something() from do_something_locked() will see those
461changes.
462
463
464Lockless Reads and Writes
465-------------------------
466
467For another example, suppose a shared variable "foo" is both read and
468updated locklessly.  The code might look as follows:
469
470	int foo;
471
472	int update_foo(int newval)
473	{
474		int ret;
475
476		ret = xchg(&foo, newval);
477		do_something(newval);
478		return ret;
479	}
480
481	int read_foo(void)
482	{
483		do_something_else();
484		return READ_ONCE(foo);
485	}
486
487Because foo is accessed locklessly, all accesses are marked.  It does
488not make sense to use ASSERT_EXCLUSIVE_WRITER() in this case because
489there really can be concurrent lockless writers.  KCSAN would
490flag any concurrent plain C-language reads from foo, and given
491CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n, also any concurrent plain
492C-language writes to foo.
493
494
495Lockless Reads and Writes, But With Single-Threaded Initialization
496------------------------------------------------------------------
497
498For yet another example, suppose that foo is initialized in a
499single-threaded manner, but that a number of kthreads are then created
500that locklessly and concurrently access foo.  Some snippets of this code
501might look as follows:
502
503	int foo;
504
505	void initialize_foo(int initval, int nkthreads)
506	{
507		int i;
508
509		foo = initval;
510		ASSERT_EXCLUSIVE_ACCESS(foo);
511		for (i = 0; i < nkthreads; i++)
512			kthread_run(access_foo_concurrently, ...);
513	}
514
515	/* Called from access_foo_concurrently(). */
516	int update_foo(int newval)
517	{
518		int ret;
519
520		ret = xchg(&foo, newval);
521		do_something(newval);
522		return ret;
523	}
524
525	/* Also called from access_foo_concurrently(). */
526	int read_foo(void)
527	{
528		do_something_else();
529		return READ_ONCE(foo);
530	}
531
532The initialize_foo() uses a plain C-language write to foo because there
533are not supposed to be concurrent accesses during initialization.  The
534ASSERT_EXCLUSIVE_ACCESS() allows KCSAN to flag buggy concurrent unmarked
535reads, and the ASSERT_EXCLUSIVE_ACCESS() call further allows KCSAN to
536flag buggy concurrent writes, even if:  (1) Those writes are marked or
537(2) The kernel was built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=y.
538
539
540Checking Stress-Test Race Coverage
541----------------------------------
542
543When designing stress tests it is important to ensure that race conditions
544of interest really do occur.  For example, consider the following code
545fragment:
546
547	int foo;
548
549	int update_foo(int newval)
550	{
551		return xchg(&foo, newval);
552	}
553
554	int xor_shift_foo(int shift, int mask)
555	{
556		int old, new, newold;
557
558		newold = data_race(foo); /* Checked by cmpxchg(). */
559		do {
560			old = newold;
561			new = (old << shift) ^ mask;
562			newold = cmpxchg(&foo, old, new);
563		} while (newold != old);
564		return old;
565	}
566
567	int read_foo(void)
568	{
569		return READ_ONCE(foo);
570	}
571
572If it is possible for update_foo(), xor_shift_foo(), and read_foo() to be
573invoked concurrently, the stress test should force this concurrency to
574actually happen.  KCSAN can evaluate the stress test when the above code
575is modified to read as follows:
576
577	int foo;
578
579	int update_foo(int newval)
580	{
581		ASSERT_EXCLUSIVE_ACCESS(foo);
582		return xchg(&foo, newval);
583	}
584
585	int xor_shift_foo(int shift, int mask)
586	{
587		int old, new, newold;
588
589		newold = data_race(foo); /* Checked by cmpxchg(). */
590		do {
591			old = newold;
592			new = (old << shift) ^ mask;
593			ASSERT_EXCLUSIVE_ACCESS(foo);
594			newold = cmpxchg(&foo, old, new);
595		} while (newold != old);
596		return old;
597	}
598
599
600	int read_foo(void)
601	{
602		ASSERT_EXCLUSIVE_ACCESS(foo);
603		return READ_ONCE(foo);
604	}
605
606If a given stress-test run does not result in KCSAN complaints from
607each possible pair of ASSERT_EXCLUSIVE_ACCESS() invocations, the
608stress test needs improvement.  If the stress test was to be evaluated
609on a regular basis, it would be wise to place the above instances of
610ASSERT_EXCLUSIVE_ACCESS() under #ifdef so that they did not result in
611false positives when not evaluating the stress test.
612
613
614REFERENCES
615==========
616
617[1] "Concurrency bugs should fear the big bad data-race detector (part 2)"
618    https://lwn.net/Articles/816854/
619
620[2] "The Kernel Concurrency Sanitizer"
621    https://www.linuxfoundation.org/webinars/the-kernel-concurrency-sanitizer
622
623[3] "Who's afraid of a big bad optimizing compiler?"
624    https://lwn.net/Articles/793253/

  1MARKING SHARED-MEMORY ACCESSES
  2==============================
  3
  4This document provides guidelines for marking intentionally concurrent
  5normal accesses to shared memory, that is "normal" as in accesses that do
  6not use read-modify-write atomic operations.  It also describes how to
  7document these accesses, both with comments and with special assertions
  8processed by the Kernel Concurrency Sanitizer (KCSAN).  This discussion
  9builds on an earlier LWN article [1].
 
 10
 11
 12ACCESS-MARKING OPTIONS
 13======================
 14
 15The Linux kernel provides the following access-marking options:
 16
 171.	Plain C-language accesses (unmarked), for example, "a = b;"
 18
 192.	Data-race marking, for example, "data_race(a = b);"
 20
 213.	READ_ONCE(), for example, "a = READ_ONCE(b);"
 22	The various forms of atomic_read() also fit in here.
 23
 244.	WRITE_ONCE(), for example, "WRITE_ONCE(a, b);"
 25	The various forms of atomic_set() also fit in here.
 26
 
 
 
 
 
 27
 28These may be used in combination, as shown in this admittedly improbable
 29example:
 30
 31	WRITE_ONCE(a, b + data_race(c + d) + READ_ONCE(e));
 32
 33Neither plain C-language accesses nor data_race() (#1 and #2 above) place
 34any sort of constraint on the compiler's choice of optimizations [2].
 35In contrast, READ_ONCE() and WRITE_ONCE() (#3 and #4 above) restrict the
 36compiler's use of code-motion and common-subexpression optimizations.
 37Therefore, if a given access is involved in an intentional data race,
 38using READ_ONCE() for loads and WRITE_ONCE() for stores is usually
 39preferable to data_race(), which in turn is usually preferable to plain
 40C-language accesses.  It is permissible to combine #2 and #3, for example,
 41data_race(READ_ONCE(a)), which will both restrict compiler optimizations
 42and disable KCSAN diagnostics.
 43
 44KCSAN will complain about many types of data races involving plain
 45C-language accesses, but marking all accesses involved in a given data
 46race with one of data_race(), READ_ONCE(), or WRITE_ONCE(), will prevent
 47KCSAN from complaining.  Of course, lack of KCSAN complaints does not
 48imply correct code.  Therefore, please take a thoughtful approach
 49when responding to KCSAN complaints.  Churning the code base with
 50ill-considered additions of data_race(), READ_ONCE(), and WRITE_ONCE()
 51is unhelpful.
 52
 53In fact, the following sections describe situations where use of
 54data_race() and even plain C-language accesses is preferable to
 55READ_ONCE() and WRITE_ONCE().
 56
 57
 58Use of the data_race() Macro
 59----------------------------
 60
 61Here are some situations where data_race() should be used instead of
 62READ_ONCE() and WRITE_ONCE():
 63
 641.	Data-racy loads from shared variables whose values are used only
 65	for diagnostic purposes.
 66
 672.	Data-racy reads whose values are checked against marked reload.
 68
 693.	Reads whose values feed into error-tolerant heuristics.
 70
 714.	Writes setting values that feed into error-tolerant heuristics.
 72
 73
 74Data-Racy Reads for Approximate Diagnostics
 75
 76Approximate diagnostics include lockdep reports, monitoring/statistics
 77(including /proc and /sys output), WARN*()/BUG*() checks whose return
 78values are ignored, and other situations where reads from shared variables
 79are not an integral part of the core concurrency design.
 80
 81In fact, use of data_race() instead READ_ONCE() for these diagnostic
 82reads can enable better checking of the remaining accesses implementing
 83the core concurrency design.  For example, suppose that the core design
 84prevents any non-diagnostic reads from shared variable x from running
 85concurrently with updates to x.  Then using plain C-language writes
 86to x allows KCSAN to detect reads from x from within regions of code
 87that fail to exclude the updates.  In this case, it is important to use
 88data_race() for the diagnostic reads because otherwise KCSAN would give
 89false-positive warnings about these diagnostic reads.
 90
 91If it is necessary to both restrict compiler optimizations and disable
 92KCSAN diagnostics, use both data_race() and READ_ONCE(), for example,
 93data_race(READ_ONCE(a)).
 94
 95In theory, plain C-language loads can also be used for this use case.
 96However, in practice this will have the disadvantage of causing KCSAN
 97to generate false positives because KCSAN will have no way of knowing
 98that the resulting data race was intentional.
 99
100
101Data-Racy Reads That Are Checked Against Marked Reload
102
103The values from some reads are not implicitly trusted.  They are instead
104fed into some operation that checks the full value against a later marked
105load from memory, which means that the occasional arbitrarily bogus value
106is not a problem.  For example, if a bogus value is fed into cmpxchg(),
107all that happens is that this cmpxchg() fails, which normally results
108in a retry.  Unless the race condition that resulted in the bogus value
109recurs, this retry will with high probability succeed, so no harm done.
110
111However, please keep in mind that a data_race() load feeding into
112a cmpxchg_relaxed() might still be subject to load fusing on some
113architectures.  Therefore, it is best to capture the return value from
114the failing cmpxchg() for the next iteration of the loop, an approach
115that provides the compiler much less scope for mischievous optimizations.
116Capturing the return value from cmpxchg() also saves a memory reference
117in many cases.
118
119In theory, plain C-language loads can also be used for this use case.
120However, in practice this will have the disadvantage of causing KCSAN
121to generate false positives because KCSAN will have no way of knowing
122that the resulting data race was intentional.
123
124
125Reads Feeding Into Error-Tolerant Heuristics
126
127Values from some reads feed into heuristics that can tolerate occasional
128errors.  Such reads can use data_race(), thus allowing KCSAN to focus on
129the other accesses to the relevant shared variables.  But please note
130that data_race() loads are subject to load fusing, which can result in
131consistent errors, which in turn are quite capable of breaking heuristics.
132Therefore use of data_race() should be limited to cases where some other
133code (such as a barrier() call) will force the occasional reload.
134
135Note that this use case requires that the heuristic be able to handle
136any possible error.  In contrast, if the heuristics might be fatally
137confused by one or more of the possible erroneous values, use READ_ONCE()
138instead of data_race().
139
140In theory, plain C-language loads can also be used for this use case.
141However, in practice this will have the disadvantage of causing KCSAN
142to generate false positives because KCSAN will have no way of knowing
143that the resulting data race was intentional.
144
145
146Writes Setting Values Feeding Into Error-Tolerant Heuristics
147
148The values read into error-tolerant heuristics come from somewhere,
149for example, from sysfs.  This means that some code in sysfs writes
150to this same variable, and these writes can also use data_race().
151After all, if the heuristic can tolerate the occasional bogus value
152due to compiler-mangled reads, it can also tolerate the occasional
153compiler-mangled write, at least assuming that the proper value is in
154place once the write completes.
155
156Plain C-language stores can also be used for this use case.  However,
157in kernels built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n, this
158will have the disadvantage of causing KCSAN to generate false positives
159because KCSAN will have no way of knowing that the resulting data race
160was intentional.
161
162
163Use of Plain C-Language Accesses
164--------------------------------
165
166Here are some example situations where plain C-language accesses should
167used instead of READ_ONCE(), WRITE_ONCE(), and data_race():
168
1691.	Accesses protected by mutual exclusion, including strict locking
170	and sequence locking.
171
1722.	Initialization-time and cleanup-time accesses.	This covers a
173	wide variety of situations, including the uniprocessor phase of
174	system boot, variables to be used by not-yet-spawned kthreads,
175	structures not yet published to reference-counted or RCU-protected
176	data structures, and the cleanup side of any of these situations.
177
1783.	Per-CPU variables that are not accessed from other CPUs.
179
1804.	Private per-task variables, including on-stack variables, some
181	fields in the task_struct structure, and task-private heap data.
182
1835.	Any other loads for which there is not supposed to be a concurrent
184	store to that same variable.
185
1866.	Any other stores for which there should be neither concurrent
187	loads nor concurrent stores to that same variable.
188
189	But note that KCSAN makes two explicit exceptions to this rule
190	by default, refraining from flagging plain C-language stores:
191
192	a.	No matter what.  You can override this default by building
193		with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n.
194
195	b.	When the store writes the value already contained in
196		that variable.	You can override this default by building
197		with CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n.
198
199	c.	When one of the stores is in an interrupt handler and
200		the other in the interrupted code.  You can override this
201		default by building with CONFIG_KCSAN_INTERRUPT_WATCHER=y.
202
203Note that it is important to use plain C-language accesses in these cases,
204because doing otherwise prevents KCSAN from detecting violations of your
205code's synchronization rules.
206
207
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
208ACCESS-DOCUMENTATION OPTIONS
209============================
210
211It is important to comment marked accesses so that people reading your
212code, yourself included, are reminded of the synchronization design.
213However, it is even more important to comment plain C-language accesses
214that are intentionally involved in data races.  Such comments are
215needed to remind people reading your code, again, yourself included,
216of how the compiler has been prevented from optimizing those accesses
217into concurrency bugs.
218
219It is also possible to tell KCSAN about your synchronization design.
220For example, ASSERT_EXCLUSIVE_ACCESS(foo) tells KCSAN that any
221concurrent access to variable foo by any other CPU is an error, even
222if that concurrent access is marked with READ_ONCE().  In addition,
223ASSERT_EXCLUSIVE_WRITER(foo) tells KCSAN that although it is OK for there
224to be concurrent reads from foo from other CPUs, it is an error for some
225other CPU to be concurrently writing to foo, even if that concurrent
226write is marked with data_race() or WRITE_ONCE().
227
228Note that although KCSAN will call out data races involving either
229ASSERT_EXCLUSIVE_ACCESS() or ASSERT_EXCLUSIVE_WRITER() on the one hand
230and data_race() writes on the other, KCSAN will not report the location
231of these data_race() writes.
232
233
234EXAMPLES
235========
236
237As noted earlier, the goal is to prevent the compiler from destroying
238your concurrent algorithm, to help the human reader, and to inform
239KCSAN of aspects of your concurrency design.  This section looks at a
240few examples showing how this can be done.
241
242
243Lock Protection With Lockless Diagnostic Access
244-----------------------------------------------
245
246For example, suppose a shared variable "foo" is read only while a
247reader-writer spinlock is read-held, written only while that same
248spinlock is write-held, except that it is also read locklessly for
249diagnostic purposes.  The code might look as follows:
250
251	int foo;
252	DEFINE_RWLOCK(foo_rwlock);
253
254	void update_foo(int newval)
255	{
256		write_lock(&foo_rwlock);
257		foo = newval;
258		do_something(newval);
259		write_unlock(&foo_rwlock);
260	}
261
262	int read_foo(void)
263	{
264		int ret;
265
266		read_lock(&foo_rwlock);
267		do_something_else();
268		ret = foo;
269		read_unlock(&foo_rwlock);
270		return ret;
271	}
272
273	void read_foo_diagnostic(void)
274	{
275		pr_info("Current value of foo: %d\n", data_race(foo));
276	}
277
278The reader-writer lock prevents the compiler from introducing concurrency
279bugs into any part of the main algorithm using foo, which means that
280the accesses to foo within both update_foo() and read_foo() can (and
281should) be plain C-language accesses.  One benefit of making them be
282plain C-language accesses is that KCSAN can detect any erroneous lockless
283reads from or updates to foo.  The data_race() in read_foo_diagnostic()
284tells KCSAN that data races are expected, and should be silently
285ignored.  This data_race() also tells the human reading the code that
286read_foo_diagnostic() might sometimes return a bogus value.
287
288If it is necessary to suppress compiler optimization and also detect
289buggy lockless writes, read_foo_diagnostic() can be updated as follows:
290
291	void read_foo_diagnostic(void)
292	{
293		pr_info("Current value of foo: %d\n", data_race(READ_ONCE(foo)));
294	}
295
296Alternatively, given that KCSAN is to ignore all accesses in this function,
297this function can be marked __no_kcsan and the data_race() can be dropped:
298
299	void __no_kcsan read_foo_diagnostic(void)
300	{
301		pr_info("Current value of foo: %d\n", READ_ONCE(foo));
302	}
303
304However, in order for KCSAN to detect buggy lockless writes, your kernel
305must be built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n.  If you
306need KCSAN to detect such a write even if that write did not change
307the value of foo, you also need CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n.
308If you need KCSAN to detect such a write happening in an interrupt handler
309running on the same CPU doing the legitimate lock-protected write, you
310also need CONFIG_KCSAN_INTERRUPT_WATCHER=y.  With some or all of these
311Kconfig options set properly, KCSAN can be quite helpful, although
312it is not necessarily a full replacement for hardware watchpoints.
313On the other hand, neither are hardware watchpoints a full replacement
314for KCSAN because it is not always easy to tell hardware watchpoint to
315conditionally trap on accesses.
316
317
318Lock-Protected Writes With Lockless Reads
319-----------------------------------------
320
321For another example, suppose a shared variable "foo" is updated only
322while holding a spinlock, but is read locklessly.  The code might look
323as follows:
324
325	int foo;
326	DEFINE_SPINLOCK(foo_lock);
327
328	void update_foo(int newval)
329	{
330		spin_lock(&foo_lock);
331		WRITE_ONCE(foo, newval);
332		ASSERT_EXCLUSIVE_WRITER(foo);
333		do_something(newval);
334		spin_unlock(&foo_wlock);
335	}
336
337	int read_foo(void)
338	{
339		do_something_else();
340		return READ_ONCE(foo);
341	}
342
343Because foo is read locklessly, all accesses are marked.  The purpose
344of the ASSERT_EXCLUSIVE_WRITER() is to allow KCSAN to check for a buggy
345concurrent lockless write.
346
347
348Lock-Protected Writes With Heuristic Lockless Reads
349---------------------------------------------------
350
351For another example, suppose that the code can normally make use of
352a per-data-structure lock, but there are times when a global lock
353is required.  These times are indicated via a global flag.  The code
354might look as follows, and is based loosely on nf_conntrack_lock(),
355nf_conntrack_all_lock(), and nf_conntrack_all_unlock():
356
357	bool global_flag;
358	DEFINE_SPINLOCK(global_lock);
359	struct foo {
360		spinlock_t f_lock;
361		int f_data;
362	};
363
364	/* All foo structures are in the following array. */
365	int nfoo;
366	struct foo *foo_array;
367
368	void do_something_locked(struct foo *fp)
369	{
370		/* This works even if data_race() returns nonsense. */
371		if (!data_race(global_flag)) {
372			spin_lock(&fp->f_lock);
373			if (!smp_load_acquire(&global_flag)) {
374				do_something(fp);
375				spin_unlock(&fp->f_lock);
376				return;
377			}
378			spin_unlock(&fp->f_lock);
379		}
380		spin_lock(&global_lock);
381		/* global_lock held, thus global flag cannot be set. */
382		spin_lock(&fp->f_lock);
383		spin_unlock(&global_lock);
384		/*
385		 * global_flag might be set here, but begin_global()
386		 * will wait for ->f_lock to be released.
387		 */
388		do_something(fp);
389		spin_unlock(&fp->f_lock);
390	}
391
392	void begin_global(void)
393	{
394		int i;
395
396		spin_lock(&global_lock);
397		WRITE_ONCE(global_flag, true);
398		for (i = 0; i < nfoo; i++) {
399			/*
400			 * Wait for pre-existing local locks.  One at
401			 * a time to avoid lockdep limitations.
402			 */
403			spin_lock(&fp->f_lock);
404			spin_unlock(&fp->f_lock);
405		}
406	}
407
408	void end_global(void)
409	{
410		smp_store_release(&global_flag, false);
411		spin_unlock(&global_lock);
412	}
413
414All code paths leading from the do_something_locked() function's first
415read from global_flag acquire a lock, so endless load fusing cannot
416happen.
417
418If the value read from global_flag is true, then global_flag is
419rechecked while holding ->f_lock, which, if global_flag is now false,
420prevents begin_global() from completing.  It is therefore safe to invoke
421do_something().
422
423Otherwise, if either value read from global_flag is true, then after
424global_lock is acquired global_flag must be false.  The acquisition of
425->f_lock will prevent any call to begin_global() from returning, which
426means that it is safe to release global_lock and invoke do_something().
427
428For this to work, only those foo structures in foo_array[] may be passed
429to do_something_locked().  The reason for this is that the synchronization
430with begin_global() relies on momentarily holding the lock of each and
431every foo structure.
432
433The smp_load_acquire() and smp_store_release() are required because
434changes to a foo structure between calls to begin_global() and
435end_global() are carried out without holding that structure's ->f_lock.
436The smp_load_acquire() and smp_store_release() ensure that the next
437invocation of do_something() from do_something_locked() will see those
438changes.
439
440
441Lockless Reads and Writes
442-------------------------
443
444For another example, suppose a shared variable "foo" is both read and
445updated locklessly.  The code might look as follows:
446
447	int foo;
448
449	int update_foo(int newval)
450	{
451		int ret;
452
453		ret = xchg(&foo, newval);
454		do_something(newval);
455		return ret;
456	}
457
458	int read_foo(void)
459	{
460		do_something_else();
461		return READ_ONCE(foo);
462	}
463
464Because foo is accessed locklessly, all accesses are marked.  It does
465not make sense to use ASSERT_EXCLUSIVE_WRITER() in this case because
466there really can be concurrent lockless writers.  KCSAN would
467flag any concurrent plain C-language reads from foo, and given
468CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n, also any concurrent plain
469C-language writes to foo.
470
471
472Lockless Reads and Writes, But With Single-Threaded Initialization
473------------------------------------------------------------------
474
475For yet another example, suppose that foo is initialized in a
476single-threaded manner, but that a number of kthreads are then created
477that locklessly and concurrently access foo.  Some snippets of this code
478might look as follows:
479
480	int foo;
481
482	void initialize_foo(int initval, int nkthreads)
483	{
484		int i;
485
486		foo = initval;
487		ASSERT_EXCLUSIVE_ACCESS(foo);
488		for (i = 0; i < nkthreads; i++)
489			kthread_run(access_foo_concurrently, ...);
490	}
491
492	/* Called from access_foo_concurrently(). */
493	int update_foo(int newval)
494	{
495		int ret;
496
497		ret = xchg(&foo, newval);
498		do_something(newval);
499		return ret;
500	}
501
502	/* Also called from access_foo_concurrently(). */
503	int read_foo(void)
504	{
505		do_something_else();
506		return READ_ONCE(foo);
507	}
508
509The initialize_foo() uses a plain C-language write to foo because there
510are not supposed to be concurrent accesses during initialization.  The
511ASSERT_EXCLUSIVE_ACCESS() allows KCSAN to flag buggy concurrent unmarked
512reads, and the ASSERT_EXCLUSIVE_ACCESS() call further allows KCSAN to
513flag buggy concurrent writes, even if:  (1) Those writes are marked or
514(2) The kernel was built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=y.
515
516
517Checking Stress-Test Race Coverage
518----------------------------------
519
520When designing stress tests it is important to ensure that race conditions
521of interest really do occur.  For example, consider the following code
522fragment:
523
524	int foo;
525
526	int update_foo(int newval)
527	{
528		return xchg(&foo, newval);
529	}
530
531	int xor_shift_foo(int shift, int mask)
532	{
533		int old, new, newold;
534
535		newold = data_race(foo); /* Checked by cmpxchg(). */
536		do {
537			old = newold;
538			new = (old << shift) ^ mask;
539			newold = cmpxchg(&foo, old, new);
540		} while (newold != old);
541		return old;
542	}
543
544	int read_foo(void)
545	{
546		return READ_ONCE(foo);
547	}
548
549If it is possible for update_foo(), xor_shift_foo(), and read_foo() to be
550invoked concurrently, the stress test should force this concurrency to
551actually happen.  KCSAN can evaluate the stress test when the above code
552is modified to read as follows:
553
554	int foo;
555
556	int update_foo(int newval)
557	{
558		ASSERT_EXCLUSIVE_ACCESS(foo);
559		return xchg(&foo, newval);
560	}
561
562	int xor_shift_foo(int shift, int mask)
563	{
564		int old, new, newold;
565
566		newold = data_race(foo); /* Checked by cmpxchg(). */
567		do {
568			old = newold;
569			new = (old << shift) ^ mask;
570			ASSERT_EXCLUSIVE_ACCESS(foo);
571			newold = cmpxchg(&foo, old, new);
572		} while (newold != old);
573		return old;
574	}
575
576
577	int read_foo(void)
578	{
579		ASSERT_EXCLUSIVE_ACCESS(foo);
580		return READ_ONCE(foo);
581	}
582
583If a given stress-test run does not result in KCSAN complaints from
584each possible pair of ASSERT_EXCLUSIVE_ACCESS() invocations, the
585stress test needs improvement.  If the stress test was to be evaluated
586on a regular basis, it would be wise to place the above instances of
587ASSERT_EXCLUSIVE_ACCESS() under #ifdef so that they did not result in
588false positives when not evaluating the stress test.
589
590
591REFERENCES
592==========
593
594[1] "Concurrency bugs should fear the big bad data-race detector (part 2)"
595    https://lwn.net/Articles/816854/
596
597[2] "Who's afraid of a big bad optimizing compiler?"
 
 
 
598    https://lwn.net/Articles/793253/