1==================================
  2Kernel Lock Torture Test Operation
  3==================================
  4
  5CONFIG_LOCK_TORTURE_TEST
  6========================
  7
  8The CONFIG_LOCK_TORTURE_TEST config option provides a kernel module
  9that runs torture tests on core kernel locking primitives. The kernel
 10module, 'locktorture', may be built after the fact on the running
 11kernel to be tested, if desired. The tests periodically output status
 12messages via printk(), which can be examined via the dmesg (perhaps
 13grepping for "torture").  The test is started when the module is loaded,
 14and stops when the module is unloaded. This program is based on how RCU
 15is tortured, via rcutorture.
 16
 17This torture test consists of creating a number of kernel threads which
 18acquire the lock and hold it for specific amount of time, thus simulating
 19different critical region behaviors. The amount of contention on the lock
 20can be simulated by either enlarging this critical region hold time and/or
 21creating more kthreads.
 22
 23
 24Module Parameters
 25=================
 26
 27This module has the following parameters:
 28
 29
 30Locktorture-specific
 31--------------------
 32
 33nwriters_stress
 34		  Number of kernel threads that will stress exclusive lock
 35		  ownership (writers). The default value is twice the number
 36		  of online CPUs.
 37
 38nreaders_stress
 39		  Number of kernel threads that will stress shared lock
 40		  ownership (readers). The default is the same amount of writer
 41		  locks. If the user did not specify nwriters_stress, then
 42		  both readers and writers be the amount of online CPUs.
 43
 44torture_type
 45		  Type of lock to torture. By default, only spinlocks will
 46		  be tortured. This module can torture the following locks,
 47		  with string values as follows:
 48
 49		     - "lock_busted":
 50				Simulates a buggy lock implementation.
 51
 52		     - "spin_lock":
 53				spin_lock() and spin_unlock() pairs.
 54
 55		     - "spin_lock_irq":
 56				spin_lock_irq() and spin_unlock_irq() pairs.
 57
 58		     - "rw_lock":
 59				read/write lock() and unlock() rwlock pairs.
 60
 61		     - "rw_lock_irq":
 62				read/write lock_irq() and unlock_irq()
 63				rwlock pairs.
 64
 65		     - "mutex_lock":
 66				mutex_lock() and mutex_unlock() pairs.
 67
 68		     - "rtmutex_lock":
 69				rtmutex_lock() and rtmutex_unlock() pairs.
 70				Kernel must have CONFIG_RT_MUTEXES=y.
 71
 72		     - "rwsem_lock":
 73				read/write down() and up() semaphore pairs.
 74
 75
 76Torture-framework (RCU + locking)
 77---------------------------------
 78
 79shutdown_secs
 80		  The number of seconds to run the test before terminating
 81		  the test and powering off the system.  The default is
 82		  zero, which disables test termination and system shutdown.
 83		  This capability is useful for automated testing.
 84
 85onoff_interval
 86		  The number of seconds between each attempt to execute a
 87		  randomly selected CPU-hotplug operation.  Defaults
 88		  to zero, which disables CPU hotplugging.  In
 89		  CONFIG_HOTPLUG_CPU=n kernels, locktorture will silently
 90		  refuse to do any CPU-hotplug operations regardless of
 91		  what value is specified for onoff_interval.
 92
 93onoff_holdoff
 94		  The number of seconds to wait until starting CPU-hotplug
 95		  operations.  This would normally only be used when
 96		  locktorture was built into the kernel and started
 97		  automatically at boot time, in which case it is useful
 98		  in order to avoid confusing boot-time code with CPUs
 99		  coming and going. This parameter is only useful if
100		  CONFIG_HOTPLUG_CPU is enabled.
101
102stat_interval
103		  Number of seconds between statistics-related printk()s.
104		  By default, locktorture will report stats every 60 seconds.
105		  Setting the interval to zero causes the statistics to
106		  be printed -only- when the module is unloaded.
107
108stutter
109		  The length of time to run the test before pausing for this
110		  same period of time.  Defaults to "stutter=5", so as
111		  to run and pause for (roughly) five-second intervals.
112		  Specifying "stutter=0" causes the test to run continuously
113		  without pausing.
114
115shuffle_interval
116		  The number of seconds to keep the test threads affinitized
117		  to a particular subset of the CPUs, defaults to 3 seconds.
118		  Used in conjunction with test_no_idle_hz.
119
120verbose
121		  Enable verbose debugging printing, via printk(). Enabled
122		  by default. This extra information is mostly related to
123		  high-level errors and reports from the main 'torture'
124		  framework.
125
126
127Statistics
128==========
129
130Statistics are printed in the following format::
131
132  spin_lock-torture: Writes:  Total: 93746064  Max/Min: 0/0   Fail: 0
133     (A)		    (B)		   (C)		  (D)	       (E)
134
135  (A): Lock type that is being tortured -- torture_type parameter.
136
137  (B): Number of writer lock acquisitions. If dealing with a read/write
138       primitive a second "Reads" statistics line is printed.
139
140  (C): Number of times the lock was acquired.
141
142  (D): Min and max number of times threads failed to acquire the lock.
143
144  (E): true/false values if there were errors acquiring the lock. This should
145       -only- be positive if there is a bug in the locking primitive's
146       implementation. Otherwise a lock should never fail (i.e., spin_lock()).
147       Of course, the same applies for (C), above. A dummy example of this is
148       the "lock_busted" type.
149
150Usage
151=====
152
153The following script may be used to torture locks::
154
155	#!/bin/sh
156
157	modprobe locktorture
158	sleep 3600
159	rmmod locktorture
160	dmesg | grep torture:
161
162The output can be manually inspected for the error flag of "!!!".
163One could of course create a more elaborate script that automatically
164checked for such errors.  The "rmmod" command forces a "SUCCESS",
165"FAILURE", or "RCU_HOTPLUG" indication to be printk()ed.  The first
166two are self-explanatory, while the last indicates that while there
167were no locking failures, CPU-hotplug problems were detected.
168
169Also see: Documentation/RCU/torture.rst