Loading...
1===========================
2InfiniBand Midlayer Locking
3===========================
4
5 This guide is an attempt to make explicit the locking assumptions
6 made by the InfiniBand midlayer. It describes the requirements on
7 both low-level drivers that sit below the midlayer and upper level
8 protocols that use the midlayer.
9
10Sleeping and interrupt context
11==============================
12
13 With the following exceptions, a low-level driver implementation of
14 all of the methods in struct ib_device may sleep. The exceptions
15 are any methods from the list:
16
17 - create_ah
18 - modify_ah
19 - query_ah
20 - destroy_ah
21 - post_send
22 - post_recv
23 - poll_cq
24 - req_notify_cq
25
26 which may not sleep and must be callable from any context.
27
28 The corresponding functions exported to upper level protocol
29 consumers:
30
31 - rdma_create_ah
32 - rdma_modify_ah
33 - rdma_query_ah
34 - rdma_destroy_ah
35 - ib_post_send
36 - ib_post_recv
37 - ib_req_notify_cq
38
39 are therefore safe to call from any context.
40
41 In addition, the function
42
43 - ib_dispatch_event
44
45 used by low-level drivers to dispatch asynchronous events through
46 the midlayer is also safe to call from any context.
47
48Reentrancy
49----------
50
51 All of the methods in struct ib_device exported by a low-level
52 driver must be fully reentrant. The low-level driver is required to
53 perform all synchronization necessary to maintain consistency, even
54 if multiple function calls using the same object are run
55 simultaneously.
56
57 The IB midlayer does not perform any serialization of function calls.
58
59 Because low-level drivers are reentrant, upper level protocol
60 consumers are not required to perform any serialization. However,
61 some serialization may be required to get sensible results. For
62 example, a consumer may safely call ib_poll_cq() on multiple CPUs
63 simultaneously. However, the ordering of the work completion
64 information between different calls of ib_poll_cq() is not defined.
65
66Callbacks
67---------
68
69 A low-level driver must not perform a callback directly from the
70 same callchain as an ib_device method call. For example, it is not
71 allowed for a low-level driver to call a consumer's completion event
72 handler directly from its post_send method. Instead, the low-level
73 driver should defer this callback by, for example, scheduling a
74 tasklet to perform the callback.
75
76 The low-level driver is responsible for ensuring that multiple
77 completion event handlers for the same CQ are not called
78 simultaneously. The driver must guarantee that only one CQ event
79 handler for a given CQ is running at a time. In other words, the
80 following situation is not allowed::
81
82 CPU1 CPU2
83
84 low-level driver ->
85 consumer CQ event callback:
86 /* ... */
87 ib_req_notify_cq(cq, ...);
88 low-level driver ->
89 /* ... */ consumer CQ event callback:
90 /* ... */
91 return from CQ event handler
92
93 The context in which completion event and asynchronous event
94 callbacks run is not defined. Depending on the low-level driver, it
95 may be process context, softirq context, or interrupt context.
96 Upper level protocol consumers may not sleep in a callback.
97
98Hot-plug
99--------
100
101 A low-level driver announces that a device is ready for use by
102 consumers when it calls ib_register_device(), all initialization
103 must be complete before this call. The device must remain usable
104 until the driver's call to ib_unregister_device() has returned.
105
106 A low-level driver must call ib_register_device() and
107 ib_unregister_device() from process context. It must not hold any
108 semaphores that could cause deadlock if a consumer calls back into
109 the driver across these calls.
110
111 An upper level protocol consumer may begin using an IB device as
112 soon as the add method of its struct ib_client is called for that
113 device. A consumer must finish all cleanup and free all resources
114 relating to a device before returning from the remove method.
115
116 A consumer is permitted to sleep in its add and remove methods.
1===========================
2InfiniBand Midlayer Locking
3===========================
4
5 This guide is an attempt to make explicit the locking assumptions
6 made by the InfiniBand midlayer. It describes the requirements on
7 both low-level drivers that sit below the midlayer and upper level
8 protocols that use the midlayer.
9
10Sleeping and interrupt context
11==============================
12
13 With the following exceptions, a low-level driver implementation of
14 all of the methods in struct ib_device may sleep. The exceptions
15 are any methods from the list:
16
17 - create_ah
18 - modify_ah
19 - query_ah
20 - destroy_ah
21 - post_send
22 - post_recv
23 - poll_cq
24 - req_notify_cq
25 - map_phys_fmr
26
27 which may not sleep and must be callable from any context.
28
29 The corresponding functions exported to upper level protocol
30 consumers:
31
32 - rdma_create_ah
33 - rdma_modify_ah
34 - rdma_query_ah
35 - rdma_destroy_ah
36 - ib_post_send
37 - ib_post_recv
38 - ib_req_notify_cq
39 - ib_map_phys_fmr
40
41 are therefore safe to call from any context.
42
43 In addition, the function
44
45 - ib_dispatch_event
46
47 used by low-level drivers to dispatch asynchronous events through
48 the midlayer is also safe to call from any context.
49
50Reentrancy
51----------
52
53 All of the methods in struct ib_device exported by a low-level
54 driver must be fully reentrant. The low-level driver is required to
55 perform all synchronization necessary to maintain consistency, even
56 if multiple function calls using the same object are run
57 simultaneously.
58
59 The IB midlayer does not perform any serialization of function calls.
60
61 Because low-level drivers are reentrant, upper level protocol
62 consumers are not required to perform any serialization. However,
63 some serialization may be required to get sensible results. For
64 example, a consumer may safely call ib_poll_cq() on multiple CPUs
65 simultaneously. However, the ordering of the work completion
66 information between different calls of ib_poll_cq() is not defined.
67
68Callbacks
69---------
70
71 A low-level driver must not perform a callback directly from the
72 same callchain as an ib_device method call. For example, it is not
73 allowed for a low-level driver to call a consumer's completion event
74 handler directly from its post_send method. Instead, the low-level
75 driver should defer this callback by, for example, scheduling a
76 tasklet to perform the callback.
77
78 The low-level driver is responsible for ensuring that multiple
79 completion event handlers for the same CQ are not called
80 simultaneously. The driver must guarantee that only one CQ event
81 handler for a given CQ is running at a time. In other words, the
82 following situation is not allowed::
83
84 CPU1 CPU2
85
86 low-level driver ->
87 consumer CQ event callback:
88 /* ... */
89 ib_req_notify_cq(cq, ...);
90 low-level driver ->
91 /* ... */ consumer CQ event callback:
92 /* ... */
93 return from CQ event handler
94
95 The context in which completion event and asynchronous event
96 callbacks run is not defined. Depending on the low-level driver, it
97 may be process context, softirq context, or interrupt context.
98 Upper level protocol consumers may not sleep in a callback.
99
100Hot-plug
101--------
102
103 A low-level driver announces that a device is ready for use by
104 consumers when it calls ib_register_device(), all initialization
105 must be complete before this call. The device must remain usable
106 until the driver's call to ib_unregister_device() has returned.
107
108 A low-level driver must call ib_register_device() and
109 ib_unregister_device() from process context. It must not hold any
110 semaphores that could cause deadlock if a consumer calls back into
111 the driver across these calls.
112
113 An upper level protocol consumer may begin using an IB device as
114 soon as the add method of its struct ib_client is called for that
115 device. A consumer must finish all cleanup and free all resources
116 relating to a device before returning from the remove method.
117
118 A consumer is permitted to sleep in its add and remove methods.