1INFINIBAND MIDLAYER LOCKING
  2
  3  This guide is an attempt to make explicit the locking assumptions
  4  made by the InfiniBand midlayer.  It describes the requirements on
  5  both low-level drivers that sit below the midlayer and upper level
  6  protocols that use the midlayer.
  7
  8Sleeping and interrupt context
  9
 10  With the following exceptions, a low-level driver implementation of
 11  all of the methods in struct ib_device may sleep.  The exceptions
 12  are any methods from the list:
 13
 14    create_ah
 15    modify_ah
 16    query_ah
 17    destroy_ah
 18    post_send
 19    post_recv
 20    poll_cq
 21    req_notify_cq
 22    map_phys_fmr
 23
 24  which may not sleep and must be callable from any context.
 25
 26  The corresponding functions exported to upper level protocol
 27  consumers:
 28
 29    ib_create_ah
 30    ib_modify_ah
 31    ib_query_ah
 32    ib_destroy_ah
 33    ib_post_send
 34    ib_post_recv
 35    ib_req_notify_cq
 36    ib_map_phys_fmr
 37
 38  are therefore safe to call from any context.
 39
 40  In addition, the function
 41
 42    ib_dispatch_event
 43
 44  used by low-level drivers to dispatch asynchronous events through
 45  the midlayer is also safe to call from any context.
 46
 47Reentrancy
 48
 49  All of the methods in struct ib_device exported by a low-level
 50  driver must be fully reentrant.  The low-level driver is required to
 51  perform all synchronization necessary to maintain consistency, even
 52  if multiple function calls using the same object are run
 53  simultaneously.
 54
 55  The IB midlayer does not perform any serialization of function calls.
 56
 57  Because low-level drivers are reentrant, upper level protocol
 58  consumers are not required to perform any serialization.  However,
 59  some serialization may be required to get sensible results.  For
 60  example, a consumer may safely call ib_poll_cq() on multiple CPUs
 61  simultaneously.  However, the ordering of the work completion
 62  information between different calls of ib_poll_cq() is not defined.
 63
 64Callbacks
 65
 66  A low-level driver must not perform a callback directly from the
 67  same callchain as an ib_device method call.  For example, it is not
 68  allowed for a low-level driver to call a consumer's completion event
 69  handler directly from its post_send method.  Instead, the low-level
 70  driver should defer this callback by, for example, scheduling a
 71  tasklet to perform the callback.
 72
 73  The low-level driver is responsible for ensuring that multiple
 74  completion event handlers for the same CQ are not called
 75  simultaneously.  The driver must guarantee that only one CQ event
 76  handler for a given CQ is running at a time.  In other words, the
 77  following situation is not allowed:
 78
 79        CPU1                                    CPU2
 80
 81  low-level driver ->
 82    consumer CQ event callback:
 83      /* ... */
 84      ib_req_notify_cq(cq, ...);
 85                                        low-level driver ->
 86      /* ... */                           consumer CQ event callback:
 87                                            /* ... */
 88      return from CQ event handler
 89
 90  The context in which completion event and asynchronous event
 91  callbacks run is not defined.  Depending on the low-level driver, it
 92  may be process context, softirq context, or interrupt context.
 93  Upper level protocol consumers may not sleep in a callback.
 94
 95Hot-plug
 96
 97  A low-level driver announces that a device is ready for use by
 98  consumers when it calls ib_register_device(), all initialization
 99  must be complete before this call.  The device must remain usable
100  until the driver's call to ib_unregister_device() has returned.
101
102  A low-level driver must call ib_register_device() and
103  ib_unregister_device() from process context.  It must not hold any
104  semaphores that could cause deadlock if a consumer calls back into
105  the driver across these calls.
106
107  An upper level protocol consumer may begin using an IB device as
108  soon as the add method of its struct ib_client is called for that
109  device.  A consumer must finish all cleanup and free all resources
110  relating to a device before returning from the remove method.
111
112  A consumer is permitted to sleep in its add and remove methods.