1 |
3 |
xianfeng |
INFINIBAND MIDLAYER LOCKING
|
2 |
|
|
|
3 |
|
|
This guide is an attempt to make explicit the locking assumptions
|
4 |
|
|
made by the InfiniBand midlayer. It describes the requirements on
|
5 |
|
|
both low-level drivers that sit below the midlayer and upper level
|
6 |
|
|
protocols that use the midlayer.
|
7 |
|
|
|
8 |
|
|
Sleeping and interrupt context
|
9 |
|
|
|
10 |
|
|
With the following exceptions, a low-level driver implementation of
|
11 |
|
|
all of the methods in struct ib_device may sleep. The exceptions
|
12 |
|
|
are any methods from the list:
|
13 |
|
|
|
14 |
|
|
create_ah
|
15 |
|
|
modify_ah
|
16 |
|
|
query_ah
|
17 |
|
|
destroy_ah
|
18 |
|
|
bind_mw
|
19 |
|
|
post_send
|
20 |
|
|
post_recv
|
21 |
|
|
poll_cq
|
22 |
|
|
req_notify_cq
|
23 |
|
|
map_phys_fmr
|
24 |
|
|
|
25 |
|
|
which may not sleep and must be callable from any context.
|
26 |
|
|
|
27 |
|
|
The corresponding functions exported to upper level protocol
|
28 |
|
|
consumers:
|
29 |
|
|
|
30 |
|
|
ib_create_ah
|
31 |
|
|
ib_modify_ah
|
32 |
|
|
ib_query_ah
|
33 |
|
|
ib_destroy_ah
|
34 |
|
|
ib_bind_mw
|
35 |
|
|
ib_post_send
|
36 |
|
|
ib_post_recv
|
37 |
|
|
ib_req_notify_cq
|
38 |
|
|
ib_map_phys_fmr
|
39 |
|
|
|
40 |
|
|
are therefore safe to call from any context.
|
41 |
|
|
|
42 |
|
|
In addition, the function
|
43 |
|
|
|
44 |
|
|
ib_dispatch_event
|
45 |
|
|
|
46 |
|
|
used by low-level drivers to dispatch asynchronous events through
|
47 |
|
|
the midlayer is also safe to call from any context.
|
48 |
|
|
|
49 |
|
|
Reentrancy
|
50 |
|
|
|
51 |
|
|
All of the methods in struct ib_device exported by a low-level
|
52 |
|
|
driver must be fully reentrant. The low-level driver is required to
|
53 |
|
|
perform all synchronization necessary to maintain consistency, even
|
54 |
|
|
if multiple function calls using the same object are run
|
55 |
|
|
simultaneously.
|
56 |
|
|
|
57 |
|
|
The IB midlayer does not perform any serialization of function calls.
|
58 |
|
|
|
59 |
|
|
Because low-level drivers are reentrant, upper level protocol
|
60 |
|
|
consumers are not required to perform any serialization. However,
|
61 |
|
|
some serialization may be required to get sensible results. For
|
62 |
|
|
example, a consumer may safely call ib_poll_cq() on multiple CPUs
|
63 |
|
|
simultaneously. However, the ordering of the work completion
|
64 |
|
|
information between different calls of ib_poll_cq() is not defined.
|
65 |
|
|
|
66 |
|
|
Callbacks
|
67 |
|
|
|
68 |
|
|
A low-level driver must not perform a callback directly from the
|
69 |
|
|
same callchain as an ib_device method call. For example, it is not
|
70 |
|
|
allowed for a low-level driver to call a consumer's completion event
|
71 |
|
|
handler directly from its post_send method. Instead, the low-level
|
72 |
|
|
driver should defer this callback by, for example, scheduling a
|
73 |
|
|
tasklet to perform the callback.
|
74 |
|
|
|
75 |
|
|
The low-level driver is responsible for ensuring that multiple
|
76 |
|
|
completion event handlers for the same CQ are not called
|
77 |
|
|
simultaneously. The driver must guarantee that only one CQ event
|
78 |
|
|
handler for a given CQ is running at a time. In other words, the
|
79 |
|
|
following situation is not allowed:
|
80 |
|
|
|
81 |
|
|
CPU1 CPU2
|
82 |
|
|
|
83 |
|
|
low-level driver ->
|
84 |
|
|
consumer CQ event callback:
|
85 |
|
|
/* ... */
|
86 |
|
|
ib_req_notify_cq(cq, ...);
|
87 |
|
|
low-level driver ->
|
88 |
|
|
/* ... */ consumer CQ event callback:
|
89 |
|
|
/* ... */
|
90 |
|
|
return from CQ event handler
|
91 |
|
|
|
92 |
|
|
The context in which completion event and asynchronous event
|
93 |
|
|
callbacks run is not defined. Depending on the low-level driver, it
|
94 |
|
|
may be process context, softirq context, or interrupt context.
|
95 |
|
|
Upper level protocol consumers may not sleep in a callback.
|
96 |
|
|
|
97 |
|
|
Hot-plug
|
98 |
|
|
|
99 |
|
|
A low-level driver announces that a device is ready for use by
|
100 |
|
|
consumers when it calls ib_register_device(), all initialization
|
101 |
|
|
must be complete before this call. The device must remain usable
|
102 |
|
|
until the driver's call to ib_unregister_device() has returned.
|
103 |
|
|
|
104 |
|
|
A low-level driver must call ib_register_device() and
|
105 |
|
|
ib_unregister_device() from process context. It must not hold any
|
106 |
|
|
semaphores that could cause deadlock if a consumer calls back into
|
107 |
|
|
the driver across these calls.
|
108 |
|
|
|
109 |
|
|
An upper level protocol consumer may begin using an IB device as
|
110 |
|
|
soon as the add method of its struct ib_client is called for that
|
111 |
|
|
device. A consumer must finish all cleanup and free all resources
|
112 |
|
|
relating to a device before returning from the remove method.
|
113 |
|
|
|
114 |
|
|
A consumer is permitted to sleep in its add and remove methods.
|