Loading...
1.. SPDX-License-Identifier: GPL-2.0
2
3======================================
4HiSilicon PCIe Tune and Trace device
5======================================
6
7Introduction
8============
9
10HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
11integrated Endpoint (RCiEP) device, providing the capability
12to dynamically monitor and tune the PCIe link's events (tune),
13and trace the TLP headers (trace). The two functions are independent,
14but is recommended to use them together to analyze and enhance the
15PCIe link's performance.
16
17On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
18PCIe cores. Each PCIe core includes several Root Ports and a PTT
19RCiEP, like below. The PTT device is capable of tuning and
20tracing the links of the PCIe core.
21::
22
23 +--------------Core 0-------+
24 | | [ PTT ] |
25 | | [Root Port]---[Endpoint]
26 | | [Root Port]---[Endpoint]
27 | | [Root Port]---[Endpoint]
28 Root Complex |------Core 1-------+
29 | | [ PTT ] |
30 | | [Root Port]---[ Switch ]---[Endpoint]
31 | | [Root Port]---[Endpoint] `-[Endpoint]
32 | | [Root Port]---[Endpoint]
33 +---------------------------+
34
35The PTT device driver registers one PMU device for each PTT device.
36The name of each PTT device is composed of 'hisi_ptt' prefix with
37the id of the SICL and the Core where it locates. The Kunpeng 930
38SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
39IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
40Complex for each SICL.
41::
42
43 /sys/bus/event_source/devices/hisi_ptt<sicl_id>_<core_id>
44
45Tune
46====
47
48PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
49Currently we support events in 2 classes. The scope of the events
50covers the PCIe core to which the PTT device belongs.
51
52Each event is presented as a file under $(PTT PMU dir)/tune, and
53a simple open/read/write/close cycle will be used to tune the event.
54::
55
56 $ cd /sys/bus/event_source/devices/hisi_ptt<sicl_id>_<core_id>/tune
57 $ ls
58 qos_tx_cpl qos_tx_np qos_tx_p
59 tx_path_rx_req_alloc_buf_level
60 tx_path_tx_req_alloc_buf_level
61 $ cat qos_tx_dp
62 1
63 $ echo 2 > qos_tx_dp
64 $ cat qos_tx_dp
65 2
66
67Current value (numerical value) of the event can be simply read
68from the file, and the desired value written to the file to tune.
69
701. Tx Path QoS Control
71------------------------
72
73The following files are provided to tune the QoS of the tx path of
74the PCIe core.
75
76- qos_tx_cpl: weight of Tx completion TLPs
77- qos_tx_np: weight of Tx non-posted TLPs
78- qos_tx_p: weight of Tx posted TLPs
79
80The weight influences the proportion of certain packets on the PCIe link.
81For example, for the storage scenario, increase the proportion
82of the completion packets on the link to enhance the performance as
83more completions are consumed.
84
85The available tune data of these events is [0, 1, 2].
86Writing a negative value will return an error, and out of range
87values will be converted to 2. Note that the event value just
88indicates a probable level, but is not precise.
89
902. Tx Path Buffer Control
91-------------------------
92
93Following files are provided to tune the buffer of tx path of the PCIe core.
94
95- rx_alloc_buf_level: watermark of Rx requested
96- tx_alloc_buf_level: watermark of Tx requested
97
98These events influence the watermark of the buffer allocated for each
99type. Rx means the inbound while Tx means outbound. The packets will
100be stored in the buffer first and then transmitted either when the
101watermark reached or when timed out. For a busy direction, you should
102increase the related buffer watermark to avoid frequently posting and
103thus enhance the performance. In most cases just keep the default value.
104
105The available tune data of above events is [0, 1, 2].
106Writing a negative value will return an error, and out of range
107values will be converted to 2. Note that the event value just
108indicates a probable level, but is not precise.
109
110Trace
111=====
112
113PTT trace is designed for dumping the TLP headers to the memory, which
114can be used to analyze the transactions and usage condition of the PCIe
115Link. You can choose to filter the traced headers by either Requester ID,
116or those downstream of a set of Root Ports on the same core of the PTT
117device. It's also supported to trace the headers of certain type and of
118certain direction.
119
120You can use the perf command `perf record` to set the parameters, start
121trace and get the data. It's also supported to decode the trace
122data with `perf report`. The control parameters for trace is inputted
123as event code for each events, which will be further illustrated later.
124An example usage is like
125::
126
127 $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
128 format=1/ -- sleep 5
129
130This will trace the TLP headers downstream root port 0000:00:10.1 (event
131code for event 'filter' is 0x80001) with type of posted TLP requests,
132direction of inbound and traced data format of 8DW.
133
1341. Filter
135---------
136
137The TLP headers to trace can be filtered by the Root Ports or the Requester ID
138of the Endpoint, which are located on the same core of the PTT device. You can
139set the filter by specifying the `filter` parameter which is required to start
140the trace. The parameter value is 20 bit. Bit 19 indicates the filter type.
1411 for Root Port filter and 0 for Requester filter. Bit[15:0] indicates the
142filter value. The value for a Root Port is a mask of the core port id which is
143calculated from its PCI Slot ID as (slotid & 7) * 2. The value for a Requester
144is the Requester ID (Device ID of the PCIe function). Bit[18:16] is currently
145reserved for extension.
146
147For example, if the desired filter is Endpoint function 0000:01:00.1 the filter
148value will be 0x00101. If the desired filter is Root Port 0000:00:10.0 then
149then filter value is calculated as 0x80001.
150
151The driver also presents every supported Root Port and Requester filter through
152sysfs. Each filter will be an individual file with name of its related PCIe
153device name (domain:bus:device.function). The files of Root Port filters are
154under $(PTT PMU dir)/root_port_filters and files of Requester filters
155are under $(PTT PMU dir)/requester_filters.
156
157Note that multiple Root Ports can be specified at one time, but only one
158Endpoint function can be specified in one trace. Specifying both Root Port
159and function at the same time is not supported. Driver maintains a list of
160available filters and will check the invalid inputs.
161
162The available filters will be dynamically updated, which means you will always
163get correct filter information when hotplug events happen, or when you manually
164remove/rescan the devices.
165
1662. Type
167-------
168
169You can trace the TLP headers of certain types by specifying the `type`
170parameter, which is required to start the trace. The parameter value is
1718 bit. Current supported types and related values are shown below:
172
173- 8'b00000001: posted requests (P)
174- 8'b00000010: non-posted requests (NP)
175- 8'b00000100: completions (CPL)
176
177You can specify multiple types when tracing inbound TLP headers, but can only
178specify one when tracing outbound TLP headers.
179
1803. Direction
181------------
182
183You can trace the TLP headers from certain direction, which is relative
184to the Root Port or the PCIe core, by specifying the `direction` parameter.
185This is optional and the default parameter is inbound. The parameter value
186is 4 bit. When the desired format is 4DW, directions and related values
187supported are shown below:
188
189- 4'b0000: inbound TLPs (P, NP, CPL)
190- 4'b0001: outbound TLPs (P, NP, CPL)
191- 4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
192- 4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
193
194When the desired format is 8DW, directions and related values supported are
195shown below:
196
197- 4'b0000: reserved
198- 4'b0001: outbound TLPs (P, NP, CPL)
199- 4'b0010: inbound TLPs (P, NP, CPL B)
200- 4'b0011: inbound TLPs (CPL A)
201
202Inbound completions are classified into two types:
203
204- completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
205- completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
206
2074. Format
208--------------
209
210You can change the format of the traced TLP headers by specifying the
211`format` parameter. The default format is 4DW. The parameter value is 4 bit.
212Current supported formats and related values are shown below:
213
214- 4'b0000: 4DW length per TLP header
215- 4'b0001: 8DW length per TLP header
216
217The traced TLP header format is different from the PCIe standard.
218
219When using the 8DW data format, the entire TLP header is logged
220(Header DW0-3 shown below). For example, the TLP header for Memory
221Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
222the header for Configuration Requests is shown in Figure 2.20, etc.
223
224In addition, 8DW trace buffer entries contain a timestamp and
225possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
226Otherwise this field will be all 0.
227
228The bit[31:11] of DW0 is always 0x1fffff, which can be
229used to distinguish the data format. 8DW format is like
230::
231
232 bits [ 31:11 ][ 10:0 ]
233 |---------------------------------------|-------------------|
234 DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
235 DW1 [ Prefix ]
236 DW2 [ Header DW0 ]
237 DW3 [ Header DW1 ]
238 DW4 [ Header DW2 ]
239 DW5 [ Header DW3 ]
240 DW6 [ Reserved (0x0) ]
241 DW7 [ Time ]
242
243When using the 4DW data format, DW0 of the trace buffer entry
244contains selected fields of DW0 of the TLP, together with a
245timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
246directly from the TLP header.
247
2484DW format is like
249::
250
251 bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
252 |-----|---------|---|---|---|---|-------------|-------------|
253 DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
254 DW1 [ Header DW1 ]
255 DW2 [ Header DW2 ]
256 DW3 [ Header DW3 ]
257
2585. Memory Management
259--------------------
260
261The traced TLP headers will be written to the memory allocated
262by the driver. The hardware accepts 4 DMA address with same size,
263and writes the buffer sequentially like below. If DMA addr 3 is
264finished and the trace is still on, it will return to addr 0.
265::
266
267 +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
268 +---------------------------------------------------------+
269
270Driver will allocate each DMA buffer of 4MiB. The finished buffer
271will be copied to the perf AUX buffer allocated by the perf core.
272Once the AUX buffer is full while the trace is still on, driver
273will commit the AUX buffer first and then apply for a new one with
274the same size. The size of AUX buffer is default to 16MiB. User can
275adjust the size by specifying the `-m` parameter of the perf command.
276
2776. Decoding
278-----------
279
280You can decode the traced data with `perf report -D` command (currently
281only support to dump the raw trace data). The traced data will be decoded
282according to the format described previously (take 8DW as an example):
283::
284
285 [...perf headers and other information]
286 . ... HISI PTT data: size 4194304 bytes
287 . 00000000: 00 00 00 00 Prefix
288 . 00000004: 01 00 00 60 Header DW0
289 . 00000008: 0f 1e 00 01 Header DW1
290 . 0000000c: 04 00 00 00 Header DW2
291 . 00000010: 40 00 81 02 Header DW3
292 . 00000014: 33 c0 04 00 Time
293 . 00000020: 00 00 00 00 Prefix
294 . 00000024: 01 00 00 60 Header DW0
295 . 00000028: 0f 1e 00 01 Header DW1
296 . 0000002c: 04 00 00 00 Header DW2
297 . 00000030: 40 00 81 02 Header DW3
298 . 00000034: 02 00 00 00 Time
299 . 00000040: 00 00 00 00 Prefix
300 . 00000044: 01 00 00 60 Header DW0
301 . 00000048: 0f 1e 00 01 Header DW1
302 . 0000004c: 04 00 00 00 Header DW2
303 . 00000050: 40 00 81 02 Header DW3
304 [...]
1.. SPDX-License-Identifier: GPL-2.0
2
3======================================
4HiSilicon PCIe Tune and Trace device
5======================================
6
7Introduction
8============
9
10HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
11integrated Endpoint (RCiEP) device, providing the capability
12to dynamically monitor and tune the PCIe link's events (tune),
13and trace the TLP headers (trace). The two functions are independent,
14but is recommended to use them together to analyze and enhance the
15PCIe link's performance.
16
17On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
18PCIe cores. Each PCIe core includes several Root Ports and a PTT
19RCiEP, like below. The PTT device is capable of tuning and
20tracing the links of the PCIe core.
21::
22
23 +--------------Core 0-------+
24 | | [ PTT ] |
25 | | [Root Port]---[Endpoint]
26 | | [Root Port]---[Endpoint]
27 | | [Root Port]---[Endpoint]
28 Root Complex |------Core 1-------+
29 | | [ PTT ] |
30 | | [Root Port]---[ Switch ]---[Endpoint]
31 | | [Root Port]---[Endpoint] `-[Endpoint]
32 | | [Root Port]---[Endpoint]
33 +---------------------------+
34
35The PTT device driver registers one PMU device for each PTT device.
36The name of each PTT device is composed of 'hisi_ptt' prefix with
37the id of the SICL and the Core where it locates. The Kunpeng 930
38SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
39IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
40Complex for each SICL.
41::
42
43 /sys/devices/hisi_ptt<sicl_id>_<core_id>
44
45Tune
46====
47
48PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
49Currently we support events in 2 classes. The scope of the events
50covers the PCIe core to which the PTT device belongs.
51
52Each event is presented as a file under $(PTT PMU dir)/tune, and
53a simple open/read/write/close cycle will be used to tune the event.
54::
55
56 $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
57 $ ls
58 qos_tx_cpl qos_tx_np qos_tx_p
59 tx_path_rx_req_alloc_buf_level
60 tx_path_tx_req_alloc_buf_level
61 $ cat qos_tx_dp
62 1
63 $ echo 2 > qos_tx_dp
64 $ cat qos_tx_dp
65 2
66
67Current value (numerical value) of the event can be simply read
68from the file, and the desired value written to the file to tune.
69
701. Tx Path QoS Control
71------------------------
72
73The following files are provided to tune the QoS of the tx path of
74the PCIe core.
75
76- qos_tx_cpl: weight of Tx completion TLPs
77- qos_tx_np: weight of Tx non-posted TLPs
78- qos_tx_p: weight of Tx posted TLPs
79
80The weight influences the proportion of certain packets on the PCIe link.
81For example, for the storage scenario, increase the proportion
82of the completion packets on the link to enhance the performance as
83more completions are consumed.
84
85The available tune data of these events is [0, 1, 2].
86Writing a negative value will return an error, and out of range
87values will be converted to 2. Note that the event value just
88indicates a probable level, but is not precise.
89
902. Tx Path Buffer Control
91-------------------------
92
93Following files are provided to tune the buffer of tx path of the PCIe core.
94
95- rx_alloc_buf_level: watermark of Rx requested
96- tx_alloc_buf_level: watermark of Tx requested
97
98These events influence the watermark of the buffer allocated for each
99type. Rx means the inbound while Tx means outbound. The packets will
100be stored in the buffer first and then transmitted either when the
101watermark reached or when timed out. For a busy direction, you should
102increase the related buffer watermark to avoid frequently posting and
103thus enhance the performance. In most cases just keep the default value.
104
105The available tune data of above events is [0, 1, 2].
106Writing a negative value will return an error, and out of range
107values will be converted to 2. Note that the event value just
108indicates a probable level, but is not precise.
109
110Trace
111=====
112
113PTT trace is designed for dumping the TLP headers to the memory, which
114can be used to analyze the transactions and usage condition of the PCIe
115Link. You can choose to filter the traced headers by either Requester ID,
116or those downstream of a set of Root Ports on the same core of the PTT
117device. It's also supported to trace the headers of certain type and of
118certain direction.
119
120You can use the perf command `perf record` to set the parameters, start
121trace and get the data. It's also supported to decode the trace
122data with `perf report`. The control parameters for trace is inputted
123as event code for each events, which will be further illustrated later.
124An example usage is like
125::
126
127 $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
128 format=1/ -- sleep 5
129
130This will trace the TLP headers downstream root port 0000:00:10.1 (event
131code for event 'filter' is 0x80001) with type of posted TLP requests,
132direction of inbound and traced data format of 8DW.
133
1341. Filter
135---------
136
137The TLP headers to trace can be filtered by the Root Ports or the Requester ID
138of the Endpoint, which are located on the same core of the PTT device. You can
139set the filter by specifying the `filter` parameter which is required to start
140the trace. The parameter value is 20 bit. Bit 19 indicates the filter type.
1411 for Root Port filter and 0 for Requester filter. Bit[15:0] indicates the
142filter value. The value for a Root Port is a mask of the core port id which is
143calculated from its PCI Slot ID as (slotid & 7) * 2. The value for a Requester
144is the Requester ID (Device ID of the PCIe function). Bit[18:16] is currently
145reserved for extension.
146
147For example, if the desired filter is Endpoint function 0000:01:00.1 the filter
148value will be 0x00101. If the desired filter is Root Port 0000:00:10.0 then
149then filter value is calculated as 0x80001.
150
151Note that multiple Root Ports can be specified at one time, but only one
152Endpoint function can be specified in one trace. Specifying both Root Port
153and function at the same time is not supported. Driver maintains a list of
154available filters and will check the invalid inputs.
155
156Currently the available filters are detected in driver's probe. If the supported
157devices are removed/added after probe, you may need to reload the driver to update
158the filters.
159
1602. Type
161-------
162
163You can trace the TLP headers of certain types by specifying the `type`
164parameter, which is required to start the trace. The parameter value is
1658 bit. Current supported types and related values are shown below:
166
167- 8'b00000001: posted requests (P)
168- 8'b00000010: non-posted requests (NP)
169- 8'b00000100: completions (CPL)
170
171You can specify multiple types when tracing inbound TLP headers, but can only
172specify one when tracing outbound TLP headers.
173
1743. Direction
175------------
176
177You can trace the TLP headers from certain direction, which is relative
178to the Root Port or the PCIe core, by specifying the `direction` parameter.
179This is optional and the default parameter is inbound. The parameter value
180is 4 bit. When the desired format is 4DW, directions and related values
181supported are shown below:
182
183- 4'b0000: inbound TLPs (P, NP, CPL)
184- 4'b0001: outbound TLPs (P, NP, CPL)
185- 4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
186- 4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
187
188When the desired format is 8DW, directions and related values supported are
189shown below:
190
191- 4'b0000: reserved
192- 4'b0001: outbound TLPs (P, NP, CPL)
193- 4'b0010: inbound TLPs (P, NP, CPL B)
194- 4'b0011: inbound TLPs (CPL A)
195
196Inbound completions are classified into two types:
197
198- completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
199- completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
200
2014. Format
202--------------
203
204You can change the format of the traced TLP headers by specifying the
205`format` parameter. The default format is 4DW. The parameter value is 4 bit.
206Current supported formats and related values are shown below:
207
208- 4'b0000: 4DW length per TLP header
209- 4'b0001: 8DW length per TLP header
210
211The traced TLP header format is different from the PCIe standard.
212
213When using the 8DW data format, the entire TLP header is logged
214(Header DW0-3 shown below). For example, the TLP header for Memory
215Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
216the header for Configuration Requests is shown in Figure 2.20, etc.
217
218In addition, 8DW trace buffer entries contain a timestamp and
219possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
220Otherwise this field will be all 0.
221
222The bit[31:11] of DW0 is always 0x1fffff, which can be
223used to distinguish the data format. 8DW format is like
224::
225
226 bits [ 31:11 ][ 10:0 ]
227 |---------------------------------------|-------------------|
228 DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
229 DW1 [ Prefix ]
230 DW2 [ Header DW0 ]
231 DW3 [ Header DW1 ]
232 DW4 [ Header DW2 ]
233 DW5 [ Header DW3 ]
234 DW6 [ Reserved (0x0) ]
235 DW7 [ Time ]
236
237When using the 4DW data format, DW0 of the trace buffer entry
238contains selected fields of DW0 of the TLP, together with a
239timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
240directly from the TLP header.
241
2424DW format is like
243::
244
245 bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
246 |-----|---------|---|---|---|---|-------------|-------------|
247 DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
248 DW1 [ Header DW1 ]
249 DW2 [ Header DW2 ]
250 DW3 [ Header DW3 ]
251
2525. Memory Management
253--------------------
254
255The traced TLP headers will be written to the memory allocated
256by the driver. The hardware accepts 4 DMA address with same size,
257and writes the buffer sequentially like below. If DMA addr 3 is
258finished and the trace is still on, it will return to addr 0.
259::
260
261 +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
262 +---------------------------------------------------------+
263
264Driver will allocate each DMA buffer of 4MiB. The finished buffer
265will be copied to the perf AUX buffer allocated by the perf core.
266Once the AUX buffer is full while the trace is still on, driver
267will commit the AUX buffer first and then apply for a new one with
268the same size. The size of AUX buffer is default to 16MiB. User can
269adjust the size by specifying the `-m` parameter of the perf command.
270
2716. Decoding
272-----------
273
274You can decode the traced data with `perf report -D` command (currently
275only support to dump the raw trace data). The traced data will be decoded
276according to the format described previously (take 8DW as an example):
277::
278
279 [...perf headers and other information]
280 . ... HISI PTT data: size 4194304 bytes
281 . 00000000: 00 00 00 00 Prefix
282 . 00000004: 01 00 00 60 Header DW0
283 . 00000008: 0f 1e 00 01 Header DW1
284 . 0000000c: 04 00 00 00 Header DW2
285 . 00000010: 40 00 81 02 Header DW3
286 . 00000014: 33 c0 04 00 Time
287 . 00000020: 00 00 00 00 Prefix
288 . 00000024: 01 00 00 60 Header DW0
289 . 00000028: 0f 1e 00 01 Header DW1
290 . 0000002c: 04 00 00 00 Header DW2
291 . 00000030: 40 00 81 02 Header DW3
292 . 00000034: 02 00 00 00 Time
293 . 00000040: 00 00 00 00 Prefix
294 . 00000044: 01 00 00 60 Header DW0
295 . 00000048: 0f 1e 00 01 Header DW1
296 . 0000004c: 04 00 00 00 Header DW2
297 . 00000050: 40 00 81 02 Header DW3
298 [...]