Loading...
Note: File does not exist in v6.13.7.
1
2
3 HOWTO for the linux packet generator
4 ------------------------------------
5
6Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
7or as a module. A module is preferred; modprobe pktgen if needed. Once
8running, pktgen creates a thread for each CPU with affinity to that CPU.
9Monitoring and controlling is done via /proc. It is easiest to select a
10suitable sample script and configure that.
11
12On a dual CPU:
13
14ps aux | grep pkt
15root 129 0.3 0.0 0 0 ? SW 2003 523:20 [kpktgend_0]
16root 130 0.3 0.0 0 0 ? SW 2003 509:50 [kpktgend_1]
17
18
19For monitoring and control pktgen creates:
20 /proc/net/pktgen/pgctrl
21 /proc/net/pktgen/kpktgend_X
22 /proc/net/pktgen/ethX
23
24
25Tuning NIC for max performance
26==============================
27
28The default NIC settings are (likely) not tuned for pktgen's artificial
29overload type of benchmarking, as this could hurt the normal use-case.
30
31Specifically increasing the TX ring buffer in the NIC:
32 # ethtool -G ethX tx 1024
33
34A larger TX ring can improve pktgen's performance, while it can hurt
35in the general case, 1) because the TX ring buffer might get larger
36than the CPU's L1/L2 cache, 2) because it allows more queueing in the
37NIC HW layer (which is bad for bufferbloat).
38
39One should hesitate to conclude that packets/descriptors in the HW
40TX ring cause delay. Drivers usually delay cleaning up the
41ring-buffers for various performance reasons, and packets stalling
42the TX ring might just be waiting for cleanup.
43
44This cleanup issue is specifically the case for the driver ixgbe
45(Intel 82599 chip). This driver (ixgbe) combines TX+RX ring cleanups,
46and the cleanup interval is affected by the ethtool --coalesce setting
47of parameter "rx-usecs".
48
49For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):
50 # ethtool -C ethX rx-usecs 30
51
52
53Kernel threads
54==============
55Pktgen creates a thread for each CPU with affinity to that CPU.
56Which is controlled through procfile /proc/net/pktgen/kpktgend_X.
57
58Example: /proc/net/pktgen/kpktgend_0
59
60 Running:
61 Stopped: eth4@0
62 Result: OK: add_device=eth4@0
63
64Most important are the devices assigned to the thread.
65
66The two basic thread commands are:
67 * add_device DEVICE@NAME -- adds a single device
68 * rem_device_all -- remove all associated devices
69
70When adding a device to a thread, a corresponding procfile is created
71which is used for configuring this device. Thus, device names need to
72be unique.
73
74To support adding the same device to multiple threads, which is useful
75with multi queue NICs, the device naming scheme is extended with "@":
76 device@something
77
78The part after "@" can be anything, but it is custom to use the thread
79number.
80
81Viewing devices
82===============
83
84The Params section holds configured information. The Current section
85holds running statistics. The Result is printed after a run or after
86interruption. Example:
87
88/proc/net/pktgen/eth4@0
89
90 Params: count 100000 min_pkt_size: 60 max_pkt_size: 60
91 frags: 0 delay: 0 clone_skb: 64 ifname: eth4@0
92 flows: 0 flowlen: 0
93 queue_map_min: 0 queue_map_max: 0
94 dst_min: 192.168.81.2 dst_max:
95 src_min: src_max:
96 src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
97 udp_src_min: 9 udp_src_max: 109 udp_dst_min: 9 udp_dst_max: 9
98 src_mac_count: 0 dst_mac_count: 0
99 Flags: UDPSRC_RND NO_TIMESTAMP QUEUE_MAP_CPU
100 Current:
101 pkts-sofar: 100000 errors: 0
102 started: 623913381008us stopped: 623913396439us idle: 25us
103 seq_num: 100001 cur_dst_mac_offset: 0 cur_src_mac_offset: 0
104 cur_saddr: 192.168.8.3 cur_daddr: 192.168.81.2
105 cur_udp_dst: 9 cur_udp_src: 42
106 cur_queue_map: 0
107 flows: 0
108 Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
109 6480562pps 3110Mb/sec (3110669760bps) errors: 0
110
111
112Configuring devices
113===================
114This is done via the /proc interface, and most easily done via pgset
115as defined in the sample scripts.
116You need to specify PGDEV environment variable to use functions from sample
117scripts, i.e.:
118export PGDEV=/proc/net/pktgen/eth4@0
119source samples/pktgen/functions.sh
120
121Examples:
122
123 pg_ctrl start starts injection.
124 pg_ctrl stop aborts injection. Also, ^C aborts generator.
125
126 pgset "clone_skb 1" sets the number of copies of the same packet
127 pgset "clone_skb 0" use single SKB for all transmits
128 pgset "burst 8" uses xmit_more API to queue 8 copies of the same
129 packet and update HW tx queue tail pointer once.
130 "burst 1" is the default
131 pgset "pkt_size 9014" sets packet size to 9014
132 pgset "frags 5" packet will consist of 5 fragments
133 pgset "count 200000" sets number of packets to send, set to zero
134 for continuous sends until explicitly stopped.
135
136 pgset "delay 5000" adds delay to hard_start_xmit(). nanoseconds
137
138 pgset "dst 10.0.0.1" sets IP destination address
139 (BEWARE! This generator is very aggressive!)
140
141 pgset "dst_min 10.0.0.1" Same as dst
142 pgset "dst_max 10.0.0.254" Set the maximum destination IP.
143 pgset "src_min 10.0.0.1" Set the minimum (or only) source IP.
144 pgset "src_max 10.0.0.254" Set the maximum source IP.
145 pgset "dst6 fec0::1" IPV6 destination address
146 pgset "src6 fec0::2" IPV6 source address
147 pgset "dstmac 00:00:00:00:00:00" sets MAC destination address
148 pgset "srcmac 00:00:00:00:00:00" sets MAC source address
149
150 pgset "queue_map_min 0" Sets the min value of tx queue interval
151 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
152 To select queue 1 of a given device,
153 use queue_map_min=1 and queue_map_max=1
154
155 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
156 The 'minimum' MAC is what you set with srcmac.
157
158 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
159 The 'minimum' MAC is what you set with dstmac.
160
161 pgset "flag [name]" Set a flag to determine behaviour. Current flags
162 are: IPSRC_RND # IP source is random (between min/max)
163 IPDST_RND # IP destination is random
164 UDPSRC_RND, UDPDST_RND,
165 MACSRC_RND, MACDST_RND
166 TXSIZE_RND, IPV6,
167 MPLS_RND, VID_RND, SVID_RND
168 FLOW_SEQ,
169 QUEUE_MAP_RND # queue map random
170 QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
171 UDPCSUM,
172 IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
173 NODE_ALLOC # node specific memory allocation
174 NO_TIMESTAMP # disable timestamping
175 pgset 'flag ![name]' Clear a flag to determine behaviour.
176 Note that you might need to use single quote in
177 interactive mode, so that your shell wouldn't expand
178 the specified flag as a history command.
179
180 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet.
181
182 pgset "udp_src_min 9" set UDP source port min, If < udp_src_max, then
183 cycle through the port range.
184
185 pgset "udp_src_max 9" set UDP source port max.
186 pgset "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then
187 cycle through the port range.
188 pgset "udp_dst_max 9" set UDP destination port max.
189
190 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
191 outer label=16,middle label=32,
192 inner label=0 (IPv4 NULL)) Note that
193 there must be no spaces between the
194 arguments. Leading zeros are required.
195 Do not set the bottom of stack bit,
196 that's done automatically. If you do
197 set the bottom of stack bit, that
198 indicates that you want to randomly
199 generate that address and the flag
200 MPLS_RND will be turned on. You
201 can have any mix of random and fixed
202 labels in the label stack.
203
204 pgset "mpls 0" turn off mpls (or any invalid argument works too!)
205
206 pgset "vlan_id 77" set VLAN ID 0-4095
207 pgset "vlan_p 3" set priority bit 0-7 (default 0)
208 pgset "vlan_cfi 0" set canonical format identifier 0-1 (default 0)
209
210 pgset "svlan_id 22" set SVLAN ID 0-4095
211 pgset "svlan_p 3" set priority bit 0-7 (default 0)
212 pgset "svlan_cfi 0" set canonical format identifier 0-1 (default 0)
213
214 pgset "vlan_id 9999" > 4095 remove vlan and svlan tags
215 pgset "svlan 9999" > 4095 remove svlan tag
216
217
218 pgset "tos XX" set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
219 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
220
221 pgset "rate 300M" set rate to 300 Mb/s
222 pgset "ratep 1000000" set rate to 1Mpps
223
224 pgset "xmit_mode netif_receive" RX inject into stack netif_receive_skb()
225 Works with "burst" but not with "clone_skb".
226 Default xmit_mode is "start_xmit".
227
228Sample scripts
229==============
230
231A collection of tutorial scripts and helpers for pktgen is in the
232samples/pktgen directory. The helper parameters.sh file support easy
233and consistent parameter parsing across the sample scripts.
234
235Usage example and help:
236 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2
237
238Usage: ./pktgen_sample01_simple.sh [-vx] -i ethX
239 -i : ($DEV) output interface/device (required)
240 -s : ($PKT_SIZE) packet size
241 -d : ($DEST_IP) destination IP
242 -m : ($DST_MAC) destination MAC-addr
243 -t : ($THREADS) threads to start
244 -c : ($SKB_CLONE) SKB clones send before alloc new SKB
245 -b : ($BURST) HW level bursting of SKBs
246 -v : ($VERBOSE) verbose
247 -x : ($DEBUG) debug
248
249The global variables being set are also listed. E.g. the required
250interface/device parameter "-i" sets variable $DEV. Copy the
251pktgen_sampleXX scripts and modify them to fit your own needs.
252
253The old scripts:
254
255pktgen.conf-1-2 # 1 CPU 2 dev
256pktgen.conf-1-1-rdos # 1 CPU 1 dev w. route DoS
257pktgen.conf-1-1-ip6 # 1 CPU 1 dev ipv6
258pktgen.conf-1-1-ip6-rdos # 1 CPU 1 dev ipv6 w. route DoS
259pktgen.conf-1-1-flows # 1 CPU 1 dev multiple flows.
260
261
262Interrupt affinity
263===================
264Note that when adding devices to a specific CPU it is a good idea to
265also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
266to the same CPU. This reduces cache bouncing when freeing skbs.
267
268Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
269to the running threads CPU (directly from smp_processor_id()).
270
271Enable IPsec
272============
273Default IPsec transformation with ESP encapsulation plus transport mode
274can be enabled by simply setting:
275
276pgset "flag IPSEC"
277pgset "flows 1"
278
279To avoid breaking existing testbed scripts for using AH type and tunnel mode,
280you can use "pgset spi SPI_VALUE" to specify which transformation mode
281to employ.
282
283
284Current commands and configuration options
285==========================================
286
287** Pgcontrol commands:
288
289start
290stop
291reset
292
293** Thread commands:
294
295add_device
296rem_device_all
297
298
299** Device commands:
300
301count
302clone_skb
303burst
304debug
305
306frags
307delay
308
309src_mac_count
310dst_mac_count
311
312pkt_size
313min_pkt_size
314max_pkt_size
315
316queue_map_min
317queue_map_max
318skb_priority
319
320tos (ipv4)
321traffic_class (ipv6)
322
323mpls
324
325udp_src_min
326udp_src_max
327
328udp_dst_min
329udp_dst_max
330
331node
332
333flag
334 IPSRC_RND
335 IPDST_RND
336 UDPSRC_RND
337 UDPDST_RND
338 MACSRC_RND
339 MACDST_RND
340 TXSIZE_RND
341 IPV6
342 MPLS_RND
343 VID_RND
344 SVID_RND
345 FLOW_SEQ
346 QUEUE_MAP_RND
347 QUEUE_MAP_CPU
348 UDPCSUM
349 IPSEC
350 NODE_ALLOC
351 NO_TIMESTAMP
352
353spi (ipsec)
354
355dst_min
356dst_max
357
358src_min
359src_max
360
361dst_mac
362src_mac
363
364clear_counters
365
366src6
367dst6
368dst6_max
369dst6_min
370
371flows
372flowlen
373
374rate
375ratep
376
377xmit_mode <start_xmit|netif_receive>
378
379vlan_cfi
380vlan_id
381vlan_p
382
383svlan_cfi
384svlan_id
385svlan_p
386
387
388References:
389ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
390ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
391
392Paper from Linux-Kongress in Erlangen 2004.
393ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
394
395Thanks to:
396Grant Grundler for testing on IA-64 and parisc, Harald Welte, Lennert Buytenhek
397Stephen Hemminger, Andi Kleen, Dave Miller and many others.
398
399
400Good luck with the linux net-development.