Loading...
Note: File does not exist in v4.6.
1============
2SNMP counter
3============
4
5This document explains the meaning of SNMP counters.
6
7General IPv4 counters
8=====================
9All layer 4 packets and ICMP packets will change these counters, but
10these counters won't be changed by layer 2 packets (such as STP) or
11ARP packets.
12
13* IpInReceives
14
15Defined in `RFC1213 ipInReceives`_
16
17.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
18
19The number of packets received by the IP layer. It gets increasing at the
20beginning of ip_rcv function, always be updated together with
21IpExtInOctets. It will be increased even if the packet is dropped
22later (e.g. due to the IP header is invalid or the checksum is wrong
23and so on). It indicates the number of aggregated segments after
24GRO/LRO.
25
26* IpInDelivers
27
28Defined in `RFC1213 ipInDelivers`_
29
30.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
31
32The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
33ICMP and so on. If no one listens on a raw socket, only kernel
34supported protocols will be delivered, if someone listens on the raw
35socket, all valid IP packets will be delivered.
36
37* IpOutRequests
38
39Defined in `RFC1213 ipOutRequests`_
40
41.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
42
43The number of packets sent via IP layer, for both single cast and
44multicast packets, and would always be updated together with
45IpExtOutOctets.
46
47* IpExtInOctets and IpExtOutOctets
48
49They are Linux kernel extensions, no RFC definitions. Please note,
50RFC1213 indeed defines ifInOctets and ifOutOctets, but they
51are different things. The ifInOctets and ifOutOctets include the MAC
52layer header size but IpExtInOctets and IpExtOutOctets don't, they
53only include the IP layer header and the IP layer data.
54
55* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
56
57They indicate the number of four kinds of ECN IP packets, please refer
58`Explicit Congestion Notification`_ for more details.
59
60.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
61
62These 4 counters calculate how many packets received per ECN
63status. They count the real frame number regardless the LRO/GRO. So
64for the same packet, you might find that IpInReceives count 1, but
65IpExtInNoECTPkts counts 2 or more.
66
67* IpInHdrErrors
68
69Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
70dropped due to the IP header error. It might happen in both IP input
71and IP forward paths.
72
73.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
74
75* IpInAddrErrors
76
77Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
78scenarios: (1) The IP address is invalid. (2) The destination IP
79address is not a local address and IP forwarding is not enabled
80
81.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
82
83* IpExtInNoRoutes
84
85This counter means the packet is dropped when the IP stack receives a
86packet and can't find a route for it from the route table. It might
87happen when IP forwarding is enabled and the destination IP address is
88not a local address and there is no route for the destination IP
89address.
90
91* IpInUnknownProtos
92
93Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
94layer 4 protocol is unsupported by kernel. If an application is using
95raw socket, kernel will always deliver the packet to the raw socket
96and this counter won't be increased.
97
98.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
99
100* IpExtInTruncatedPkts
101
102For IPv4 packet, it means the actual data size is smaller than the
103"Total Length" field in the IPv4 header.
104
105* IpInDiscards
106
107Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
108in the IP receiving path and due to kernel internal reasons (e.g. no
109enough memory).
110
111.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
112
113* IpOutDiscards
114
115Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
116dropped in the IP sending path and due to kernel internal reasons.
117
118.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
119
120* IpOutNoRoutes
121
122Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
123dropped in the IP sending path and no route is found for it.
124
125.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
126
127ICMP counters
128=============
129* IcmpInMsgs and IcmpOutMsgs
130
131Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
132
133.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
134.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
135
136As mentioned in the RFC1213, these two counters include errors, they
137would be increased even if the ICMP packet has an invalid type. The
138ICMP output path will check the header of a raw socket, so the
139IcmpOutMsgs would still be updated if the IP header is constructed by
140a userspace program.
141
142* ICMP named types
143
144| These counters include most of common ICMP types, they are:
145| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
146| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
147| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
148| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
149| IcmpInRedirects: `RFC1213 icmpInRedirects`_
150| IcmpInEchos: `RFC1213 icmpInEchos`_
151| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
152| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
153| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
154| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
155| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
156| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
157| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
158| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
159| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
160| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
161| IcmpOutEchos: `RFC1213 icmpOutEchos`_
162| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
163| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
164| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
165| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
166| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
167
168.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
169.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
170.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
171.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
172.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
173.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
174.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
175.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
176.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
177.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
178.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
179
180.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
181.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
182.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
183.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
184.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
185.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
186.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
187.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
188.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
189.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
190.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
191
192Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
193Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
194straightforward. The 'In' counter means kernel receives such a packet
195and the 'Out' counter means kernel sends such a packet.
196
197* ICMP numeric types
198
199They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
200ICMP type number. These counters track all kinds of ICMP packets. The
201ICMP type number definition could be found in the `ICMP parameters`_
202document.
203
204.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
205
206For example, if the Linux kernel sends an ICMP Echo packet, the
207IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
208packet, IcmpMsgInType0 would increase 1.
209
210* IcmpInCsumErrors
211
212This counter indicates the checksum of the ICMP packet is
213wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
214before updating IcmpMsgInType[N]. If a packet has bad checksum, the
215IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
216
217* IcmpInErrors and IcmpOutErrors
218
219Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
220
221.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
222.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
223
224When an error occurs in the ICMP packet handler path, these two
225counters would be updated. The receiving packet path use IcmpInErrors
226and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
227is increased, IcmpInErrors would always be increased too.
228
229relationship of the ICMP counters
230---------------------------------
231The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
232are updated at the same time. The sum of IcmpMsgInType[N] plus
233IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
234receives an ICMP packet, kernel follows below logic:
235
2361. increase IcmpInMsgs
2372. if has any error, update IcmpInErrors and finish the process
2383. update IcmpMsgOutType[N]
2394. handle the packet depending on the type, if has any error, update
240 IcmpInErrors and finish the process
241
242So if all errors occur in step (2), IcmpInMsgs should be equal to the
243sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
244step (4), IcmpInMsgs should be equal to the sum of
245IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
246IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
247IcmpInErrors.
248
249General TCP counters
250====================
251* TcpInSegs
252
253Defined in `RFC1213 tcpInSegs`_
254
255.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
256
257The number of packets received by the TCP layer. As mentioned in
258RFC1213, it includes the packets received in error, such as checksum
259error, invalid TCP header and so on. Only one error won't be included:
260if the layer 2 destination address is not the NIC's layer 2
261address. It might happen if the packet is a multicast or broadcast
262packet, or the NIC is in promiscuous mode. In these situations, the
263packets would be delivered to the TCP layer, but the TCP layer will discard
264these packets before increasing TcpInSegs. The TcpInSegs counter
265isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
266counter would only increase 1.
267
268* TcpOutSegs
269
270Defined in `RFC1213 tcpOutSegs`_
271
272.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
273
274The number of packets sent by the TCP layer. As mentioned in RFC1213,
275it excludes the retransmitted packets. But it includes the SYN, ACK
276and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
277GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
278increase 2.
279
280* TcpActiveOpens
281
282Defined in `RFC1213 tcpActiveOpens`_
283
284.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
285
286It means the TCP layer sends a SYN, and come into the SYN-SENT
287state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
288increase 1.
289
290* TcpPassiveOpens
291
292Defined in `RFC1213 tcpPassiveOpens`_
293
294.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
295
296It means the TCP layer receives a SYN, replies a SYN+ACK, come into
297the SYN-RCVD state.
298
299* TcpExtTCPRcvCoalesce
300
301When packets are received by the TCP layer and are not be read by the
302application, the TCP layer will try to merge them. This counter
303indicate how many packets are merged in such situation. If GRO is
304enabled, lots of packets would be merged by GRO, these packets
305wouldn't be counted to TcpExtTCPRcvCoalesce.
306
307* TcpExtTCPAutoCorking
308
309When sending packets, the TCP layer will try to merge small packets to
310a bigger one. This counter increase 1 for every packet merged in such
311situation. Please refer to the LWN article for more details:
312https://lwn.net/Articles/576263/
313
314* TcpExtTCPOrigDataSent
315
316This counter is explained by kernel commit f19c29e3e391, I pasted the
317explanation below::
318
319 TCPOrigDataSent: number of outgoing packets with original data (excluding
320 retransmission but including data-in-SYN). This counter is different from
321 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
322 more useful to track the TCP retransmission rate.
323
324* TCPSynRetrans
325
326This counter is explained by kernel commit f19c29e3e391, I pasted the
327explanation below::
328
329 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
330 retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
331
332* TCPFastOpenActiveFail
333
334This counter is explained by kernel commit f19c29e3e391, I pasted the
335explanation below::
336
337 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
338 the remote does not accept it or the attempts timed out.
339
340* TcpExtListenOverflows and TcpExtListenDrops
341
342When kernel receives a SYN from a client, and if the TCP accept queue
343is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
344At the same time kernel will also add 1 to TcpExtListenDrops. When a
345TCP socket is in LISTEN state, and kernel need to drop a packet,
346kernel would always add 1 to TcpExtListenDrops. So increase
347TcpExtListenOverflows would let TcpExtListenDrops increasing at the
348same time, but TcpExtListenDrops would also increase without
349TcpExtListenOverflows increasing, e.g. a memory allocation fail would
350also let TcpExtListenDrops increase.
351
352Note: The above explanation is based on kernel 4.10 or above version, on
353an old kernel, the TCP stack has different behavior when TCP accept
354queue is full. On the old kernel, TCP stack won't drop the SYN, it
355would complete the 3-way handshake. As the accept queue is full, TCP
356stack will keep the socket in the TCP half-open queue. As it is in the
357half open queue, TCP stack will send SYN+ACK on an exponential backoff
358timer, after client replies ACK, TCP stack checks whether the accept
359queue is still full, if it is not full, moves the socket to the accept
360queue, if it is full, keeps the socket in the half-open queue, at next
361time client replies ACK, this socket will get another chance to move
362to the accept queue.
363
364
365TCP Fast Open
366=============
367* TcpEstabResets
368
369Defined in `RFC1213 tcpEstabResets`_.
370
371.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
372
373* TcpAttemptFails
374
375Defined in `RFC1213 tcpAttemptFails`_.
376
377.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
378
379* TcpOutRsts
380
381Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
382the 'segments sent containing the RST flag', but in linux kernel, this
383counter indicates the segments kernel tried to send. The sending
384process might be failed due to some errors (e.g. memory alloc failed).
385
386.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
387
388* TcpExtTCPSpuriousRtxHostQueues
389
390When the TCP stack wants to retransmit a packet, and finds that packet
391is not lost in the network, but the packet is not sent yet, the TCP
392stack would give up the retransmission and update this counter. It
393might happen if a packet stays too long time in a qdisc or driver
394queue.
395
396* TcpEstabResets
397
398The socket receives a RST packet in Establish or CloseWait state.
399
400* TcpExtTCPKeepAlive
401
402This counter indicates many keepalive packets were sent. The keepalive
403won't be enabled by default. A userspace program could enable it by
404setting the SO_KEEPALIVE socket option.
405
406* TcpExtTCPSpuriousRTOs
407
408The spurious retransmission timeout detected by the `F-RTO`_
409algorithm.
410
411.. _F-RTO: https://tools.ietf.org/html/rfc5682
412
413TCP Fast Path
414=============
415When kernel receives a TCP packet, it has two paths to handler the
416packet, one is fast path, another is slow path. The comment in kernel
417code provides a good explanation of them, I pasted them below::
418
419 It is split into a fast path and a slow path. The fast path is
420 disabled when:
421
422 - A zero window was announced from us
423 - zero window probing
424 is only handled properly on the slow path.
425 - Out of order segments arrived.
426 - Urgent data is expected.
427 - There is no buffer space left
428 - Unexpected TCP flags/window values/header lengths are received
429 (detected by checking the TCP header against pred_flags)
430 - Data is sent in both directions. The fast path only supports pure senders
431 or pure receivers (this means either the sequence number or the ack
432 value must stay constant)
433 - Unexpected TCP option.
434
435Kernel will try to use fast path unless any of the above conditions
436are satisfied. If the packets are out of order, kernel will handle
437them in slow path, which means the performance might be not very
438good. Kernel would also come into slow path if the "Delayed ack" is
439used, because when using "Delayed ack", the data is sent in both
440directions. When the TCP window scale option is not used, kernel will
441try to enable fast path immediately when the connection comes into the
442established state, but if the TCP window scale option is used, kernel
443will disable the fast path at first, and try to enable it after kernel
444receives packets.
445
446* TcpExtTCPPureAcks and TcpExtTCPHPAcks
447
448If a packet set ACK flag and has no data, it is a pure ACK packet, if
449kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
450if kernel handles it in the slow path, TcpExtTCPPureAcks will
451increase 1.
452
453* TcpExtTCPHPHits
454
455If a TCP packet has data (which means it is not a pure ACK packet),
456and this packet is handled in the fast path, TcpExtTCPHPHits will
457increase 1.
458
459
460TCP abort
461=========
462* TcpExtTCPAbortOnData
463
464It means TCP layer has data in flight, but need to close the
465connection. So TCP layer sends a RST to the other side, indicate the
466connection is not closed very graceful. An easy way to increase this
467counter is using the SO_LINGER option. Please refer to the SO_LINGER
468section of the `socket man page`_:
469
470.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
471
472By default, when an application closes a connection, the close function
473will return immediately and kernel will try to send the in-flight data
474async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
475to a positive number, the close function won't return immediately, but
476wait for the in-flight data are acked by the other side, the max wait
477time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
478when the application closes a connection, kernel will send a RST
479immediately and increase the TcpExtTCPAbortOnData counter.
480
481* TcpExtTCPAbortOnClose
482
483This counter means the application has unread data in the TCP layer when
484the application wants to close the TCP connection. In such a situation,
485kernel will send a RST to the other side of the TCP connection.
486
487* TcpExtTCPAbortOnMemory
488
489When an application closes a TCP connection, kernel still need to track
490the connection, let it complete the TCP disconnect process. E.g. an
491app calls the close method of a socket, kernel sends fin to the other
492side of the connection, then the app has no relationship with the
493socket any more, but kernel need to keep the socket, this socket
494becomes an orphan socket, kernel waits for the reply of the other side,
495and would come to the TIME_WAIT state finally. When kernel has no
496enough memory to keep the orphan socket, kernel would send an RST to
497the other side, and delete the socket, in such situation, kernel will
498increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
499TcpExtTCPAbortOnMemory:
500
5011. the memory used by the TCP protocol is higher than the third value of
502the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
503
504.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
505
5062. the orphan socket count is higher than net.ipv4.tcp_max_orphans
507
508
509* TcpExtTCPAbortOnTimeout
510
511This counter will increase when any of the TCP timers expire. In such
512situation, kernel won't send RST, just give up the connection.
513
514* TcpExtTCPAbortOnLinger
515
516When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
517for the fin packet from the other side, kernel could send a RST and
518delete the socket immediately. This is not the default behavior of
519Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
520you could let kernel follow this behavior.
521
522* TcpExtTCPAbortFailed
523
524The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
525satisfied. If an internal error occurs during this process,
526TcpExtTCPAbortFailed will be increased.
527
528.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
529
530TCP Hybrid Slow Start
531=====================
532The Hybrid Slow Start algorithm is an enhancement of the traditional
533TCP congestion window Slow Start algorithm. It uses two pieces of
534information to detect whether the max bandwidth of the TCP path is
535approached. The two pieces of information are ACK train length and
536increase in packet delay. For detail information, please refer the
537`Hybrid Slow Start paper`_. Either ACK train length or packet delay
538hits a specific threshold, the congestion control algorithm will come
539into the Congestion Avoidance state. Until v4.20, two congestion
540control algorithms are using Hybrid Slow Start, they are cubic (the
541default congestion control algorithm) and cdg. Four snmp counters
542relate with the Hybrid Slow Start algorithm.
543
544.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf
545
546* TcpExtTCPHystartTrainDetect
547
548How many times the ACK train length threshold is detected
549
550* TcpExtTCPHystartTrainCwnd
551
552The sum of CWND detected by ACK train length. Dividing this value by
553TcpExtTCPHystartTrainDetect is the average CWND which detected by the
554ACK train length.
555
556* TcpExtTCPHystartDelayDetect
557
558How many times the packet delay threshold is detected.
559
560* TcpExtTCPHystartDelayCwnd
561
562The sum of CWND detected by packet delay. Dividing this value by
563TcpExtTCPHystartDelayDetect is the average CWND which detected by the
564packet delay.
565
566TCP retransmission and congestion control
567=========================================
568The TCP protocol has two retransmission mechanisms: SACK and fast
569recovery. They are exclusive with each other. When SACK is enabled,
570the kernel TCP stack would use SACK, or kernel would use fast
571recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
572the fast recovery is defined in `RFC6582`_, which is also called
573'Reno'.
574
575The TCP congestion control is a big and complex topic. To understand
576the related snmp counter, we need to know the states of the congestion
577control state machine. There are 5 states: Open, Disorder, CWR,
578Recovery and Loss. For details about these states, please refer page 5
579and page 6 of this document:
580https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf
581
582.. _RFC2018: https://tools.ietf.org/html/rfc2018
583.. _RFC6582: https://tools.ietf.org/html/rfc6582
584
585* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery
586
587When the congestion control comes into Recovery state, if sack is
588used, TcpExtTCPSackRecovery increases 1, if sack is not used,
589TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
590stack begins to retransmit the lost packets.
591
592* TcpExtTCPSACKReneging
593
594A packet was acknowledged by SACK, but the receiver has dropped this
595packet, so the sender needs to retransmit this packet. In this
596situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
597could drop a packet which has been acknowledged by SACK, although it is
598unusual, it is allowed by the TCP protocol. The sender doesn't really
599know what happened on the receiver side. The sender just waits until
600the RTO expires for this packet, then the sender assumes this packet
601has been dropped by the receiver.
602
603* TcpExtTCPRenoReorder
604
605The reorder packet is detected by fast recovery. It would only be used
606if SACK is disabled. The fast recovery algorithm detects recorder by
607the duplicate ACK number. E.g., if retransmission is triggered, and
608the original retransmitted packet is not lost, it is just out of
609order, the receiver would acknowledge multiple times, one for the
610retransmitted packet, another for the arriving of the original out of
611order packet. Thus the sender would find more ACks than its
612expectation, and the sender knows out of order occurs.
613
614* TcpExtTCPTSReorder
615
616The reorder packet is detected when a hole is filled. E.g., assume the
617sender sends packet 1,2,3,4,5, and the receiving order is
6181,2,4,5,3. When the sender receives the ACK of packet 3 (which will
619fill the hole), two conditions will let TcpExtTCPTSReorder increase
6201: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
6213 is retransmitted but the timestamp of the packet 3's ACK is earlier
622than the retransmission timestamp.
623
624* TcpExtTCPSACKReorder
625
626The reorder packet detected by SACK. The SACK has two methods to
627detect reorder: (1) DSACK is received by the sender. It means the
628sender sends the same packet more than one times. And the only reason
629is the sender believes an out of order packet is lost so it sends the
630packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
631the sender has received SACKs for packet 2 and 5, now the sender
632receives SACK for packet 4 and the sender doesn't retransmit the
633packet yet, the sender would know packet 4 is out of order. The TCP
634stack of kernel will increase TcpExtTCPSACKReorder for both of the
635above scenarios.
636
637* TcpExtTCPSlowStartRetrans
638
639The TCP stack wants to retransmit a packet and the congestion control
640state is 'Loss'.
641
642* TcpExtTCPFastRetrans
643
644The TCP stack wants to retransmit a packet and the congestion control
645state is not 'Loss'.
646
647* TcpExtTCPLostRetransmit
648
649A SACK points out that a retransmission packet is lost again.
650
651* TcpExtTCPRetransFail
652
653The TCP stack tries to deliver a retransmission packet to lower layers
654but the lower layers return an error.
655
656* TcpExtTCPSynRetrans
657
658The TCP stack retransmits a SYN packet.
659
660DSACK
661=====
662The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
663duplicate packets to the sender. There are two kinds of
664duplications: (1) a packet which has been acknowledged is
665duplicate. (2) an out of order packet is duplicate. The TCP stack
666counts these two kinds of duplications on both receiver side and
667sender side.
668
669.. _RFC2883 : https://tools.ietf.org/html/rfc2883
670
671* TcpExtTCPDSACKOldSent
672
673The TCP stack receives a duplicate packet which has been acked, so it
674sends a DSACK to the sender.
675
676* TcpExtTCPDSACKOfoSent
677
678The TCP stack receives an out of order duplicate packet, so it sends a
679DSACK to the sender.
680
681* TcpExtTCPDSACKRecv
682
683The TCP stack receives a DSACK, which indicates an acknowledged
684duplicate packet is received.
685
686* TcpExtTCPDSACKOfoRecv
687
688The TCP stack receives a DSACK, which indicate an out of order
689duplicate packet is received.
690
691invalid SACK and DSACK
692======================
693When a SACK (or DSACK) block is invalid, a corresponding counter would
694be updated. The validation method is base on the start/end sequence
695number of the SACK block. For more details, please refer the comment
696of the function tcp_is_sackblock_valid in the kernel source code. A
697SACK option could have up to 4 blocks, they are checked
698individually. E.g., if 3 blocks of a SACk is invalid, the
699corresponding counter would be updated 3 times. The comment of commit
70018f02545a9a1 ("[TCP] MIB: Add counters for discarded SACK blocks")
701has additional explanation:
702
703* TcpExtTCPSACKDiscard
704
705This counter indicates how many SACK blocks are invalid. If the invalid
706SACK block is caused by ACK recording, the TCP stack will only ignore
707it and won't update this counter.
708
709* TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo
710
711When a DSACK block is invalid, one of these two counters would be
712updated. Which counter will be updated depends on the undo_marker flag
713of the TCP socket. If the undo_marker is not set, the TCP stack isn't
714likely to re-transmit any packets, and we still receive an invalid
715DSACK block, the reason might be that the packet is duplicated in the
716middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
717will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
718will be updated. As implied in its name, it might be an old packet.
719
720SACK shift
721==========
722The linux networking stack stores data in sk_buff struct (skb for
723short). If a SACK block acrosses multiple skb, the TCP stack will try
724to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
72510 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
72615 in skb2 would be moved to skb1. This operation is 'shift'. If a
727SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
728seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
729discard, this operation is 'merge'.
730
731* TcpExtTCPSackShifted
732
733A skb is shifted
734
735* TcpExtTCPSackMerged
736
737A skb is merged
738
739* TcpExtTCPSackShiftFallback
740
741A skb should be shifted or merged, but the TCP stack doesn't do it for
742some reasons.
743
744TCP out of order
745================
746* TcpExtTCPOFOQueue
747
748The TCP layer receives an out of order packet and has enough memory
749to queue it.
750
751* TcpExtTCPOFODrop
752
753The TCP layer receives an out of order packet but doesn't have enough
754memory, so drops it. Such packets won't be counted into
755TcpExtTCPOFOQueue.
756
757* TcpExtTCPOFOMerge
758
759The received out of order packet has an overlay with the previous
760packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
761packets will also be counted into TcpExtTCPOFOQueue.
762
763TCP PAWS
764========
765PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
766which is used to drop old packets. It depends on the TCP
767timestamps. For detail information, please refer the `timestamp wiki`_
768and the `RFC of PAWS`_.
769
770.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
771.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps
772
773* TcpExtPAWSActive
774
775Packets are dropped by PAWS in Syn-Sent status.
776
777* TcpExtPAWSEstab
778
779Packets are dropped by PAWS in any status other than Syn-Sent.
780
781TCP ACK skip
782============
783In some scenarios, kernel would avoid sending duplicate ACKs too
784frequently. Please find more details in the tcp_invalid_ratelimit
785section of the `sysctl document`_. When kernel decides to skip an ACK
786due to tcp_invalid_ratelimit, kernel would update one of below
787counters to indicate the ACK is skipped in which scenario. The ACK
788would only be skipped if the received packet is either a SYN packet or
789it has no data.
790
791.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
792
793* TcpExtTCPACKSkippedSynRecv
794
795The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
796TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
797waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
798in the Syn-Recv status. But in several scenarios, the TCP stack need
799to send an ACK. E.g., the TCP stack receives the same SYN packet
800repeately, the received packet does not pass the PAWS check, or the
801received packet sequence number is out of window. In these scenarios,
802the TCP stack needs to send ACK. If the ACk sending frequency is higher than
803tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
804increase TcpExtTCPACKSkippedSynRecv.
805
806
807* TcpExtTCPACKSkippedPAWS
808
809The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
810numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
811or Time-Wait statuses, the skipped ACK would be counted to
812TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
813TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
814would be counted to TcpExtTCPACKSkippedPAWS.
815
816* TcpExtTCPACKSkippedSeq
817
818The sequence number is out of window and the timestamp passes the PAWS
819check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
820
821* TcpExtTCPACKSkippedFinWait2
822
823The ACK is skipped in Fin-Wait-2 status, the reason would be either
824PAWS check fails or the received sequence number is out of window.
825
826* TcpExtTCPACKSkippedTimeWait
827
828The ACK is skipped in Time-Wait status, the reason would be either
829PAWS check failed or the received sequence number is out of window.
830
831* TcpExtTCPACKSkippedChallenge
832
833The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
8343 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
835`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
836three scenarios, In some TCP status, the linux TCP stack would also
837send challenge ACKs if the ACK number is before the first
838unacknowledged number (more strict than `RFC 5961 section 5.2`_).
839
840.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
841.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
842.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
843
844TCP receive window
845==================
846* TcpExtTCPWantZeroWindowAdv
847
848Depending on current memory usage, the TCP stack tries to set receive
849window to zero. But the receive window might still be a no-zero
850value. For example, if the previous window size is 10, and the TCP
851stack receives 3 bytes, the current window size would be 7 even if the
852window size calculated by the memory usage is zero.
853
854* TcpExtTCPToZeroWindowAdv
855
856The TCP receive window is set to zero from a no-zero value.
857
858* TcpExtTCPFromZeroWindowAdv
859
860The TCP receive window is set to no-zero value from zero.
861
862
863Delayed ACK
864===========
865The TCP Delayed ACK is a technique which is used for reducing the
866packet count in the network. For more details, please refer the
867`Delayed ACK wiki`_
868
869.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
870
871* TcpExtDelayedACKs
872
873A delayed ACK timer expires. The TCP stack will send a pure ACK packet
874and exit the delayed ACK mode.
875
876* TcpExtDelayedACKLocked
877
878A delayed ACK timer expires, but the TCP stack can't send an ACK
879immediately due to the socket is locked by a userspace program. The
880TCP stack will send a pure ACK later (after the userspace program
881unlock the socket). When the TCP stack sends the pure ACK later, the
882TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
883mode.
884
885* TcpExtDelayedACKLost
886
887It will be updated when the TCP stack receives a packet which has been
888ACKed. A Delayed ACK loss might cause this issue, but it would also be
889triggered by other reasons, such as a packet is duplicated in the
890network.
891
892Tail Loss Probe (TLP)
893=====================
894TLP is an algorithm which is used to detect TCP packet loss. For more
895details, please refer the `TLP paper`_.
896
897.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
898
899* TcpExtTCPLossProbes
900
901A TLP probe packet is sent.
902
903* TcpExtTCPLossProbeRecovery
904
905A packet loss is detected and recovered by TLP.
906
907TCP Fast Open description
908=========================
909TCP Fast Open is a technology which allows data transfer before the
9103-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
911general description.
912
913.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open
914
915* TcpExtTCPFastOpenActive
916
917When the TCP stack receives an ACK packet in the SYN-SENT status, and
918the ACK packet acknowledges the data in the SYN packet, the TCP stack
919understand the TFO cookie is accepted by the other side, then it
920updates this counter.
921
922* TcpExtTCPFastOpenActiveFail
923
924This counter indicates that the TCP stack initiated a TCP Fast Open,
925but it failed. This counter would be updated in three scenarios: (1)
926the other side doesn't acknowledge the data in the SYN packet. (2) The
927SYN packet which has the TFO cookie is timeout at least once. (3)
928after the 3-way handshake, the retransmission timeout happens
929net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
930fast open after the handshake.
931
932* TcpExtTCPFastOpenPassive
933
934This counter indicates how many times the TCP stack accepts the fast
935open request.
936
937* TcpExtTCPFastOpenPassiveFail
938
939This counter indicates how many times the TCP stack rejects the fast
940open request. It is caused by either the TFO cookie is invalid or the
941TCP stack finds an error during the socket creating process.
942
943* TcpExtTCPFastOpenListenOverflow
944
945When the pending fast open request number is larger than
946fastopenq->max_qlen, the TCP stack will reject the fast open request
947and update this counter. When this counter is updated, the TCP stack
948won't update TcpExtTCPFastOpenPassive or
949TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
950TCP_FASTOPEN socket operation and it could not be larger than
951net.core.somaxconn. For example:
952
953setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
954
955* TcpExtTCPFastOpenCookieReqd
956
957This counter indicates how many times a client wants to request a TFO
958cookie.
959
960SYN cookies
961===========
962SYN cookies are used to mitigate SYN flood, for details, please refer
963the `SYN cookies wiki`_.
964
965.. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies
966
967* TcpExtSyncookiesSent
968
969It indicates how many SYN cookies are sent.
970
971* TcpExtSyncookiesRecv
972
973How many reply packets of the SYN cookies the TCP stack receives.
974
975* TcpExtSyncookiesFailed
976
977The MSS decoded from the SYN cookie is invalid. When this counter is
978updated, the received packet won't be treated as a SYN cookie and the
979TcpExtSyncookiesRecv counter won't be updated.
980
981Challenge ACK
982=============
983For details of challenge ACK, please refer the explanation of
984TcpExtTCPACKSkippedChallenge.
985
986* TcpExtTCPChallengeACK
987
988The number of challenge acks sent.
989
990* TcpExtTCPSYNChallenge
991
992The number of challenge acks sent in response to SYN packets. After
993updates this counter, the TCP stack might send a challenge ACK and
994update the TcpExtTCPChallengeACK counter, or it might also skip to
995send the challenge and update the TcpExtTCPACKSkippedChallenge.
996
997prune
998=====
999When a socket is under memory pressure, the TCP stack will try to
1000reclaim memory from the receiving queue and out of order queue. One of
1001the reclaiming method is 'collapse', which means allocate a big skb,
1002copy the contiguous skbs to the single big skb, and free these
1003contiguous skbs.
1004
1005* TcpExtPruneCalled
1006
1007The TCP stack tries to reclaim memory for a socket. After updates this
1008counter, the TCP stack will try to collapse the out of order queue and
1009the receiving queue. If the memory is still not enough, the TCP stack
1010will try to discard packets from the out of order queue (and update the
1011TcpExtOfoPruned counter)
1012
1013* TcpExtOfoPruned
1014
1015The TCP stack tries to discard packet on the out of order queue.
1016
1017* TcpExtRcvPruned
1018
1019After 'collapse' and discard packets from the out of order queue, if
1020the actually used memory is still larger than the max allowed memory,
1021this counter will be updated. It means the 'prune' fails.
1022
1023* TcpExtTCPRcvCollapsed
1024
1025This counter indicates how many skbs are freed during 'collapse'.
1026
1027examples
1028========
1029
1030ping test
1031---------
1032Run the ping command against the public dns server 8.8.8.8::
1033
1034 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
1035 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
1036 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
1037
1038 --- 8.8.8.8 ping statistics ---
1039 1 packets transmitted, 1 received, 0% packet loss, time 0ms
1040 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
1041
1042The nstayt result::
1043
1044 nstatuser@nstat-a:~$ nstat
1045 #kernel
1046 IpInReceives 1 0.0
1047 IpInDelivers 1 0.0
1048 IpOutRequests 1 0.0
1049 IcmpInMsgs 1 0.0
1050 IcmpInEchoReps 1 0.0
1051 IcmpOutMsgs 1 0.0
1052 IcmpOutEchos 1 0.0
1053 IcmpMsgInType0 1 0.0
1054 IcmpMsgOutType8 1 0.0
1055 IpExtInOctets 84 0.0
1056 IpExtOutOctets 84 0.0
1057 IpExtInNoECTPkts 1 0.0
1058
1059The Linux server sent an ICMP Echo packet, so IpOutRequests,
1060IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
1061server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
1062IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
1063was passed to the ICMP layer via IP layer, so IpInDelivers was
1064increased 1. The default ping data size is 48, so an ICMP Echo packet
1065and its corresponding Echo Reply packet are constructed by:
1066
1067* 14 bytes MAC header
1068* 20 bytes IP header
1069* 16 bytes ICMP header
1070* 48 bytes data (default value of the ping command)
1071
1072So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
1073
1074tcp 3-way handshake
1075-------------------
1076On server side, we run::
1077
1078 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
1079 Listening on [0.0.0.0] (family 0, port 9000)
1080
1081On client side, we run::
1082
1083 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
1084 Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
1085
1086The server listened on tcp 9000 port, the client connected to it, they
1087completed the 3-way handshake.
1088
1089On server side, we can find below nstat output::
1090
1091 nstatuser@nstat-b:~$ nstat | grep -i tcp
1092 TcpPassiveOpens 1 0.0
1093 TcpInSegs 2 0.0
1094 TcpOutSegs 1 0.0
1095 TcpExtTCPPureAcks 1 0.0
1096
1097On client side, we can find below nstat output::
1098
1099 nstatuser@nstat-a:~$ nstat | grep -i tcp
1100 TcpActiveOpens 1 0.0
1101 TcpInSegs 1 0.0
1102 TcpOutSegs 2 0.0
1103
1104When the server received the first SYN, it replied a SYN+ACK, and came into
1105SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
1106SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
1107packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
1108of the 3-way handshake is a pure ACK without data, so
1109TcpExtTCPPureAcks increased 1.
1110
1111When the client sent SYN, the client came into the SYN-SENT state, so
1112TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
1113ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
11141, TcpOutSegs increased 2.
1115
1116TCP normal traffic
1117------------------
1118Run nc on server::
1119
1120 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1121 Listening on [0.0.0.0] (family 0, port 9000)
1122
1123Run nc on client::
1124
1125 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1126 Connection to nstat-b 9000 port [tcp/*] succeeded!
1127
1128Input a string in the nc client ('hello' in our example)::
1129
1130 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1131 Connection to nstat-b 9000 port [tcp/*] succeeded!
1132 hello
1133
1134The client side nstat output::
1135
1136 nstatuser@nstat-a:~$ nstat
1137 #kernel
1138 IpInReceives 1 0.0
1139 IpInDelivers 1 0.0
1140 IpOutRequests 1 0.0
1141 TcpInSegs 1 0.0
1142 TcpOutSegs 1 0.0
1143 TcpExtTCPPureAcks 1 0.0
1144 TcpExtTCPOrigDataSent 1 0.0
1145 IpExtInOctets 52 0.0
1146 IpExtOutOctets 58 0.0
1147 IpExtInNoECTPkts 1 0.0
1148
1149The server side nstat output::
1150
1151 nstatuser@nstat-b:~$ nstat
1152 #kernel
1153 IpInReceives 1 0.0
1154 IpInDelivers 1 0.0
1155 IpOutRequests 1 0.0
1156 TcpInSegs 1 0.0
1157 TcpOutSegs 1 0.0
1158 IpExtInOctets 58 0.0
1159 IpExtOutOctets 52 0.0
1160 IpExtInNoECTPkts 1 0.0
1161
1162Input a string in nc client side again ('world' in our example)::
1163
1164 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1165 Connection to nstat-b 9000 port [tcp/*] succeeded!
1166 hello
1167 world
1168
1169Client side nstat output::
1170
1171 nstatuser@nstat-a:~$ nstat
1172 #kernel
1173 IpInReceives 1 0.0
1174 IpInDelivers 1 0.0
1175 IpOutRequests 1 0.0
1176 TcpInSegs 1 0.0
1177 TcpOutSegs 1 0.0
1178 TcpExtTCPHPAcks 1 0.0
1179 TcpExtTCPOrigDataSent 1 0.0
1180 IpExtInOctets 52 0.0
1181 IpExtOutOctets 58 0.0
1182 IpExtInNoECTPkts 1 0.0
1183
1184
1185Server side nstat output::
1186
1187 nstatuser@nstat-b:~$ nstat
1188 #kernel
1189 IpInReceives 1 0.0
1190 IpInDelivers 1 0.0
1191 IpOutRequests 1 0.0
1192 TcpInSegs 1 0.0
1193 TcpOutSegs 1 0.0
1194 TcpExtTCPHPHits 1 0.0
1195 IpExtInOctets 58 0.0
1196 IpExtOutOctets 52 0.0
1197 IpExtInNoECTPkts 1 0.0
1198
1199Compare the first client-side nstat and the second client-side nstat,
1200we could find one difference: the first one had a 'TcpExtTCPPureAcks',
1201but the second one had a 'TcpExtTCPHPAcks'. The first server-side
1202nstat and the second server-side nstat had a difference too: the
1203second server-side nstat had a TcpExtTCPHPHits, but the first
1204server-side nstat didn't have it. The network traffic patterns were
1205exactly the same: the client sent a packet to the server, the server
1206replied an ACK. But kernel handled them in different ways. When the
1207TCP window scale option is not used, kernel will try to enable fast
1208path immediately when the connection comes into the established state,
1209but if the TCP window scale option is used, kernel will disable the
1210fast path at first, and try to enable it after kernel receives
1211packets. We could use the 'ss' command to verify whether the window
1212scale option is used. e.g. run below command on either server or
1213client::
1214
1215 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
1216 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
1217 tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
1218 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
1219
1220The 'wscale:7,7' means both server and client set the window scale
1221option to 7. Now we could explain the nstat output in our test:
1222
1223In the first nstat output of client side, the client sent a packet, server
1224reply an ACK, when kernel handled this ACK, the fast path was not
1225enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
1226
1227In the second nstat output of client side, the client sent a packet again,
1228and received another ACK from the server, in this time, the fast path is
1229enabled, and the ACK was qualified for fast path, so it was handled by
1230the fast path, so this ACK was counted into TcpExtTCPHPAcks.
1231
1232In the first nstat output of server side, fast path was not enabled,
1233so there was no 'TcpExtTCPHPHits'.
1234
1235In the second nstat output of server side, the fast path was enabled,
1236and the packet received from client qualified for fast path, so it
1237was counted into 'TcpExtTCPHPHits'.
1238
1239TcpExtTCPAbortOnClose
1240---------------------
1241On the server side, we run below python script::
1242
1243 import socket
1244 import time
1245
1246 port = 9000
1247
1248 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1249 s.bind(('0.0.0.0', port))
1250 s.listen(1)
1251 sock, addr = s.accept()
1252 while True:
1253 time.sleep(9999999)
1254
1255This python script listen on 9000 port, but doesn't read anything from
1256the connection.
1257
1258On the client side, we send the string "hello" by nc::
1259
1260 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
1261
1262Then, we come back to the server side, the server has received the "hello"
1263packet, and the TCP layer has acked this packet, but the application didn't
1264read it yet. We type Ctrl-C to terminate the server script. Then we
1265could find TcpExtTCPAbortOnClose increased 1 on the server side::
1266
1267 nstatuser@nstat-b:~$ nstat | grep -i abort
1268 TcpExtTCPAbortOnClose 1 0.0
1269
1270If we run tcpdump on the server side, we could find the server sent a
1271RST after we type Ctrl-C.
1272
1273TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
1274---------------------------------------------------
1275Below is an example which let the orphan socket count be higher than
1276net.ipv4.tcp_max_orphans.
1277Change tcp_max_orphans to a smaller value on client::
1278
1279 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
1280
1281Client code (create 64 connection to server)::
1282
1283 nstatuser@nstat-a:~$ cat client_orphan.py
1284 import socket
1285 import time
1286
1287 server = 'nstat-b' # server address
1288 port = 9000
1289
1290 count = 64
1291
1292 connection_list = []
1293
1294 for i in range(64):
1295 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1296 s.connect((server, port))
1297 connection_list.append(s)
1298 print("connection_count: %d" % len(connection_list))
1299
1300 while True:
1301 time.sleep(99999)
1302
1303Server code (accept 64 connection from client)::
1304
1305 nstatuser@nstat-b:~$ cat server_orphan.py
1306 import socket
1307 import time
1308
1309 port = 9000
1310 count = 64
1311
1312 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1313 s.bind(('0.0.0.0', port))
1314 s.listen(count)
1315 connection_list = []
1316 while True:
1317 sock, addr = s.accept()
1318 connection_list.append((sock, addr))
1319 print("connection_count: %d" % len(connection_list))
1320
1321Run the python scripts on server and client.
1322
1323On server::
1324
1325 python3 server_orphan.py
1326
1327On client::
1328
1329 python3 client_orphan.py
1330
1331Run iptables on server::
1332
1333 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
1334
1335Type Ctrl-C on client, stop client_orphan.py.
1336
1337Check TcpExtTCPAbortOnMemory on client::
1338
1339 nstatuser@nstat-a:~$ nstat | grep -i abort
1340 TcpExtTCPAbortOnMemory 54 0.0
1341
1342Check orphaned socket count on client::
1343
1344 nstatuser@nstat-a:~$ ss -s
1345 Total: 131 (kernel 0)
1346 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
1347
1348 Transport Total IP IPv6
1349 * 0 - -
1350 RAW 1 0 1
1351 UDP 1 1 0
1352 TCP 14 13 1
1353 INET 16 14 2
1354 FRAG 0 0 0
1355
1356The explanation of the test: after run server_orphan.py and
1357client_orphan.py, we set up 64 connections between server and
1358client. Run the iptables command, the server will drop all packets from
1359the client, type Ctrl-C on client_orphan.py, the system of the client
1360would try to close these connections, and before they are closed
1361gracefully, these connections became orphan sockets. As the iptables
1362of the server blocked packets from the client, the server won't receive fin
1363from the client, so all connection on clients would be stuck on FIN_WAIT_1
1364stage, so they will keep as orphan sockets until timeout. We have echo
136510 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
1366only keep 10 orphan sockets, for all other orphan sockets, the client
1367system sent RST for them and delete them. We have 64 connections, so
1368the 'ss -s' command shows the system has 10 orphan sockets, and the
1369value of TcpExtTCPAbortOnMemory was 54.
1370
1371An additional explanation about orphan socket count: You could find the
1372exactly orphan socket count by the 'ss -s' command, but when kernel
1373decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
1374doesn't always check the exactly orphan socket count. For increasing
1375performance, kernel checks an approximate count firstly, if the
1376approximate count is more than tcp_max_orphans, kernel checks the
1377exact count again. So if the approximate count is less than
1378tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
1379would find TcpExtTCPAbortOnMemory is not increased at all. If
1380tcp_max_orphans is large enough, it won't occur, but if you decrease
1381tcp_max_orphans to a small value like our test, you might find this
1382issue. So in our test, the client set up 64 connections although the
1383tcp_max_orphans is 10. If the client only set up 11 connections, we
1384can't find the change of TcpExtTCPAbortOnMemory.
1385
1386Continue the previous test, we wait for several minutes. Because of the
1387iptables on the server blocked the traffic, the server wouldn't receive
1388fin, and all the client's orphan sockets would timeout on the
1389FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
139010 timeout on the client::
1391
1392 nstatuser@nstat-a:~$ nstat | grep -i abort
1393 TcpExtTCPAbortOnTimeout 10 0.0
1394
1395TcpExtTCPAbortOnLinger
1396----------------------
1397The server side code::
1398
1399 nstatuser@nstat-b:~$ cat server_linger.py
1400 import socket
1401 import time
1402
1403 port = 9000
1404
1405 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1406 s.bind(('0.0.0.0', port))
1407 s.listen(1)
1408 sock, addr = s.accept()
1409 while True:
1410 time.sleep(9999999)
1411
1412The client side code::
1413
1414 nstatuser@nstat-a:~$ cat client_linger.py
1415 import socket
1416 import struct
1417
1418 server = 'nstat-b' # server address
1419 port = 9000
1420
1421 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1422 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
1423 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
1424 s.connect((server, port))
1425 s.close()
1426
1427Run server_linger.py on server::
1428
1429 nstatuser@nstat-b:~$ python3 server_linger.py
1430
1431Run client_linger.py on client::
1432
1433 nstatuser@nstat-a:~$ python3 client_linger.py
1434
1435After run client_linger.py, check the output of nstat::
1436
1437 nstatuser@nstat-a:~$ nstat | grep -i abort
1438 TcpExtTCPAbortOnLinger 1 0.0
1439
1440TcpExtTCPRcvCoalesce
1441--------------------
1442On the server, we run a program which listen on TCP port 9000, but
1443doesn't read any data::
1444
1445 import socket
1446 import time
1447 port = 9000
1448 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1449 s.bind(('0.0.0.0', port))
1450 s.listen(1)
1451 sock, addr = s.accept()
1452 while True:
1453 time.sleep(9999999)
1454
1455Save the above code as server_coalesce.py, and run::
1456
1457 python3 server_coalesce.py
1458
1459On the client, save below code as client_coalesce.py::
1460
1461 import socket
1462 server = 'nstat-b'
1463 port = 9000
1464 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1465 s.connect((server, port))
1466
1467Run::
1468
1469 nstatuser@nstat-a:~$ python3 -i client_coalesce.py
1470
1471We use '-i' to come into the interactive mode, then a packet::
1472
1473 >>> s.send(b'foo')
1474 3
1475
1476Send a packet again::
1477
1478 >>> s.send(b'bar')
1479 3
1480
1481On the server, run nstat::
1482
1483 ubuntu@nstat-b:~$ nstat
1484 #kernel
1485 IpInReceives 2 0.0
1486 IpInDelivers 2 0.0
1487 IpOutRequests 2 0.0
1488 TcpInSegs 2 0.0
1489 TcpOutSegs 2 0.0
1490 TcpExtTCPRcvCoalesce 1 0.0
1491 IpExtInOctets 110 0.0
1492 IpExtOutOctets 104 0.0
1493 IpExtInNoECTPkts 2 0.0
1494
1495The client sent two packets, server didn't read any data. When
1496the second packet arrived at server, the first packet was still in
1497the receiving queue. So the TCP layer merged the two packets, and we
1498could find the TcpExtTCPRcvCoalesce increased 1.
1499
1500TcpExtListenOverflows and TcpExtListenDrops
1501-------------------------------------------
1502On server, run the nc command, listen on port 9000::
1503
1504 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1505 Listening on [0.0.0.0] (family 0, port 9000)
1506
1507On client, run 3 nc commands in different terminals::
1508
1509 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1510 Connection to nstat-b 9000 port [tcp/*] succeeded!
1511
1512The nc command only accepts 1 connection, and the accept queue length
1513is 1. On current linux implementation, set queue length to n means the
1514actual queue length is n+1. Now we create 3 connections, 1 is accepted
1515by nc, 2 in accepted queue, so the accept queue is full.
1516
1517Before running the 4th nc, we clean the nstat history on the server::
1518
1519 nstatuser@nstat-b:~$ nstat -n
1520
1521Run the 4th nc on the client::
1522
1523 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1524
1525If the nc server is running on kernel 4.10 or higher version, you
1526won't see the "Connection to ... succeeded!" string, because kernel
1527will drop the SYN if the accept queue is full. If the nc client is running
1528on an old kernel, you would see that the connection is succeeded,
1529because kernel would complete the 3 way handshake and keep the socket
1530on half open queue. I did the test on kernel 4.15. Below is the nstat
1531on the server::
1532
1533 nstatuser@nstat-b:~$ nstat
1534 #kernel
1535 IpInReceives 4 0.0
1536 IpInDelivers 4 0.0
1537 TcpInSegs 4 0.0
1538 TcpExtListenOverflows 4 0.0
1539 TcpExtListenDrops 4 0.0
1540 IpExtInOctets 240 0.0
1541 IpExtInNoECTPkts 4 0.0
1542
1543Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
1544between the 4th nc and the nstat was longer, the value of
1545TcpExtListenOverflows and TcpExtListenDrops would be larger, because
1546the SYN of the 4th nc was dropped, the client was retrying.
1547
1548IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
1549-------------------------------------------------
1550server A IP address: 192.168.122.250
1551server B IP address: 192.168.122.251
1552Prepare on server A, add a route to server B::
1553
1554 $ sudo ip route add 8.8.8.8/32 via 192.168.122.251
1555
1556Prepare on server B, disable send_redirects for all interfaces::
1557
1558 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
1559 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
1560 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
1561 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
1562
1563We want to let sever A send a packet to 8.8.8.8, and route the packet
1564to server B. When server B receives such packet, it might send a ICMP
1565Redirect message to server A, set send_redirects to 0 will disable
1566this behavior.
1567
1568First, generate InAddrErrors. On server B, we disable IP forwarding::
1569
1570 $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
1571
1572On server A, we send packets to 8.8.8.8::
1573
1574 $ nc -v 8.8.8.8 53
1575
1576On server B, we check the output of nstat::
1577
1578 $ nstat
1579 #kernel
1580 IpInReceives 3 0.0
1581 IpInAddrErrors 3 0.0
1582 IpExtInOctets 180 0.0
1583 IpExtInNoECTPkts 3 0.0
1584
1585As we have let server A route 8.8.8.8 to server B, and we disabled IP
1586forwarding on server B, Server A sent packets to server B, then server B
1587dropped packets and increased IpInAddrErrors. As the nc command would
1588re-send the SYN packet if it didn't receive a SYN+ACK, we could find
1589multiple IpInAddrErrors.
1590
1591Second, generate IpExtInNoRoutes. On server B, we enable IP
1592forwarding::
1593
1594 $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
1595
1596Check the route table of server B and remove the default route::
1597
1598 $ ip route show
1599 default via 192.168.122.1 dev ens3 proto static
1600 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
1601 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static
1602
1603On server A, we contact 8.8.8.8 again::
1604
1605 $ nc -v 8.8.8.8 53
1606 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable
1607
1608On server B, run nstat::
1609
1610 $ nstat
1611 #kernel
1612 IpInReceives 1 0.0
1613 IpOutRequests 1 0.0
1614 IcmpOutMsgs 1 0.0
1615 IcmpOutDestUnreachs 1 0.0
1616 IcmpMsgOutType3 1 0.0
1617 IpExtInNoRoutes 1 0.0
1618 IpExtInOctets 60 0.0
1619 IpExtOutOctets 88 0.0
1620 IpExtInNoECTPkts 1 0.0
1621
1622We enabled IP forwarding on server B, when server B received a packet
1623which destination IP address is 8.8.8.8, server B will try to forward
1624this packet. We have deleted the default route, there was no route for
16258.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
1626Destination Unreachable" message to server A.
1627
1628Third, generate IpOutNoRoutes. Run ping command on server B::
1629
1630 $ ping -c 1 8.8.8.8
1631 connect: Network is unreachable
1632
1633Run nstat on server B::
1634
1635 $ nstat
1636 #kernel
1637 IpOutNoRoutes 1 0.0
1638
1639We have deleted the default route on server B. Server B couldn't find
1640a route for the 8.8.8.8 IP address, so server B increased
1641IpOutNoRoutes.
1642
1643TcpExtTCPACKSkippedSynRecv
1644--------------------------
1645In this test, we send 3 same SYN packets from client to server. The
1646first SYN will let server create a socket, set it to Syn-Recv status,
1647and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
1648again, and record the reply time (the duplicate ACK reply time). The
1649third SYN will let server check the previous duplicate ACK reply time,
1650and decide to skip the duplicate ACK, then increase the
1651TcpExtTCPACKSkippedSynRecv counter.
1652
1653Run tcpdump to capture a SYN packet::
1654
1655 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
1656 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1657
1658Open another terminal, run nc command::
1659
1660 nstatuser@nstat-a:~$ nc nstat-b 9000
1661
1662As the nstat-b didn't listen on port 9000, it should reply a RST, and
1663the nc command exited immediately. It was enough for the tcpdump
1664command to capture a SYN packet. A linux server might use hardware
1665offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
1666might be not correct. We call tcprewrite to fix it::
1667
1668 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
1669
1670On nstat-b, we run nc to listen on port 9000::
1671
1672 nstatuser@nstat-b:~$ nc -lkv 9000
1673 Listening on [0.0.0.0] (family 0, port 9000)
1674
1675On nstat-a, we blocked the packet from port 9000, or nstat-a would send
1676RST to nstat-b::
1677
1678 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
1679
1680Send 3 SYN repeatedly to nstat-b::
1681
1682 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
1683
1684Check snmp counter on nstat-b::
1685
1686 nstatuser@nstat-b:~$ nstat | grep -i skip
1687 TcpExtTCPACKSkippedSynRecv 1 0.0
1688
1689As we expected, TcpExtTCPACKSkippedSynRecv is 1.
1690
1691TcpExtTCPACKSkippedPAWS
1692-----------------------
1693To trigger PAWS, we could send an old SYN.
1694
1695On nstat-b, let nc listen on port 9000::
1696
1697 nstatuser@nstat-b:~$ nc -lkv 9000
1698 Listening on [0.0.0.0] (family 0, port 9000)
1699
1700On nstat-a, run tcpdump to capture a SYN::
1701
1702 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
1703 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1704
1705On nstat-a, run nc as a client to connect nstat-b::
1706
1707 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1708 Connection to nstat-b 9000 port [tcp/*] succeeded!
1709
1710Now the tcpdump has captured the SYN and exit. We should fix the
1711checksum::
1712
1713 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
1714
1715Send the SYN packet twice::
1716
1717 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
1718
1719On nstat-b, check the snmp counter::
1720
1721 nstatuser@nstat-b:~$ nstat | grep -i skip
1722 TcpExtTCPACKSkippedPAWS 1 0.0
1723
1724We sent two SYN via tcpreplay, both of them would let PAWS check
1725failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
1726for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
1727
1728TcpExtTCPACKSkippedSeq
1729----------------------
1730To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
1731timestamp (to pass PAWS check) but the sequence number is out of
1732window. The linux TCP stack would avoid to skip if the packet has
1733data, so we need a pure ACK packet. To generate such a packet, we
1734could create two sockets: one on port 9000, another on port 9001. Then
1735we capture an ACK on port 9001, change the source/destination port
1736numbers to match the port 9000 socket. Then we could trigger
1737TcpExtTCPACKSkippedSeq via this packet.
1738
1739On nstat-b, open two terminals, run two nc commands to listen on both
1740port 9000 and port 9001::
1741
1742 nstatuser@nstat-b:~$ nc -lkv 9000
1743 Listening on [0.0.0.0] (family 0, port 9000)
1744
1745 nstatuser@nstat-b:~$ nc -lkv 9001
1746 Listening on [0.0.0.0] (family 0, port 9001)
1747
1748On nstat-a, run two nc clients::
1749
1750 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1751 Connection to nstat-b 9000 port [tcp/*] succeeded!
1752
1753 nstatuser@nstat-a:~$ nc -v nstat-b 9001
1754 Connection to nstat-b 9001 port [tcp/*] succeeded!
1755
1756On nstat-a, run tcpdump to capture an ACK::
1757
1758 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
1759 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1760
1761On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
1762string 'foo' in our example::
1763
1764 nstatuser@nstat-b:~$ nc -lkv 9001
1765 Listening on [0.0.0.0] (family 0, port 9001)
1766 Connection from nstat-a 42132 received!
1767 foo
1768
1769On nstat-a, the tcpdump should have captured the ACK. We should check
1770the source port numbers of the two nc clients::
1771
1772 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
1773 State Recv-Q Send-Q Local Address:Port Peer Address:Port
1774 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000
1775 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001
1776
1777Run tcprewrite, change port 9001 to port 9000, change port 42132 to
1778port 50208::
1779
1780 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum
1781
1782Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
1783
1784 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
1785
1786Check TcpExtTCPACKSkippedSeq on nstat-b::
1787
1788 nstatuser@nstat-b:~$ nstat | grep -i skip
1789 TcpExtTCPACKSkippedSeq 1 0.0