Loading...
1
21. Control Interfaces
3
4The interfaces for receiving network packages timestamps are:
5
6* SO_TIMESTAMP
7 Generates a timestamp for each incoming packet in (not necessarily
8 monotonic) system time. Reports the timestamp via recvmsg() in a
9 control message as struct timeval (usec resolution).
10
11* SO_TIMESTAMPNS
12 Same timestamping mechanism as SO_TIMESTAMP, but reports the
13 timestamp as struct timespec (nsec resolution).
14
15* IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
16 Only for multicast:approximate transmit timestamp obtained by
17 reading the looped packet receive timestamp.
18
19* SO_TIMESTAMPING
20 Generates timestamps on reception, transmission or both. Supports
21 multiple timestamp sources, including hardware. Supports generating
22 timestamps for stream sockets.
23
24
251.1 SO_TIMESTAMP:
26
27This socket option enables timestamping of datagrams on the reception
28path. Because the destination socket, if any, is not known early in
29the network stack, the feature has to be enabled for all packets. The
30same is true for all early receive timestamp options.
31
32For interface details, see `man 7 socket`.
33
34
351.2 SO_TIMESTAMPNS:
36
37This option is identical to SO_TIMESTAMP except for the returned data type.
38Its struct timespec allows for higher resolution (ns) timestamps than the
39timeval of SO_TIMESTAMP (ms).
40
41
421.3 SO_TIMESTAMPING:
43
44Supports multiple types of timestamp requests. As a result, this
45socket option takes a bitmap of flags, not a boolean. In
46
47 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val,
48 sizeof(val));
49
50val is an integer with any of the following bits set. Setting other
51bit returns EINVAL and does not change the current state.
52
53The socket option configures timestamp generation for individual
54sk_buffs (1.3.1), timestamp reporting to the socket's error
55queue (1.3.2) and options (1.3.3). Timestamp generation can also
56be enabled for individual sendmsg calls using cmsg (1.3.4).
57
58
591.3.1 Timestamp Generation
60
61Some bits are requests to the stack to try to generate timestamps. Any
62combination of them is valid. Changes to these bits apply to newly
63created packets, not to packets already in the stack. As a result, it
64is possible to selectively request timestamps for a subset of packets
65(e.g., for sampling) by embedding an send() call within two setsockopt
66calls, one to enable timestamp generation and one to disable it.
67Timestamps may also be generated for reasons other than being
68requested by a particular socket, such as when receive timestamping is
69enabled system wide, as explained earlier.
70
71SOF_TIMESTAMPING_RX_HARDWARE:
72 Request rx timestamps generated by the network adapter.
73
74SOF_TIMESTAMPING_RX_SOFTWARE:
75 Request rx timestamps when data enters the kernel. These timestamps
76 are generated just after a device driver hands a packet to the
77 kernel receive stack.
78
79SOF_TIMESTAMPING_TX_HARDWARE:
80 Request tx timestamps generated by the network adapter. This flag
81 can be enabled via both socket options and control messages.
82
83SOF_TIMESTAMPING_TX_SOFTWARE:
84 Request tx timestamps when data leaves the kernel. These timestamps
85 are generated in the device driver as close as possible, but always
86 prior to, passing the packet to the network interface. Hence, they
87 require driver support and may not be available for all devices.
88 This flag can be enabled via both socket options and control messages.
89
90
91SOF_TIMESTAMPING_TX_SCHED:
92 Request tx timestamps prior to entering the packet scheduler. Kernel
93 transmit latency is, if long, often dominated by queuing delay. The
94 difference between this timestamp and one taken at
95 SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
96 of protocol processing. The latency incurred in protocol
97 processing, if any, can be computed by subtracting a userspace
98 timestamp taken immediately before send() from this timestamp. On
99 machines with virtual devices where a transmitted packet travels
100 through multiple devices and, hence, multiple packet schedulers,
101 a timestamp is generated at each layer. This allows for fine
102 grained measurement of queuing delay. This flag can be enabled
103 via both socket options and control messages.
104
105SOF_TIMESTAMPING_TX_ACK:
106 Request tx timestamps when all data in the send buffer has been
107 acknowledged. This only makes sense for reliable protocols. It is
108 currently only implemented for TCP. For that protocol, it may
109 over-report measurement, because the timestamp is generated when all
110 data up to and including the buffer at send() was acknowledged: the
111 cumulative acknowledgment. The mechanism ignores SACK and FACK.
112 This flag can be enabled via both socket options and control messages.
113
114
1151.3.2 Timestamp Reporting
116
117The other three bits control which timestamps will be reported in a
118generated control message. Changes to the bits take immediate
119effect at the timestamp reporting locations in the stack. Timestamps
120are only reported for packets that also have the relevant timestamp
121generation request set.
122
123SOF_TIMESTAMPING_SOFTWARE:
124 Report any software timestamps when available.
125
126SOF_TIMESTAMPING_SYS_HARDWARE:
127 This option is deprecated and ignored.
128
129SOF_TIMESTAMPING_RAW_HARDWARE:
130 Report hardware timestamps as generated by
131 SOF_TIMESTAMPING_TX_HARDWARE when available.
132
133
1341.3.3 Timestamp Options
135
136The interface supports the options
137
138SOF_TIMESTAMPING_OPT_ID:
139
140 Generate a unique identifier along with each packet. A process can
141 have multiple concurrent timestamping requests outstanding. Packets
142 can be reordered in the transmit path, for instance in the packet
143 scheduler. In that case timestamps will be queued onto the error
144 queue out of order from the original send() calls. It is not always
145 possible to uniquely match timestamps to the original send() calls
146 based on timestamp order or payload inspection alone, then.
147
148 This option associates each packet at send() with a unique
149 identifier and returns that along with the timestamp. The identifier
150 is derived from a per-socket u32 counter (that wraps). For datagram
151 sockets, the counter increments with each sent packet. For stream
152 sockets, it increments with every byte.
153
154 The counter starts at zero. It is initialized the first time that
155 the socket option is enabled. It is reset each time the option is
156 enabled after having been disabled. Resetting the counter does not
157 change the identifiers of existing packets in the system.
158
159 This option is implemented only for transmit timestamps. There, the
160 timestamp is always looped along with a struct sock_extended_err.
161 The option modifies field ee_data to pass an id that is unique
162 among all possibly concurrently outstanding timestamp requests for
163 that socket.
164
165
166SOF_TIMESTAMPING_OPT_CMSG:
167
168 Support recv() cmsg for all timestamped packets. Control messages
169 are already supported unconditionally on all packets with receive
170 timestamps and on IPv6 packets with transmit timestamp. This option
171 extends them to IPv4 packets with transmit timestamp. One use case
172 is to correlate packets with their egress device, by enabling socket
173 option IP_PKTINFO simultaneously.
174
175
176SOF_TIMESTAMPING_OPT_TSONLY:
177
178 Applies to transmit timestamps only. Makes the kernel return the
179 timestamp as a cmsg alongside an empty packet, as opposed to
180 alongside the original packet. This reduces the amount of memory
181 charged to the socket's receive budget (SO_RCVBUF) and delivers
182 the timestamp even if sysctl net.core.tstamp_allow_data is 0.
183 This option disables SOF_TIMESTAMPING_OPT_CMSG.
184
185SOF_TIMESTAMPING_OPT_STATS:
186
187 Optional stats that are obtained along with the transmit timestamps.
188 It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the
189 transmit timestamp is available, the stats are available in a
190 separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a
191 list of TLVs (struct nlattr) of types. These stats allow the
192 application to associate various transport layer stats with
193 the transmit timestamps, such as how long a certain block of
194 data was limited by peer's receiver window.
195
196New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
197disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
198regardless of the setting of sysctl net.core.tstamp_allow_data.
199
200An exception is when a process needs additional cmsg data, for
201instance SOL_IP/IP_PKTINFO to detect the egress network interface.
202Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
203having access to the contents of the original packet, so cannot be
204combined with SOF_TIMESTAMPING_OPT_TSONLY.
205
206
2071.3.4. Enabling timestamps via control messages
208
209In addition to socket options, timestamp generation can be requested
210per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).
211Using this feature, applications can sample timestamps per sendmsg()
212without paying the overhead of enabling and disabling timestamps via
213setsockopt:
214
215 struct msghdr *msg;
216 ...
217 cmsg = CMSG_FIRSTHDR(msg);
218 cmsg->cmsg_level = SOL_SOCKET;
219 cmsg->cmsg_type = SO_TIMESTAMPING;
220 cmsg->cmsg_len = CMSG_LEN(sizeof(__u32));
221 *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |
222 SOF_TIMESTAMPING_TX_SOFTWARE |
223 SOF_TIMESTAMPING_TX_ACK;
224 err = sendmsg(fd, msg, 0);
225
226The SOF_TIMESTAMPING_TX_* flags set via cmsg will override
227the SOF_TIMESTAMPING_TX_* flags set via setsockopt.
228
229Moreover, applications must still enable timestamp reporting via
230setsockopt to receive timestamps:
231
232 __u32 val = SOF_TIMESTAMPING_SOFTWARE |
233 SOF_TIMESTAMPING_OPT_ID /* or any other flag */;
234 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val,
235 sizeof(val));
236
237
2381.4 Bytestream Timestamps
239
240The SO_TIMESTAMPING interface supports timestamping of bytes in a
241bytestream. Each request is interpreted as a request for when the
242entire contents of the buffer has passed a timestamping point. That
243is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
244when all bytes have reached the device driver, regardless of how
245many packets the data has been converted into.
246
247In general, bytestreams have no natural delimiters and therefore
248correlating a timestamp with data is non-trivial. A range of bytes
249may be split across segments, any segments may be merged (possibly
250coalescing sections of previously segmented buffers associated with
251independent send() calls). Segments can be reordered and the same
252byte range can coexist in multiple segments for protocols that
253implement retransmissions.
254
255It is essential that all timestamps implement the same semantics,
256regardless of these possible transformations, as otherwise they are
257incomparable. Handling "rare" corner cases differently from the
258simple case (a 1:1 mapping from buffer to skb) is insufficient
259because performance debugging often needs to focus on such outliers.
260
261In practice, timestamps can be correlated with segments of a
262bytestream consistently, if both semantics of the timestamp and the
263timing of measurement are chosen correctly. This challenge is no
264different from deciding on a strategy for IP fragmentation. There, the
265definition is that only the first fragment is timestamped. For
266bytestreams, we chose that a timestamp is generated only when all
267bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
268implement and reason about. An implementation that has to take into
269account SACK would be more complex due to possible transmission holes
270and out of order arrival.
271
272On the host, TCP can also break the simple 1:1 mapping from buffer to
273skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
274implementation ensures correctness in all cases by tracking the
275individual last byte passed to send(), even if it is no longer the
276last byte after an skbuff extend or merge operation. It stores the
277relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
278has only one such field, only one timestamp can be generated.
279
280In rare cases, a timestamp request can be missed if two requests are
281collapsed onto the same skb. A process can detect this situation by
282enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
283send time with the value returned for each timestamp. It can prevent
284the situation by always flushing the TCP stack in between requests,
285for instance by enabling TCP_NODELAY and disabling TCP_CORK and
286autocork.
287
288These precautions ensure that the timestamp is generated only when all
289bytes have passed a timestamp point, assuming that the network stack
290itself does not reorder the segments. The stack indeed tries to avoid
291reordering. The one exception is under administrator control: it is
292possible to construct a packet scheduler configuration that delays
293segments from the same stream differently. Such a setup would be
294unusual.
295
296
2972 Data Interfaces
298
299Timestamps are read using the ancillary data feature of recvmsg().
300See `man 3 cmsg` for details of this interface. The socket manual
301page (`man 7 socket`) describes how timestamps generated with
302SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
303
304
3052.1 SCM_TIMESTAMPING records
306
307These timestamps are returned in a control message with cmsg_level
308SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
309
310struct scm_timestamping {
311 struct timespec ts[3];
312};
313
314The structure can return up to three timestamps. This is a legacy
315feature. Only one field is non-zero at any time. Most timestamps
316are passed in ts[0]. Hardware timestamps are passed in ts[2].
317
318ts[1] used to hold hardware timestamps converted to system time.
319Instead, expose the hardware clock device on the NIC directly as
320a HW PTP clock source, to allow time conversion in userspace and
321optionally synchronize system time with a userspace PTP stack such
322as linuxptp. For the PTP clock API, see Documentation/ptp/ptp.txt.
323
3242.1.1 Transmit timestamps with MSG_ERRQUEUE
325
326For transmit timestamps the outgoing packet is looped back to the
327socket's error queue with the send timestamp(s) attached. A process
328receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
329set and with a msg_control buffer sufficiently large to receive the
330relevant metadata structures. The recvmsg call returns the original
331outgoing data packet with two ancillary messages attached.
332
333A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
334embeds a struct sock_extended_err. This defines the error type. For
335timestamps, the ee_errno field is ENOMSG. The other ancillary message
336will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
337embeds the struct scm_timestamping.
338
339
3402.1.1.2 Timestamp types
341
342The semantics of the three struct timespec are defined by field
343ee_info in the extended error structure. It contains a value of
344type SCM_TSTAMP_* to define the actual timestamp passed in
345scm_timestamping.
346
347The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
348control fields discussed previously, with one exception. For legacy
349reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
350SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
351is the first if ts[2] is non-zero, the second otherwise, in which
352case the timestamp is stored in ts[0].
353
354
3552.1.1.3 Fragmentation
356
357Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
358explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
359then only the first fragment is timestamped and returned to the sending
360socket.
361
362
3632.1.1.4 Packet Payload
364
365The calling application is often not interested in receiving the whole
366packet payload that it passed to the stack originally: the socket
367error queue mechanism is just a method to piggyback the timestamp on.
368In this case, the application can choose to read datagrams with a
369smaller buffer, possibly even of length 0. The payload is truncated
370accordingly. Until the process calls recvmsg() on the error queue,
371however, the full packet is queued, taking up budget from SO_RCVBUF.
372
373
3742.1.1.5 Blocking Read
375
376Reading from the error queue is always a non-blocking operation. To
377block waiting on a timestamp, use poll or select. poll() will return
378POLLERR in pollfd.revents if any data is ready on the error queue.
379There is no need to pass this flag in pollfd.events. This flag is
380ignored on request. See also `man 2 poll`.
381
382
3832.1.2 Receive timestamps
384
385On reception, there is no reason to read from the socket error queue.
386The SCM_TIMESTAMPING ancillary data is sent along with the packet data
387on a normal recvmsg(). Since this is not a socket error, it is not
388accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
389the meaning of the three fields in struct scm_timestamping is
390implicitly defined. ts[0] holds a software timestamp if set, ts[1]
391is again deprecated and ts[2] holds a hardware timestamp if set.
392
393
3943. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP
395
396Hardware time stamping must also be initialized for each device driver
397that is expected to do hardware time stamping. The parameter is defined in
398/include/linux/net_tstamp.h as:
399
400struct hwtstamp_config {
401 int flags; /* no flags defined right now, must be zero */
402 int tx_type; /* HWTSTAMP_TX_* */
403 int rx_filter; /* HWTSTAMP_FILTER_* */
404};
405
406Desired behavior is passed into the kernel and to a specific device by
407calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose
408ifr_data points to a struct hwtstamp_config. The tx_type and
409rx_filter are hints to the driver what it is expected to do. If
410the requested fine-grained filtering for incoming packets is not
411supported, the driver may time stamp more than just the requested types
412of packets.
413
414Drivers are free to use a more permissive configuration than the requested
415configuration. It is expected that drivers should only implement directly the
416most generic mode that can be supported. For example if the hardware can
417support HWTSTAMP_FILTER_V2_EVENT, then it should generally always upscale
418HWTSTAMP_FILTER_V2_L2_SYNC_MESSAGE, and so forth, as HWTSTAMP_FILTER_V2_EVENT
419is more generic (and more useful to applications).
420
421A driver which supports hardware time stamping shall update the struct
422with the actual, possibly more permissive configuration. If the
423requested packets cannot be time stamped, then nothing should be
424changed and ERANGE shall be returned (in contrast to EINVAL, which
425indicates that SIOCSHWTSTAMP is not supported at all).
426
427Only a processes with admin rights may change the configuration. User
428space is responsible to ensure that multiple processes don't interfere
429with each other and that the settings are reset.
430
431Any process can read the actual configuration by passing this
432structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has
433not been implemented in all drivers.
434
435/* possible values for hwtstamp_config->tx_type */
436enum {
437 /*
438 * no outgoing packet will need hardware time stamping;
439 * should a packet arrive which asks for it, no hardware
440 * time stamping will be done
441 */
442 HWTSTAMP_TX_OFF,
443
444 /*
445 * enables hardware time stamping for outgoing packets;
446 * the sender of the packet decides which are to be
447 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
448 * before sending the packet
449 */
450 HWTSTAMP_TX_ON,
451};
452
453/* possible values for hwtstamp_config->rx_filter */
454enum {
455 /* time stamp no incoming packet at all */
456 HWTSTAMP_FILTER_NONE,
457
458 /* time stamp any incoming packet */
459 HWTSTAMP_FILTER_ALL,
460
461 /* return value: time stamp all packets requested plus some others */
462 HWTSTAMP_FILTER_SOME,
463
464 /* PTP v1, UDP, any kind of event packet */
465 HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
466
467 /* for the complete list of values, please check
468 * the include file /include/linux/net_tstamp.h
469 */
470};
471
4723.1 Hardware Timestamping Implementation: Device Drivers
473
474A driver which supports hardware time stamping must support the
475SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with
476the actual values as described in the section on SIOCSHWTSTAMP. It
477should also support SIOCGHWTSTAMP.
478
479Time stamps for received packets must be stored in the skb. To get a pointer
480to the shared time stamp structure of the skb call skb_hwtstamps(). Then
481set the time stamps in the structure:
482
483struct skb_shared_hwtstamps {
484 /* hardware time stamp transformed into duration
485 * since arbitrary point in time
486 */
487 ktime_t hwtstamp;
488};
489
490Time stamps for outgoing packets are to be generated as follows:
491- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
492 is set no-zero. If yes, then the driver is expected to do hardware time
493 stamping.
494- If this is possible for the skb and requested, then declare
495 that the driver is doing the time stamping by setting the flag
496 SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with
497
498 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
499
500 You might want to keep a pointer to the associated skb for the next step
501 and not free the skb. A driver not supporting hardware time stamping doesn't
502 do that. A driver must never touch sk_buff::tstamp! It is used to store
503 software generated time stamps by the network subsystem.
504- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
505 as possible. skb_tx_timestamp() provides a software time stamp if requested
506 and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
507- As soon as the driver has sent the packet and/or obtained a
508 hardware time stamp for it, it passes the time stamp back by
509 calling skb_hwtstamp_tx() with the original skb, the raw
510 hardware time stamp. skb_hwtstamp_tx() clones the original skb and
511 adds the timestamps, therefore the original skb has to be freed now.
512 If obtaining the hardware time stamp somehow fails, then the driver
513 should not fall back to software time stamping. The rationale is that
514 this would occur at a later time in the processing pipeline than other
515 software time stamping and therefore could lead to unexpected deltas
516 between time stamps.