Loading...
Note: File does not exist in v3.1.
1.. SPDX-License-Identifier: GPL-2.0
2
3=======================================
4DSA switch configuration from userspace
5=======================================
6
7The DSA switch configuration is not integrated into the main userspace
8network configuration suites by now and has to be performed manually.
9
10.. _dsa-config-showcases:
11
12Configuration showcases
13-----------------------
14
15To configure a DSA switch a couple of commands need to be executed. In this
16documentation some common configuration scenarios are handled as showcases:
17
18*single port*
19 Every switch port acts as a different configurable Ethernet port
20
21*bridge*
22 Every switch port is part of one configurable Ethernet bridge
23
24*gateway*
25 Every switch port except one upstream port is part of a configurable
26 Ethernet bridge.
27 The upstream port acts as different configurable Ethernet port.
28
29All configurations are performed with tools from iproute2, which is available
30at https://www.kernel.org/pub/linux/utils/net/iproute2/
31
32Through DSA every port of a switch is handled like a normal linux Ethernet
33interface. The CPU port is the switch port connected to an Ethernet MAC chip.
34The corresponding linux Ethernet interface is called the conduit interface.
35All other corresponding linux interfaces are called user interfaces.
36
37The user interfaces depend on the conduit interface being up in order for them
38to send or receive traffic. Prior to kernel v5.12, the state of the conduit
39interface had to be managed explicitly by the user. Starting with kernel v5.12,
40the behavior is as follows:
41
42- when a DSA user interface is brought up, the conduit interface is
43 automatically brought up.
44- when the conduit interface is brought down, all DSA user interfaces are
45 automatically brought down.
46
47In this documentation the following Ethernet interfaces are used:
48
49*eth0*
50 the conduit interface
51
52*eth1*
53 another conduit interface
54
55*lan1*
56 a user interface
57
58*lan2*
59 another user interface
60
61*lan3*
62 a third user interface
63
64*wan*
65 A user interface dedicated for upstream traffic
66
67Further Ethernet interfaces can be configured similar.
68The configured IPs and networks are:
69
70*single port*
71 * lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
72 * lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7)
73 * lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11)
74
75*bridge*
76 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
77
78*gateway*
79 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
80 * wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
81
82.. _dsa-tagged-configuration:
83
84Configuration with tagging support
85----------------------------------
86
87The tagging based configuration is desired and supported by the majority of
88DSA switches. These switches are capable to tag incoming and outgoing traffic
89without using a VLAN based configuration.
90
91*single port*
92 .. code-block:: sh
93
94 # configure each interface
95 ip addr add 192.0.2.1/30 dev lan1
96 ip addr add 192.0.2.5/30 dev lan2
97 ip addr add 192.0.2.9/30 dev lan3
98
99 # For kernels earlier than v5.12, the conduit interface needs to be
100 # brought up manually before the user ports.
101 ip link set eth0 up
102
103 # bring up the user interfaces
104 ip link set lan1 up
105 ip link set lan2 up
106 ip link set lan3 up
107
108*bridge*
109 .. code-block:: sh
110
111 # For kernels earlier than v5.12, the conduit interface needs to be
112 # brought up manually before the user ports.
113 ip link set eth0 up
114
115 # bring up the user interfaces
116 ip link set lan1 up
117 ip link set lan2 up
118 ip link set lan3 up
119
120 # create bridge
121 ip link add name br0 type bridge
122
123 # add ports to bridge
124 ip link set dev lan1 master br0
125 ip link set dev lan2 master br0
126 ip link set dev lan3 master br0
127
128 # configure the bridge
129 ip addr add 192.0.2.129/25 dev br0
130
131 # bring up the bridge
132 ip link set dev br0 up
133
134*gateway*
135 .. code-block:: sh
136
137 # For kernels earlier than v5.12, the conduit interface needs to be
138 # brought up manually before the user ports.
139 ip link set eth0 up
140
141 # bring up the user interfaces
142 ip link set wan up
143 ip link set lan1 up
144 ip link set lan2 up
145
146 # configure the upstream port
147 ip addr add 192.0.2.1/30 dev wan
148
149 # create bridge
150 ip link add name br0 type bridge
151
152 # add ports to bridge
153 ip link set dev lan1 master br0
154 ip link set dev lan2 master br0
155
156 # configure the bridge
157 ip addr add 192.0.2.129/25 dev br0
158
159 # bring up the bridge
160 ip link set dev br0 up
161
162.. _dsa-vlan-configuration:
163
164Configuration without tagging support
165-------------------------------------
166
167A minority of switches are not capable to use a taging protocol
168(DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based
169configuration.
170
171*single port*
172 The configuration can only be set up via VLAN tagging and bridge setup.
173
174 .. code-block:: sh
175
176 # tag traffic on CPU port
177 ip link add link eth0 name eth0.1 type vlan id 1
178 ip link add link eth0 name eth0.2 type vlan id 2
179 ip link add link eth0 name eth0.3 type vlan id 3
180
181 # For kernels earlier than v5.12, the conduit interface needs to be
182 # brought up manually before the user ports.
183 ip link set eth0 up
184 ip link set eth0.1 up
185 ip link set eth0.2 up
186 ip link set eth0.3 up
187
188 # bring up the user interfaces
189 ip link set lan1 up
190 ip link set lan2 up
191 ip link set lan3 up
192
193 # create bridge
194 ip link add name br0 type bridge
195
196 # activate VLAN filtering
197 ip link set dev br0 type bridge vlan_filtering 1
198
199 # add ports to bridges
200 ip link set dev lan1 master br0
201 ip link set dev lan2 master br0
202 ip link set dev lan3 master br0
203
204 # tag traffic on ports
205 bridge vlan add dev lan1 vid 1 pvid untagged
206 bridge vlan add dev lan2 vid 2 pvid untagged
207 bridge vlan add dev lan3 vid 3 pvid untagged
208
209 # configure the VLANs
210 ip addr add 192.0.2.1/30 dev eth0.1
211 ip addr add 192.0.2.5/30 dev eth0.2
212 ip addr add 192.0.2.9/30 dev eth0.3
213
214 # bring up the bridge devices
215 ip link set br0 up
216
217
218*bridge*
219 .. code-block:: sh
220
221 # tag traffic on CPU port
222 ip link add link eth0 name eth0.1 type vlan id 1
223
224 # For kernels earlier than v5.12, the conduit interface needs to be
225 # brought up manually before the user ports.
226 ip link set eth0 up
227 ip link set eth0.1 up
228
229 # bring up the user interfaces
230 ip link set lan1 up
231 ip link set lan2 up
232 ip link set lan3 up
233
234 # create bridge
235 ip link add name br0 type bridge
236
237 # activate VLAN filtering
238 ip link set dev br0 type bridge vlan_filtering 1
239
240 # add ports to bridge
241 ip link set dev lan1 master br0
242 ip link set dev lan2 master br0
243 ip link set dev lan3 master br0
244 ip link set eth0.1 master br0
245
246 # tag traffic on ports
247 bridge vlan add dev lan1 vid 1 pvid untagged
248 bridge vlan add dev lan2 vid 1 pvid untagged
249 bridge vlan add dev lan3 vid 1 pvid untagged
250
251 # configure the bridge
252 ip addr add 192.0.2.129/25 dev br0
253
254 # bring up the bridge
255 ip link set dev br0 up
256
257*gateway*
258 .. code-block:: sh
259
260 # tag traffic on CPU port
261 ip link add link eth0 name eth0.1 type vlan id 1
262 ip link add link eth0 name eth0.2 type vlan id 2
263
264 # For kernels earlier than v5.12, the conduit interface needs to be
265 # brought up manually before the user ports.
266 ip link set eth0 up
267 ip link set eth0.1 up
268 ip link set eth0.2 up
269
270 # bring up the user interfaces
271 ip link set wan up
272 ip link set lan1 up
273 ip link set lan2 up
274
275 # create bridge
276 ip link add name br0 type bridge
277
278 # activate VLAN filtering
279 ip link set dev br0 type bridge vlan_filtering 1
280
281 # add ports to bridges
282 ip link set dev wan master br0
283 ip link set eth0.1 master br0
284 ip link set dev lan1 master br0
285 ip link set dev lan2 master br0
286
287 # tag traffic on ports
288 bridge vlan add dev lan1 vid 1 pvid untagged
289 bridge vlan add dev lan2 vid 1 pvid untagged
290 bridge vlan add dev wan vid 2 pvid untagged
291
292 # configure the VLANs
293 ip addr add 192.0.2.1/30 dev eth0.2
294 ip addr add 192.0.2.129/25 dev br0
295
296 # bring up the bridge devices
297 ip link set br0 up
298
299Forwarding database (FDB) management
300------------------------------------
301
302The existing DSA switches do not have the necessary hardware support to keep
303the software FDB of the bridge in sync with the hardware tables, so the two
304tables are managed separately (``bridge fdb show`` queries both, and depending
305on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb
306add`` or ``bridge fdb del`` command acts upon entries from one or both tables).
307
308Up until kernel v4.14, DSA only supported user space management of bridge FDB
309entries using the bridge bypass operations (which do not update the software
310FDB, just the hardware one) using the ``self`` flag (which is optional and can
311be omitted).
312
313 .. code-block:: sh
314
315 bridge fdb add dev swp0 00:01:02:03:04:05 self static
316 # or shorthand
317 bridge fdb add dev swp0 00:01:02:03:04:05 static
318
319Due to a bug, the bridge bypass FDB implementation provided by DSA did not
320distinguish between ``static`` and ``local`` FDB entries (``static`` are meant
321to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent
322to the host port). Instead, all FDB entries with the ``self`` flag (implicit or
323explicit) are treated by DSA as ``static`` even if they are ``local``.
324
325 .. code-block:: sh
326
327 # This command:
328 bridge fdb add dev swp0 00:01:02:03:04:05 static
329 # behaves the same for DSA as this command:
330 bridge fdb add dev swp0 00:01:02:03:04:05 local
331 # or shorthand, because the 'local' flag is implicit if 'static' is not
332 # specified, it also behaves the same as:
333 bridge fdb add dev swp0 00:01:02:03:04:05
334
335The last command is an incorrect way of adding a static bridge FDB entry to a
336DSA switch using the bridge bypass operations, and works by mistake. Other
337drivers will treat an FDB entry added by the same command as ``local`` and as
338such, will not forward it, as opposed to DSA.
339
340Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of
341adding a bridge FDB entry to the switch: the bridge bypass discussed above, as
342well as a new mode using the ``master`` flag which installs FDB entries in the
343software bridge too.
344
345 .. code-block:: sh
346
347 bridge fdb add dev swp0 00:01:02:03:04:05 master static
348
349Since kernel v5.14, DSA has gained stronger integration with the bridge's
350software FDB, and the support for its bridge bypass FDB implementation (using
351the ``self`` flag) has been removed. This results in the following changes:
352
353 .. code-block:: sh
354
355 # This is the only valid way of adding an FDB entry that is supported,
356 # compatible with v4.14 kernels and later:
357 bridge fdb add dev swp0 00:01:02:03:04:05 master static
358 # This command is no longer buggy and the entry is properly treated as
359 # 'local' instead of being forwarded:
360 bridge fdb add dev swp0 00:01:02:03:04:05
361 # This command no longer installs a static FDB entry to hardware:
362 bridge fdb add dev swp0 00:01:02:03:04:05 static
363
364Script writers are therefore encouraged to use the ``master static`` set of
365flags when working with bridge FDB entries on DSA switch interfaces.
366
367Affinity of user ports to CPU ports
368-----------------------------------
369
370Typically, DSA switches are attached to the host via a single Ethernet
371interface, but in cases where the switch chip is discrete, the hardware design
372may permit the use of 2 or more ports connected to the host, for an increase in
373termination throughput.
374
375DSA can make use of multiple CPU ports in two ways. First, it is possible to
376statically assign the termination traffic associated with a certain user port
377to be processed by a certain CPU port. This way, user space can implement
378custom policies of static load balancing between user ports, by spreading the
379affinities according to the available CPU ports.
380
381Secondly, it is possible to perform load balancing between CPU ports on a per
382packet basis, rather than statically assigning user ports to CPU ports.
383This can be achieved by placing the DSA conduits under a LAG interface (bonding
384or team). DSA monitors this operation and creates a mirror of this software LAG
385on the CPU ports facing the physical DSA conduits that constitute the LAG slave
386devices.
387
388To make use of multiple CPU ports, the firmware (device tree) description of
389the switch must mark all the links between CPU ports and their DSA conduits
390using the ``ethernet`` reference/phandle. At startup, only a single CPU port
391and DSA conduit will be used - the numerically first port from the firmware
392description which has an ``ethernet`` property. It is up to the user to
393configure the system for the switch to use other conduits.
394
395DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow
396changing the DSA conduit of a user port. The ``IFLA_DSA_CONDUIT`` u32 netlink
397attribute contains the ifindex of the conduit device that handles each user
398device. The DSA conduit must be a valid candidate based on firmware node
399information, or a LAG interface which contains only slaves which are valid
400candidates.
401
402Using iproute2, the following manipulations are possible:
403
404 .. code-block:: sh
405
406 # See the DSA conduit in current use
407 ip -d link show dev swp0
408 (...)
409 dsa master eth0
410
411 # Static CPU port distribution
412 ip link set swp0 type dsa master eth1
413 ip link set swp1 type dsa master eth0
414 ip link set swp2 type dsa master eth1
415 ip link set swp3 type dsa master eth0
416
417 # CPU ports in LAG, using explicit assignment of the DSA conduit
418 ip link add bond0 type bond mode balance-xor && ip link set bond0 up
419 ip link set eth1 down && ip link set eth1 master bond0
420 ip link set swp0 type dsa master bond0
421 ip link set swp1 type dsa master bond0
422 ip link set swp2 type dsa master bond0
423 ip link set swp3 type dsa master bond0
424 ip link set eth0 down && ip link set eth0 master bond0
425 ip -d link show dev swp0
426 (...)
427 dsa master bond0
428
429 # CPU ports in LAG, relying on implicit migration of the DSA conduit
430 ip link add bond0 type bond mode balance-xor && ip link set bond0 up
431 ip link set eth0 down && ip link set eth0 master bond0
432 ip link set eth1 down && ip link set eth1 master bond0
433 ip -d link show dev swp0
434 (...)
435 dsa master bond0
436
437Notice that in the case of CPU ports under a LAG, the use of the
438``IFLA_DSA_CONDUIT`` netlink attribute is not strictly needed, but rather, DSA
439reacts to the ``IFLA_MASTER`` attribute change of its present conduit (``eth0``)
440and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly,
441when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports
442that were assigned to this interface to the first physical DSA conduit which is
443eligible, based on the firmware description (it effectively reverts to the
444startup configuration).
445
446In a setup with more than 2 physical CPU ports, it is therefore possible to mix
447static user to CPU port assignment with LAG between DSA conduits. It is not
448possible to statically assign a user port towards a DSA conduit that has any
449upper interfaces (this includes LAG devices - the conduit must always be the LAG
450in this case).
451
452Live changing of the DSA conduit (and thus CPU port) affinity of a user port is
453permitted, in order to allow dynamic redistribution in response to traffic.
454
455Physical DSA conduits are allowed to join and leave at any time a LAG interface
456used as a DSA conduit; however, DSA will reject a LAG interface as a valid
457candidate for being a DSA conduit unless it has at least one physical DSA conduit
458as a slave device.