Loading...
1.. SPDX-License-Identifier: GPL-2.0
2
3=======================================
4DSA switch configuration from userspace
5=======================================
6
7The DSA switch configuration is not integrated into the main userspace
8network configuration suites by now and has to be performed manually.
9
10.. _dsa-config-showcases:
11
12Configuration showcases
13-----------------------
14
15To configure a DSA switch a couple of commands need to be executed. In this
16documentation some common configuration scenarios are handled as showcases:
17
18*single port*
19 Every switch port acts as a different configurable Ethernet port
20
21*bridge*
22 Every switch port is part of one configurable Ethernet bridge
23
24*gateway*
25 Every switch port except one upstream port is part of a configurable
26 Ethernet bridge.
27 The upstream port acts as different configurable Ethernet port.
28
29All configurations are performed with tools from iproute2, which is available
30at https://www.kernel.org/pub/linux/utils/net/iproute2/
31
32Through DSA every port of a switch is handled like a normal linux Ethernet
33interface. The CPU port is the switch port connected to an Ethernet MAC chip.
34The corresponding linux Ethernet interface is called the conduit interface.
35All other corresponding linux interfaces are called user interfaces.
36
37The user interfaces depend on the conduit interface being up in order for them
38to send or receive traffic. Prior to kernel v5.12, the state of the conduit
39interface had to be managed explicitly by the user. Starting with kernel v5.12,
40the behavior is as follows:
41
42- when a DSA user interface is brought up, the conduit interface is
43 automatically brought up.
44- when the conduit interface is brought down, all DSA user interfaces are
45 automatically brought down.
46
47In this documentation the following Ethernet interfaces are used:
48
49*eth0*
50 the conduit interface
51
52*eth1*
53 another conduit interface
54
55*lan1*
56 a user interface
57
58*lan2*
59 another user interface
60
61*lan3*
62 a third user interface
63
64*wan*
65 A user interface dedicated for upstream traffic
66
67Further Ethernet interfaces can be configured similar.
68The configured IPs and networks are:
69
70*single port*
71 * lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
72 * lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7)
73 * lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11)
74
75*bridge*
76 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
77
78*gateway*
79 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
80 * wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
81
82.. _dsa-tagged-configuration:
83
84Configuration with tagging support
85----------------------------------
86
87The tagging based configuration is desired and supported by the majority of
88DSA switches. These switches are capable to tag incoming and outgoing traffic
89without using a VLAN based configuration.
90
91*single port*
92 .. code-block:: sh
93
94 # configure each interface
95 ip addr add 192.0.2.1/30 dev lan1
96 ip addr add 192.0.2.5/30 dev lan2
97 ip addr add 192.0.2.9/30 dev lan3
98
99 # For kernels earlier than v5.12, the conduit interface needs to be
100 # brought up manually before the user ports.
101 ip link set eth0 up
102
103 # bring up the user interfaces
104 ip link set lan1 up
105 ip link set lan2 up
106 ip link set lan3 up
107
108*bridge*
109 .. code-block:: sh
110
111 # For kernels earlier than v5.12, the conduit interface needs to be
112 # brought up manually before the user ports.
113 ip link set eth0 up
114
115 # bring up the user interfaces
116 ip link set lan1 up
117 ip link set lan2 up
118 ip link set lan3 up
119
120 # create bridge
121 ip link add name br0 type bridge
122
123 # add ports to bridge
124 ip link set dev lan1 master br0
125 ip link set dev lan2 master br0
126 ip link set dev lan3 master br0
127
128 # configure the bridge
129 ip addr add 192.0.2.129/25 dev br0
130
131 # bring up the bridge
132 ip link set dev br0 up
133
134*gateway*
135 .. code-block:: sh
136
137 # For kernels earlier than v5.12, the conduit interface needs to be
138 # brought up manually before the user ports.
139 ip link set eth0 up
140
141 # bring up the user interfaces
142 ip link set wan up
143 ip link set lan1 up
144 ip link set lan2 up
145
146 # configure the upstream port
147 ip addr add 192.0.2.1/30 dev wan
148
149 # create bridge
150 ip link add name br0 type bridge
151
152 # add ports to bridge
153 ip link set dev lan1 master br0
154 ip link set dev lan2 master br0
155
156 # configure the bridge
157 ip addr add 192.0.2.129/25 dev br0
158
159 # bring up the bridge
160 ip link set dev br0 up
161
162.. _dsa-vlan-configuration:
163
164Configuration without tagging support
165-------------------------------------
166
167A minority of switches are not capable to use a taging protocol
168(DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based
169configuration.
170
171*single port*
172 The configuration can only be set up via VLAN tagging and bridge setup.
173
174 .. code-block:: sh
175
176 # tag traffic on CPU port
177 ip link add link eth0 name eth0.1 type vlan id 1
178 ip link add link eth0 name eth0.2 type vlan id 2
179 ip link add link eth0 name eth0.3 type vlan id 3
180
181 # For kernels earlier than v5.12, the conduit interface needs to be
182 # brought up manually before the user ports.
183 ip link set eth0 up
184 ip link set eth0.1 up
185 ip link set eth0.2 up
186 ip link set eth0.3 up
187
188 # bring up the user interfaces
189 ip link set lan1 up
190 ip link set lan2 up
191 ip link set lan3 up
192
193 # create bridge
194 ip link add name br0 type bridge
195
196 # activate VLAN filtering
197 ip link set dev br0 type bridge vlan_filtering 1
198
199 # add ports to bridges
200 ip link set dev lan1 master br0
201 ip link set dev lan2 master br0
202 ip link set dev lan3 master br0
203
204 # tag traffic on ports
205 bridge vlan add dev lan1 vid 1 pvid untagged
206 bridge vlan add dev lan2 vid 2 pvid untagged
207 bridge vlan add dev lan3 vid 3 pvid untagged
208
209 # configure the VLANs
210 ip addr add 192.0.2.1/30 dev eth0.1
211 ip addr add 192.0.2.5/30 dev eth0.2
212 ip addr add 192.0.2.9/30 dev eth0.3
213
214 # bring up the bridge devices
215 ip link set br0 up
216
217
218*bridge*
219 .. code-block:: sh
220
221 # tag traffic on CPU port
222 ip link add link eth0 name eth0.1 type vlan id 1
223
224 # For kernels earlier than v5.12, the conduit interface needs to be
225 # brought up manually before the user ports.
226 ip link set eth0 up
227 ip link set eth0.1 up
228
229 # bring up the user interfaces
230 ip link set lan1 up
231 ip link set lan2 up
232 ip link set lan3 up
233
234 # create bridge
235 ip link add name br0 type bridge
236
237 # activate VLAN filtering
238 ip link set dev br0 type bridge vlan_filtering 1
239
240 # add ports to bridge
241 ip link set dev lan1 master br0
242 ip link set dev lan2 master br0
243 ip link set dev lan3 master br0
244 ip link set eth0.1 master br0
245
246 # tag traffic on ports
247 bridge vlan add dev lan1 vid 1 pvid untagged
248 bridge vlan add dev lan2 vid 1 pvid untagged
249 bridge vlan add dev lan3 vid 1 pvid untagged
250
251 # configure the bridge
252 ip addr add 192.0.2.129/25 dev br0
253
254 # bring up the bridge
255 ip link set dev br0 up
256
257*gateway*
258 .. code-block:: sh
259
260 # tag traffic on CPU port
261 ip link add link eth0 name eth0.1 type vlan id 1
262 ip link add link eth0 name eth0.2 type vlan id 2
263
264 # For kernels earlier than v5.12, the conduit interface needs to be
265 # brought up manually before the user ports.
266 ip link set eth0 up
267 ip link set eth0.1 up
268 ip link set eth0.2 up
269
270 # bring up the user interfaces
271 ip link set wan up
272 ip link set lan1 up
273 ip link set lan2 up
274
275 # create bridge
276 ip link add name br0 type bridge
277
278 # activate VLAN filtering
279 ip link set dev br0 type bridge vlan_filtering 1
280
281 # add ports to bridges
282 ip link set dev wan master br0
283 ip link set eth0.1 master br0
284 ip link set dev lan1 master br0
285 ip link set dev lan2 master br0
286
287 # tag traffic on ports
288 bridge vlan add dev lan1 vid 1 pvid untagged
289 bridge vlan add dev lan2 vid 1 pvid untagged
290 bridge vlan add dev wan vid 2 pvid untagged
291
292 # configure the VLANs
293 ip addr add 192.0.2.1/30 dev eth0.2
294 ip addr add 192.0.2.129/25 dev br0
295
296 # bring up the bridge devices
297 ip link set br0 up
298
299Forwarding database (FDB) management
300------------------------------------
301
302The existing DSA switches do not have the necessary hardware support to keep
303the software FDB of the bridge in sync with the hardware tables, so the two
304tables are managed separately (``bridge fdb show`` queries both, and depending
305on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb
306add`` or ``bridge fdb del`` command acts upon entries from one or both tables).
307
308Up until kernel v4.14, DSA only supported user space management of bridge FDB
309entries using the bridge bypass operations (which do not update the software
310FDB, just the hardware one) using the ``self`` flag (which is optional and can
311be omitted).
312
313 .. code-block:: sh
314
315 bridge fdb add dev swp0 00:01:02:03:04:05 self static
316 # or shorthand
317 bridge fdb add dev swp0 00:01:02:03:04:05 static
318
319Due to a bug, the bridge bypass FDB implementation provided by DSA did not
320distinguish between ``static`` and ``local`` FDB entries (``static`` are meant
321to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent
322to the host port). Instead, all FDB entries with the ``self`` flag (implicit or
323explicit) are treated by DSA as ``static`` even if they are ``local``.
324
325 .. code-block:: sh
326
327 # This command:
328 bridge fdb add dev swp0 00:01:02:03:04:05 static
329 # behaves the same for DSA as this command:
330 bridge fdb add dev swp0 00:01:02:03:04:05 local
331 # or shorthand, because the 'local' flag is implicit if 'static' is not
332 # specified, it also behaves the same as:
333 bridge fdb add dev swp0 00:01:02:03:04:05
334
335The last command is an incorrect way of adding a static bridge FDB entry to a
336DSA switch using the bridge bypass operations, and works by mistake. Other
337drivers will treat an FDB entry added by the same command as ``local`` and as
338such, will not forward it, as opposed to DSA.
339
340Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of
341adding a bridge FDB entry to the switch: the bridge bypass discussed above, as
342well as a new mode using the ``master`` flag which installs FDB entries in the
343software bridge too.
344
345 .. code-block:: sh
346
347 bridge fdb add dev swp0 00:01:02:03:04:05 master static
348
349Since kernel v5.14, DSA has gained stronger integration with the bridge's
350software FDB, and the support for its bridge bypass FDB implementation (using
351the ``self`` flag) has been removed. This results in the following changes:
352
353 .. code-block:: sh
354
355 # This is the only valid way of adding an FDB entry that is supported,
356 # compatible with v4.14 kernels and later:
357 bridge fdb add dev swp0 00:01:02:03:04:05 master static
358 # This command is no longer buggy and the entry is properly treated as
359 # 'local' instead of being forwarded:
360 bridge fdb add dev swp0 00:01:02:03:04:05
361 # This command no longer installs a static FDB entry to hardware:
362 bridge fdb add dev swp0 00:01:02:03:04:05 static
363
364Script writers are therefore encouraged to use the ``master static`` set of
365flags when working with bridge FDB entries on DSA switch interfaces.
366
367Affinity of user ports to CPU ports
368-----------------------------------
369
370Typically, DSA switches are attached to the host via a single Ethernet
371interface, but in cases where the switch chip is discrete, the hardware design
372may permit the use of 2 or more ports connected to the host, for an increase in
373termination throughput.
374
375DSA can make use of multiple CPU ports in two ways. First, it is possible to
376statically assign the termination traffic associated with a certain user port
377to be processed by a certain CPU port. This way, user space can implement
378custom policies of static load balancing between user ports, by spreading the
379affinities according to the available CPU ports.
380
381Secondly, it is possible to perform load balancing between CPU ports on a per
382packet basis, rather than statically assigning user ports to CPU ports.
383This can be achieved by placing the DSA conduits under a LAG interface (bonding
384or team). DSA monitors this operation and creates a mirror of this software LAG
385on the CPU ports facing the physical DSA conduits that constitute the LAG slave
386devices.
387
388To make use of multiple CPU ports, the firmware (device tree) description of
389the switch must mark all the links between CPU ports and their DSA conduits
390using the ``ethernet`` reference/phandle. At startup, only a single CPU port
391and DSA conduit will be used - the numerically first port from the firmware
392description which has an ``ethernet`` property. It is up to the user to
393configure the system for the switch to use other conduits.
394
395DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow
396changing the DSA conduit of a user port. The ``IFLA_DSA_CONDUIT`` u32 netlink
397attribute contains the ifindex of the conduit device that handles each user
398device. The DSA conduit must be a valid candidate based on firmware node
399information, or a LAG interface which contains only slaves which are valid
400candidates.
401
402Using iproute2, the following manipulations are possible:
403
404 .. code-block:: sh
405
406 # See the DSA conduit in current use
407 ip -d link show dev swp0
408 (...)
409 dsa master eth0
410
411 # Static CPU port distribution
412 ip link set swp0 type dsa master eth1
413 ip link set swp1 type dsa master eth0
414 ip link set swp2 type dsa master eth1
415 ip link set swp3 type dsa master eth0
416
417 # CPU ports in LAG, using explicit assignment of the DSA conduit
418 ip link add bond0 type bond mode balance-xor && ip link set bond0 up
419 ip link set eth1 down && ip link set eth1 master bond0
420 ip link set swp0 type dsa master bond0
421 ip link set swp1 type dsa master bond0
422 ip link set swp2 type dsa master bond0
423 ip link set swp3 type dsa master bond0
424 ip link set eth0 down && ip link set eth0 master bond0
425 ip -d link show dev swp0
426 (...)
427 dsa master bond0
428
429 # CPU ports in LAG, relying on implicit migration of the DSA conduit
430 ip link add bond0 type bond mode balance-xor && ip link set bond0 up
431 ip link set eth0 down && ip link set eth0 master bond0
432 ip link set eth1 down && ip link set eth1 master bond0
433 ip -d link show dev swp0
434 (...)
435 dsa master bond0
436
437Notice that in the case of CPU ports under a LAG, the use of the
438``IFLA_DSA_CONDUIT`` netlink attribute is not strictly needed, but rather, DSA
439reacts to the ``IFLA_MASTER`` attribute change of its present conduit (``eth0``)
440and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly,
441when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports
442that were assigned to this interface to the first physical DSA conduit which is
443eligible, based on the firmware description (it effectively reverts to the
444startup configuration).
445
446In a setup with more than 2 physical CPU ports, it is therefore possible to mix
447static user to CPU port assignment with LAG between DSA conduits. It is not
448possible to statically assign a user port towards a DSA conduit that has any
449upper interfaces (this includes LAG devices - the conduit must always be the LAG
450in this case).
451
452Live changing of the DSA conduit (and thus CPU port) affinity of a user port is
453permitted, in order to allow dynamic redistribution in response to traffic.
454
455Physical DSA conduits are allowed to join and leave at any time a LAG interface
456used as a DSA conduit; however, DSA will reject a LAG interface as a valid
457candidate for being a DSA conduit unless it has at least one physical DSA conduit
458as a slave device.
1.. SPDX-License-Identifier: GPL-2.0
2
3=======================================
4DSA switch configuration from userspace
5=======================================
6
7The DSA switch configuration is not integrated into the main userspace
8network configuration suites by now and has to be performed manualy.
9
10.. _dsa-config-showcases:
11
12Configuration showcases
13-----------------------
14
15To configure a DSA switch a couple of commands need to be executed. In this
16documentation some common configuration scenarios are handled as showcases:
17
18*single port*
19 Every switch port acts as a different configurable Ethernet port
20
21*bridge*
22 Every switch port is part of one configurable Ethernet bridge
23
24*gateway*
25 Every switch port except one upstream port is part of a configurable
26 Ethernet bridge.
27 The upstream port acts as different configurable Ethernet port.
28
29All configurations are performed with tools from iproute2, which is available
30at https://www.kernel.org/pub/linux/utils/net/iproute2/
31
32Through DSA every port of a switch is handled like a normal linux Ethernet
33interface. The CPU port is the switch port connected to an Ethernet MAC chip.
34The corresponding linux Ethernet interface is called the master interface.
35All other corresponding linux interfaces are called slave interfaces.
36
37The slave interfaces depend on the master interface. They can only brought up,
38when the master interface is up.
39
40In this documentation the following Ethernet interfaces are used:
41
42*eth0*
43 the master interface
44
45*lan1*
46 a slave interface
47
48*lan2*
49 another slave interface
50
51*lan3*
52 a third slave interface
53
54*wan*
55 A slave interface dedicated for upstream traffic
56
57Further Ethernet interfaces can be configured similar.
58The configured IPs and networks are:
59
60*single port*
61 * lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
62 * lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7)
63 * lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11)
64
65*bridge*
66 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
67
68*gateway*
69 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
70 * wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
71
72.. _dsa-tagged-configuration:
73
74Configuration with tagging support
75----------------------------------
76
77The tagging based configuration is desired and supported by the majority of
78DSA switches. These switches are capable to tag incoming and outgoing traffic
79without using a VLAN based configuration.
80
81single port
82~~~~~~~~~~~
83
84.. code-block:: sh
85
86 # configure each interface
87 ip addr add 192.0.2.1/30 dev lan1
88 ip addr add 192.0.2.5/30 dev lan2
89 ip addr add 192.0.2.9/30 dev lan3
90
91 # The master interface needs to be brought up before the slave ports.
92 ip link set eth0 up
93
94 # bring up the slave interfaces
95 ip link set lan1 up
96 ip link set lan2 up
97 ip link set lan3 up
98
99bridge
100~~~~~~
101
102.. code-block:: sh
103
104 # The master interface needs to be brought up before the slave ports.
105 ip link set eth0 up
106
107 # bring up the slave interfaces
108 ip link set lan1 up
109 ip link set lan2 up
110 ip link set lan3 up
111
112 # create bridge
113 ip link add name br0 type bridge
114
115 # add ports to bridge
116 ip link set dev lan1 master br0
117 ip link set dev lan2 master br0
118 ip link set dev lan3 master br0
119
120 # configure the bridge
121 ip addr add 192.0.2.129/25 dev br0
122
123 # bring up the bridge
124 ip link set dev br0 up
125
126gateway
127~~~~~~~
128
129.. code-block:: sh
130
131 # The master interface needs to be brought up before the slave ports.
132 ip link set eth0 up
133
134 # bring up the slave interfaces
135 ip link set wan up
136 ip link set lan1 up
137 ip link set lan2 up
138
139 # configure the upstream port
140 ip addr add 192.0.2.1/30 dev wan
141
142 # create bridge
143 ip link add name br0 type bridge
144
145 # add ports to bridge
146 ip link set dev lan1 master br0
147 ip link set dev lan2 master br0
148
149 # configure the bridge
150 ip addr add 192.0.2.129/25 dev br0
151
152 # bring up the bridge
153 ip link set dev br0 up
154
155.. _dsa-vlan-configuration:
156
157Configuration without tagging support
158-------------------------------------
159
160A minority of switches are not capable to use a taging protocol
161(DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based
162configuration.
163
164single port
165~~~~~~~~~~~
166The configuration can only be set up via VLAN tagging and bridge setup.
167
168.. code-block:: sh
169
170 # tag traffic on CPU port
171 ip link add link eth0 name eth0.1 type vlan id 1
172 ip link add link eth0 name eth0.2 type vlan id 2
173 ip link add link eth0 name eth0.3 type vlan id 3
174
175 # The master interface needs to be brought up before the slave ports.
176 ip link set eth0 up
177 ip link set eth0.1 up
178 ip link set eth0.2 up
179 ip link set eth0.3 up
180
181 # bring up the slave interfaces
182 ip link set lan1 up
183 ip link set lan1 up
184 ip link set lan3 up
185
186 # create bridge
187 ip link add name br0 type bridge
188
189 # activate VLAN filtering
190 ip link set dev br0 type bridge vlan_filtering 1
191
192 # add ports to bridges
193 ip link set dev lan1 master br0
194 ip link set dev lan2 master br0
195 ip link set dev lan3 master br0
196
197 # tag traffic on ports
198 bridge vlan add dev lan1 vid 1 pvid untagged
199 bridge vlan add dev lan2 vid 2 pvid untagged
200 bridge vlan add dev lan3 vid 3 pvid untagged
201
202 # configure the VLANs
203 ip addr add 192.0.2.1/30 dev eth0.1
204 ip addr add 192.0.2.5/30 dev eth0.2
205 ip addr add 192.0.2.9/30 dev eth0.3
206
207 # bring up the bridge devices
208 ip link set br0 up
209
210
211bridge
212~~~~~~
213
214.. code-block:: sh
215
216 # tag traffic on CPU port
217 ip link add link eth0 name eth0.1 type vlan id 1
218
219 # The master interface needs to be brought up before the slave ports.
220 ip link set eth0 up
221 ip link set eth0.1 up
222
223 # bring up the slave interfaces
224 ip link set lan1 up
225 ip link set lan2 up
226 ip link set lan3 up
227
228 # create bridge
229 ip link add name br0 type bridge
230
231 # activate VLAN filtering
232 ip link set dev br0 type bridge vlan_filtering 1
233
234 # add ports to bridge
235 ip link set dev lan1 master br0
236 ip link set dev lan2 master br0
237 ip link set dev lan3 master br0
238 ip link set eth0.1 master br0
239
240 # tag traffic on ports
241 bridge vlan add dev lan1 vid 1 pvid untagged
242 bridge vlan add dev lan2 vid 1 pvid untagged
243 bridge vlan add dev lan3 vid 1 pvid untagged
244
245 # configure the bridge
246 ip addr add 192.0.2.129/25 dev br0
247
248 # bring up the bridge
249 ip link set dev br0 up
250
251gateway
252~~~~~~~
253
254.. code-block:: sh
255
256 # tag traffic on CPU port
257 ip link add link eth0 name eth0.1 type vlan id 1
258 ip link add link eth0 name eth0.2 type vlan id 2
259
260 # The master interface needs to be brought up before the slave ports.
261 ip link set eth0 up
262 ip link set eth0.1 up
263 ip link set eth0.2 up
264
265 # bring up the slave interfaces
266 ip link set wan up
267 ip link set lan1 up
268 ip link set lan2 up
269
270 # create bridge
271 ip link add name br0 type bridge
272
273 # activate VLAN filtering
274 ip link set dev br0 type bridge vlan_filtering 1
275
276 # add ports to bridges
277 ip link set dev wan master br0
278 ip link set eth0.1 master br0
279 ip link set dev lan1 master br0
280 ip link set dev lan2 master br0
281
282 # tag traffic on ports
283 bridge vlan add dev lan1 vid 1 pvid untagged
284 bridge vlan add dev lan2 vid 1 pvid untagged
285 bridge vlan add dev wan vid 2 pvid untagged
286
287 # configure the VLANs
288 ip addr add 192.0.2.1/30 dev eth0.2
289 ip addr add 192.0.2.129/25 dev br0
290
291 # bring up the bridge devices
292 ip link set br0 up