Bridged VLANs with NetworkManager
24 January 2023 Leave a comment
This setup is for my libvirt/KVM virtualization environment. There are lots of tutorials out there but none that showed exactly what I wanted nor how I wanted to accomplish it (ie by using non-interactive CLI tools). I wanted to configure my virtual host with one NIC for management traffic (ie SSH to the host) and a second NIC for all guest traffic. The second NIC would not have any IP addresses on it (on the host) so as to not provide an attack surface to the other networks. I have several VLANs and need to run VMs on each of them. The switch port for the second NIC is set up as a trunk where all the desired VLANs are sent as tagged frames to the virtual host’s NIC. Jumbo frames (MTU) is also set to 9000 on the switch and router.
The following are the VLANs we will be using:
| VLAN | Use |
| 10 | Internal network |
| 20 | Lab network |
| 50 | Work network |
Assumptions
I am running on RHEL 9 for this example, but these commands also work on RHEL 8 and RHEL 7. libvirtd is already running.
I have also installed a special NetworkManager configuration that does not get installed with the Base OS. From the NetworkManager-config-server RPM’s description:
This adds a NetworkManager configuration file to make it behave more like the old “network” service. In particular, it stops NetworkManager from automatically running DHCP on unconfigured ethernet devices, and allows connections with static IP addresses to be brought up even on ethernet devices with no carrier.
This package is intended to be installed by default for server deployments.
Fromrpm -qi NetworkManager-config-server
dnf install NetworkManager-config-server
systemctl restart NetworkManager.servicePreparation
This is the output of nmcli on a freshly installed system. virbr0 is the NAT network created by default during the libvirt installation.
[root@host ~]# nmcli
enp2s0f0: connected to enp2s0f0
"Intel 82576"
ethernet (igb), 00:25:90:AA:AA:AA, hw, mtu 9000
ip4 default, ip6 default
inet4 192.0.2.20/24
route4 default via 192.0.2.1 metric 100
route4 192.0.2.0/24 metric 100
inet6 2001:db8:225:90ff:feaa:aaaa/64
inet6 fe80::225:90ff:feaa:aaaa/64
route6 2001:db8::/64 metric 100
route6 fe80::/64 metric 1024
route6 default via fe80::b6fb:a1ff:fe1a:0101 metric 100
virbr0: connected (externally) to virbr0
"virbr0"
bridge, 52:54:00:AA:B8:F8, sw, mtu 1500
inet4 192.168.122.1/24
route4 192.168.122.0/24 metric 0
enp2s0f1: connected to enp2s0f1
"Intel 82576"
ethernet (igb), 00:25:90:AA:AA:AB, hw, mtu 1500
lo: unmanaged
"lo"
loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
DNS configuration:
servers: 1.1.1.1 8.8.8.8
interface: enp2s0f0
servers: 2606:4700:4700::1111 2001:4860:4860::8888
interface: enp2s0f0
Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.
Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.Configuration
Now I am kind of partial to naming NetworkManager connections after their interface and I also prefer to use the old brN bridge interface naming, so the following example is not going to be terribly descriptive. You can always change the values of con-name and ifname in the following commands.
First we are going to remove the configuration for the second NIC and then add it back making sure to not set any IP addresses. Then we will add the bridge and the VLAN, assign the VLAN to the bridge, and bring up the VLAN.
nmcli con del enp2s0f1
nmcli con add type ethernet con-name enp2s0f1 ifname enp2s0f1 ipv4.method disabled ipv6.method disabled 802-3-ethernet.mtu 9000
nmcli con add type bridge con-name br10 ifname br10 ipv4.method disabled ipv6.method disabled 802-3-ethernet.mtu 9000
nmcli con add type vlan con-name enp2s0f1.10 dev enp2s0f1 id 10 ipv4.method disabled ipv6.method disabled
nmcli con mod enp2s0f1.10 master br10
nmcli con up enp2s0f1.10
nmcli con add type bridge con-name br20 ifname br20 ipv4.method disabled ipv6.method disabled 802-3-ethernet.mtu 9000
nmcli con add type vlan con-name enp2s0f1.20 dev enp2s0f1 id 20 ipv4.method disabled ipv6.method disabled
nmcli con mod enp2s0f1.20 master br20
nmcli con up enp2s0f1.20
nmcli con add type bridge con-name br50 ifname br50 ipv4.method disabled ipv6.method disabled 802-3-ethernet.mtu 9000
nmcli con add type vlan con-name enp2s0f1.50 dev enp2s0f1 id 50 ipv4.method disabled ipv6.method disabled
nmcli con mod enp2s0f1.50 master br50
nmcli con up enp2s0f1.50Completion
This is the output of nmcli after configuration. Notice the lack of IP addresses on the second NIC and the VLAN interfaces.
[root@host ~]# nmcli
enp2s0f0: connected to enp2s0f0
"Intel 82576"
ethernet (igb), 00:25:90:AA:AA:AA, hw, mtu 9000
ip4 default, ip6 default
inet4 192.0.2.20/24
route4 default via 192.0.2.1 metric 100
route4 192.0.2.0/24 metric 100
inet6 2001:db8:225:90ff:feaa:aaaa/64
inet6 fe80::225:90ff:feaa:aaaa/64
route6 2001:db8::/64 metric 100
route6 fe80::/64 metric 1024
route6 default via fe80::b6fb:a1ff:fe1a:0101 metric 100
virbr0: connected (externally) to virbr0
"virbr0"
bridge, 52:54:00:AA:B8:F8, sw, mtu 1500
inet4 192.168.122.1/24
route4 192.168.122.0/24 metric 0
br10: connected to br10
"br10"
bridge, 00:25:90:AA:AA:AB, sw, mtu 9000
br20: connected to br20
"br20"
bridge, 00:25:90:AA:AA:AB, sw, mtu 9000
br50: connected to br50
"br50"
bridge, 00:25:90:AA:AA:AB, sw, mtu 9000
enp2s0f1: connected to enp2s0f1
"Intel 82576"
ethernet (igb), 00:25:90:AA:AA:AB, hw, mtu 9000
enp2s0f1.10: connected to enp2s0f1.10
"enp2s0f1.10"
vlan, 00:25:90:AA:AA:AB, sw, mtu 9000
master br10
enp2s0f1.20: connected to enp2s0f1.20
"enp2s0f1.20"
vlan, 00:25:90:AA:AA:AB, sw, mtu 9000
master br20
enp2s0f1.50: connected to enp2s0f1.50
"enp2s0f1.50"
vlan, 00:25:90:AA:AA:AB, sw, mtu 9000
master br50
lo: unmanaged
"lo"
loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
DNS configuration:
servers: 1.1.1.1 8.8.8.8
interface: enp2s0f0
servers: 2606:4700:4700::1111 2001:4860:4860::8888
interface: enp2s0f0
Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.
Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.
Testing
Useful commands:
bridge link show
bridge vlan global
brctl show
nmcli connection show
nmcli device show
Conclusion
Remember, a bridge runs atop a VLAN and a VLAN runs through the NIC interface. You also have to set up VLAN tagging (trunking) on the switch side of the connection.
Switch (VLAN trunk) -> NIC -> interface -> VLAN -> Bridge
ge/12 -> server port 2 -> enp2s0f1 -> enp2s0f1.20 -> br20Using bridged VLANs means that there is no need for configuring the guest VM to tag the traffic. The guest domain’s libvirt configuration needs to choose the correct bridge to use. The guest just uses the virtual NIC as it is.
The Razor's Edge 