With the release of VCF 3.9.1, new VCF deployments require Border Gateway Protocol (BGP) be configured and functioning prior to initiating the VCF deployment. Using BGP simplifies the management of routes to the Application Virtual Networks (AVNs) deployed by VCF.  

VMware Lab Constructor (VLC) provides an option to automatically configure the deployed Cloud Builder appliance to provide BGP services. But what if you want to provide your own infrastructure for your home lab?  In this case, you have to provide BGP services within your environment. One method to do this is to use a software-based router, such as VyOS. Today, I’ll walk through how to configure VyOS to provide these services in your home lab.

Before I get to the configuration, let’s take a moment and look at the environment and define what we want to do with it.

For my home lab, I have a router that connects to the Internet. Anything that connects to the internet will go through this router. This router also provides DHCP services so that anything I connect to it will be given an IP and allowed access to the internet.

The 192.168.0.0/24 network is used for all the systems connected to the internet router. I have a Windows based system that I call my ‘jump host’. This system is where I will use the VMware Lab Constructor (VLC) to help me deploy VCF in a nested configuration.

Prior to VCF 3.9.1, I simply configured my jump host to be multi-homed, meaning that I had two network adapters configured on the system. One was connected to my public 192.168.0.0/24 network while the other was connected to a 10.0.0.0/24 network. The 10.0.0.0/24 network was where I would deploy VCF to.

This configuration worked to completely isolate the environment, as the only way to get to the internet from the 10.0.0.0/24 network was through the jump host. The disadvantage to this is that it works to completely isolate the environment. As mentioned earlier, with VCF 3.9.1, BGP is required. I needed something that could provide for multiple networks and the routing between them, support VLAN tagging and BGP, and could provide some kind of firewall services. I would also prefer that I didn’t have to buy more hardware.

Enter VyOS. VyOS is an open source network operating system that functions as router and firewall. There are several software-based routers on the market, like pfssense. I just happened to pick VyOS to use.

With VyOS, I would connect one leg to the public 192.168.0.0/24 network and I would add additional legs to other networks (such as my 10.0.0.0/24 network) as needed. The jump host would still be multi-homed, as it would allow me to troubleshoot any VyOS related issues.

The first step is to obtain VyOS. You can read about how to obtain a copy of it here. I am using version 1.2.1. This is important. People have had issues configuring VyOS for VCF when using other builds of VyOS. I’m not saying that later versions will not work, but if you get a bleeding edge development version understand that it might have bugs.

Once you have obtained a copy, go ahead and install it. Of course, I deployed it as a virtual machine. I added several network adapters to the VyOS virtual machine and connected them to a couple of portgroups. Referring to the image below, the VM Network portgroup connected to Network Adapter 4 is connected to my public 192.168.0.0/24 network. The other three Network Adapters are all connected to the same portgroup tr-pg-01.

The tr-pg-01 portgroup is a trunked portgroup. This means the portgroup has a VLAN ID set to 4095 and it allows the use of multiple VLAN IDs to use it. Also note the security policy settings for the portgroup. These are specifically set to allow the VMware Lab Constructor to function properly.

Once you have VyOS connected and online, you can begin configuring it. The first configuration action would be to configure the network interfaces. There are a couple different ways you may want to configure the network interfaces. One method is to define a separate virtual sub-interface or VIF in VyOS for each of the networks you will configure. Each VIF can be assigned a unique VLAN ID. For each of the networks required by VCF, you could configure a VIF specific to that network. For example, the Management network would have a VLAN ID of 200 and be configured as one VIF. Likewise, the vMotion and vSAN networks would have their own VLAN IDs and VIFs.

The other option is just to use VLAN 0 for the networks and simply configure a single network interface with multiple IP addresses.

The screenshot below shows how an interface configuration in VyOS would look when using both a VIF and a virtual IP. Notice how the eth0 interface has two IP addresses (172.16.0.1 and 10.0.0.1) assigned to it. These interfaces would both use VLAN ID 0.

You will also notice how a VIF is defined for the 172.27.13.0/24 network. The number 13 represents the VLAN ID that will need to be used to communicate over that network.

Which method to use is up to you. Setting all the VLAN IDs to 0 will work with VCF 3.9.1. However, using VIFs or separate network interfaces provides some flexibility in the future. In my case, I’m going to use a blend of the two methods.

I’ll start by configuring the interface that will be connected to my public network. To do this, simply log into VyOS and enter configuration mode and setting the IP address like this:

$ configure
# set interfaces ethernet eth3 address '192.168.0.2/24'
# set interfaces ethernet eth3 description 'Public'

In my configuration, I’m going to dedicate a network interface to each AVN uplink. I’m also going to use a VIF to specify a specific VLAN ID to be used for these networks. In this case, I would use the following commands:

# set interfaces ethernet eth1 description 'Uplink 1'
# set interfaces ethernet eth1 vif 11 address '172.27.11.1/24'

# set interfaces ethernet eth2 description 'Uplink 2'
# set interfaces ethernet eth2 vif 12 address '172.27.12.1/24'

I’ll configure the remaining network interface with two IP addresses and a vif for another network I’ll use:

# set interfaces ethernet eth0 address '172.16.0.1/24'
# set interfaces ethernet eth0 address '10.0.0.1/24'
# set interfaces ethernet eth0 description 'Trunked PG'

# set interfaces ethernet eth0 vif 13 address '172.27.13.1/24'

When I configure the eth0 interface, I am going to ensure that the first IP address configured is for the network that will be used for the DHCP service that I’ll configure on VyOS. DHCP is required to provide the IP addresses to the VTEPs for VXLAN that VCF deploys. To do this, I would use the following commands:

# set service dhcp-server shared-network-name DHCP-VTEP authoritative
# set service dhcp-server shared-network-name DHCP-VTEP subnet 172.16.0.0/24 default-router '172.16.0.1'
# set service dhcp-server shared-network-name DHCP-VTEP subnet 172.16.0.0/24 dns-server '172.16.0.1'
# set service dhcp-server shared-network-name DHCP-VTEP subnet 172.16.0.0/24 lease '86400'
# set service dhcp-server shared-network-name DHCP-VTEP subnet 172.16.0.0/24 range 0 start '172.16.0.100'
# set service dhcp-server shared-network-name DHCP-VTEP subnet 172.16.0.0/24 range 0 stop '172.16.0.199'

Earlier I mentioned how my internet router also provided DHCP services to allow anyone to connect to the internet. This can cause issues, as I don’t want the DHCP server on VyOS to start handing out IP addresses on my public network or vice versa. To prevent this, I’ll configure a firewall within VyOS and have it drop any DHCP requests:

# set firewall name eth3-local default-action 'accept'
# set firewall name eth3-local rule 10 action 'drop'
# set firewall name eth3-local rule 10 description 'Drop DHCP requests'
# set firewall name eth3-local rule 10 destination port '67'
# set firewall name eth3-local rule 10 protocol 'udp'

Once that is completed, I’ll apply that firewall rule to the eth3 interface, which is connected to the public network:

# set interfaces ethernet eth3 firewall local name 'eth3-local'

Now that the firewall is in place, I’ll configure IP masquerading on the eth3 interface. This will allow anything that is on the 10.0.0.0/24 network to access the internet by using the single IP address configured on that network for the public interface. The 10.0.0.0/24 network is what will be my VCF Management network and will be where all the core infrastructure components (like the SDDC Manager, the NSX Managers, vCenter, ect) are deployed to. As a result, this will allow all of them to connect directly to the internet to perform actions, like getting software updates. (But I’ll still have my own private VCF software depot configured, mainly for speed…)

# set nat source rule 50 outbound-interface 'eth3'
# set nat source rule 50 translation address 'masquerade'
# set nat source rule 100 outbound-interface 'eth3'
# set nat source rule 100 source address '10.0.0.0/24'
# set nat source rule 100 translation address 'masquerade'

Lastly, I will configure BGP. I’ll set the VyOS router as AS 65001 and when I deploy VCF, I’ll configure it to use AS 65003. There will be two Edge Service Gateways (ESG) deployed with VCF. I’ll configure VCF to use two different networks (172.27.11.x and 172.27.12.x) for each uplink. I don’t need an entire class C network, but since this is just a lab, I’m not worried about it.

Given the above, I’ll use a total of four IP addresses, two each, for each ESG. Thus 172.27.11.2 and 172.27.12.2 will be configured on the first ESG and 172.27.11.3 and 172.27.12.3 will be configured on the second ESG. These four IP addresses will be the BGP neighbors that the VyOS router will talk to. I will also configure a password to be used. This is not mandatory with VCF 3.9.1, but it’s a good practice. The VyOS BGP configuration would then be configured by use of the following commands:

# set protocols bgp 65001 neighbor 172.27.11.2 password 'VMware123!'
# set protocols bgp 65001 neighbor 172.27.11.2 remote-as '65003'
# set protocols bgp 65001 neighbor 172.27.11.3 password 'VMware123!'
# set protocols bgp 65001 neighbor 172.27.11.3 remote-as '65003'
# set protocols bgp 65001 neighbor 172.27.12.2 password 'VMware123!'
# set protocols bgp 65001 neighbor 172.27.12.2 remote-as '65003'
# set protocols bgp 65001 neighbor 172.27.12.3 password 'VMware123!'
# set protocols bgp 65001 neighbor 172.27.12.3 remote-as '65003'

There is one more critical configuration command I need to add. This is to setup BGP to advertise the 10.0.0.0/24 network. If you do not do this, your VCF deployment will fail as it will not be able to route from the management network (the 10.0.0.0/24 network) to the uplinks (172.27.12.x/24). To do this, just add in this command:

# set protocols bgp 65001 address-family ipv4-unicast network 10.0.0.0/24

At this point, the VyOS configuration is complete. Just make sure you perform the following commands to commit and save your configuration:

# commit
# save

Nothing left to do but fire up VMware Lab Constructor and deploy VMware Cloud Foundation!