Building a VCF lab with pfSense (Part 1)

VCF, VLC

Part 1: Defining the Architecture

Building a lab running VCF is made easy through the use of the VCF Lab Constructor (VLC). When run in the ‘automated’ mode, VLC will build out a nested environment and deploy VCF in it. It even abstracts a lot of the networking setup to make it as easy as possible. But what if you want to build out the network manually? In this series of posts, I’ll walk you through doing exactly that!

First, why would you want to build out the network manually? There are several reasons, but I think most people would say they just want to learn more about how VCF works. Some may say that they want as a realistic of a deployment as possible. Others may already have some networking for their lab in place and want to utilize it. Whatever the reason, it’s a common ask.

Before we begin, it’s important to understand the big picture of what we want to accomplish. To help visualize everything, please refer to the following diagram:

In this environment, we have a home router that provides access to the Internet. We want to deploy VCF in a network separate from our home network. We also need to provide for all of the networks that are required by VCF.

As we’ll have multiple different networks to route traffic between, we need a router. I’ve talked before about how configure VyOS to work with VCF. This time, I’m going to use another freely available software-based router called pfSense. Based on FreeBSD, this open-source distribution provides routing and firewall functions that is managed through a web interface. You can get the ISO image from here.

From our pfSense router, we will have two ‘physical’ links. One that will connect to our home network and the other that will provide the networks for our VCF installation. To separate the various networks needed for VCF, we will use a series of VLANs:

  • VLAN 10 will be used as our Management network. It will also provide for vMotion and vSAN traffic.
  • VLAN 11 will be used for the first uplink for our Application Virtual Network (AVN)
  • VLAN 12 will be used for the second uplink for our AVN
  • VLAN 13 will be used for our Edge TEP interfaces
  • VLAN 14 will be used for our Host TEP interfaces

For each of the defined networks, the pfSense router will be assigned the first IP address. For example, it will have the IP addresses 10.0.0.1, 172.27.11.1, and so on.

It is also important to note that this environment is hosted on a single physical server, which has ESXi loaded on it (of course). There are two portgroups configured on this host. One of which will be connected to the home network, while the other is a trunked portgroup with a VLAN ID of 4095. This gives us the ability to run all the above VLANs on the single wire.

If we were fortunate enough to have multiple physical servers in a cluster then this would change slightly.  In this case, you would have a DVS and you would simply enable trunking on the portgroup and ensure the VLANs were properly plumbed on the physical switch to each host.  

We’ll have a couple of VMs that will be multi-homed. These are our Windows-based jumphost where we will run VLC from and our AD server. If you were to use VLC in the automated mode, then it would leverage the Cloud Builder appliance to provide DNS services. In our case, we will need to provide those services manually. The AD server provides this service for us and also lets us play with other features, such as user authentication or automated certificate replacement.

One important point to call out with the multi-homed systems, especially the AD server, is to ensure the default gateway is set appropriately on them. Each of these systems is connected to two different networks and you can only set one default gateway. Setting the incorrect gateway can cause issues later. For example, let’s say the jumphost was configured to point to the home router for its default gateway. This would allow the jumphost to access the internet, but as VMs were added to the cross-region or region-specific networks, static routes would have to be added to the jumphost in order to reach these networks. The same problem would occur with the AD server, however as the AD server is also providing DNS services, this can cause other issues. Now once we get pfSense up and running, you should be able to point the gateway on both the jumphost and the AD server to the pfSense router. This would allow for internet access as well as access to the regional networks without having to add any static routes.

Once we start the deployment, we will deploy several VMs on the Management network, including the SDDC Manager, the NSX-T Manager(s), and the Management Domain vCenter. As we will be deploying a AVN configuration, we will also deploy two edges. These edges provide a T0 and T1 router which ultimately gives us access to two additional networks – The cross-region network and the region-specific network. It will be the cross-region network where we will deploy vRSLCM in preparation for deploying the other products in the vRealize Suite (vRA, vROps, ect).

To provide routing to the cross-region and region-specific networks, the edges will communicate over the AVN uplinks to the pfSense router and use BGP for route distribution. In a real environment, these uplinks would cross connect to a pair of top-of-rack switches. This means that we need to configure the pfSense router to support BGP and we also need to define the AS numbers to be used. For our purposes, we will say the pfSense router is AS 65001 and our edges are in AS 65003.

We will also need a few more networking services. With VCF 4.1 we will need a DHCP server. This server automatically allocates the IP addresses for the Host TEP interfaces from a pool that we will configure. We will also need a NTP server. This is important to ensure secure communication between the various components. It also helps when looking at log files, but that’s a story for another article. We will configure the pfSense appliance to provide these services.

At this point, we have identified the networks needed and we also know about the network services we need to configure. Let’s continue this in part two of this series where we will focus on the configuration of pfSense.