I’ve talked about how to use hosts that have multiple physical NICs to create NSX-V and NSX-T backed workload domains and even how to expand a cluster in one of these workload domains. But what if you want to do an initial installation of VMware Cloud Foundation (VCF) using hosts with multiple physical NICs?
As I’ve mentioned before, the support for multiple physical NICs with VCF is new with VCF 3.9.1. All of the operations we performed previously relied on the VCF API. This worked well for our intended use, but bringup is a different animal.
During a normal bringup, you would normally upload a Excel spreadsheet containing all the required configuration data to the Cloud Builder appliance. Cloud Builder then converts that Excel document into a JSON formatted file. The resulting JSON file is what actually provides the configuration data to the bringup workflow.
You can skip the conversation process and just upload the JSON file directly to the Cloud Builder appliance. We use this to our advantage today, as the deployment Excel spreadsheet does not currently support the use of multiple physical NICs. First, we need to get a converted JSON file…
There are a couple different ways that you can get a copy of this file. For the users of the VMware Lab Constructor (VLC), you will find two files in the home directory called *default-vcf-ems.json. One is used if you select the ‘Build it for me’ option while the other is used if you select the ‘I will build it’ option. I’ll use the later version for the example today, but the other version would be handled in the same way.
To generate a file of your own, simply fill out the VCF Deployment Spreadsheet as you would normally. Copy the Excel file to a deployed instance of Cloud Builder. Now you want to use the sos utility in /opt/vmware/sddc-support directory to convert the Excel Document to a JSON formatted file. Using sos –help will show you the command arguments that you would use for this.
By default, with the current version of VCF, you will find a single DVS defined under the dvsSpecs section. In our example, this is called ‘SDDC-Dswitch-Private’. Let’s say that we wanted to add another VDS and segment our public traffic from our vSAN and vMotion traffic. We would simply add in another definition here like this:
Notice that we will specify the use of vmnic2 and vmnic3 for this VDS.
This, by itself is not enough, however. We also need to define the network specification for our Public VDS. We can do this by adding in the appropriate lines under the networkSpecs section as shown here:
Once we have these changes made to the JSON file, the only thing left to do is to upload it to the Cloud Builder Appliance and start the bringup workflow, just like you would normally.
After bringup completes, you will be able to look at the networking details in the vSphere Client and see the multiple distributed switches and the portgroups that have been created. You can also see the proper NICs have also been assigned as we defined in the bringup JSON.
A few things to note:
- If you are using VLC to do this, you need to tell it not to do bringup automatically for you. The reason for this is that you need to shutdown the nested ESXi hosts that it deployed and add the additional NICs before you can do bringup.
- You cannot use NSX-T for this at this point as the binaries are not available.
Finally, here is the complete example file I used (sans the license key information).
{
"workflowName":"workflowconfig/workflowspec-ems.json",
"workflowVersion":"3.9.1",
"sddcManagerSpec":{
"secondUserCredentials":{
"userName":"vcf",
"password":"VMware123!"
},
"ipAddress":"10.0.0.4",
"netmask":"255.255.255.0",
"hostname":"sddc-manager",
"rootUserCredentials":{
"userName":"root",
"password":"VMware123!"
},
"restApiCredentials":{
"userName":"admin",
"password":"VMware123!"
},
"vcenterId":"vcenter-1"
},
"sddcId":"SSO-Site1-workflowspec-ems",
"skipEsxThumbprintValidation":true,
"esxLicense":"XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"managementPoolName":"bringup-networkpool",
"ceipEnabled":true,
"ntpSpec":{
"ntpServers":[
"10.0.0.221"
]
},
"dnsSpec":{
"subdomain":"vcf.sddc.local",
"domain":"vcf.sddc.local",
"nameserver":"10.0.0.221"
},
"esxiHostSpecs":[
{
"association":"SDDC-Datacenter1",
"ipAddressPrivate":{
"ipAddress":"10.0.0.100",
"cidr":"10.0.0.0/24",
"gateway":"10.0.0.1"
},
"esxiHostname":"esxi-1",
"esxiCredentials":{
"userName":"root",
"password":"VMware123!"
},
"vSwitch":"vSwitch0",
"serverId":"host-1"
},
{
"association":"SDDC-Datacenter1",
"ipAddressPrivate":{
"ipAddress":"10.0.0.101",
"cidr":"10.0.0.0/24",
"gateway":"10.0.0.1"
},
"esxiHostname":"esxi-2",
"esxiCredentials":{
"userName":"root",
"password":"VMware123!"
},
"vSwitch":"vSwitch0",
"serverId":"host-2"
},
{
"association":"SDDC-Datacenter1",
"ipAddressPrivate":{
"ipAddress":"10.0.0.102",
"cidr":"10.0.0.0/24",
"gateway":"10.0.0.1"
},
"esxiHostname":"esxi-3",
"esxiCredentials":{
"userName":"root",
"password":"VMware123!"
},
"vSwitch":"vSwitch0",
"serverId":"host-3"
},
{
"association":"SDDC-Datacenter1",
"ipAddressPrivate":{
"ipAddress":"10.0.0.103",
"cidr":"10.0.0.0/24",
"gateway":"10.0.0.1"
},
"esxiHostname":"esxi-4",
"esxiCredentials":{
"userName":"root",
"password":"VMware123!"
},
"vSwitch":"vSwitch0",
"serverId":"host-4"
}
],
"vCenterSpecs":[
{
"vcenterIP":"10.0.0.12",
"vcenterHostname":"vcenter-1",
"vcenterId":"vcenter-1",
"licenseFile":"XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"vmSize":"small",
"rootVcenterPassword":"VMware123!"
}
],
"pscSpecs":[
{
"pscId":"psc-1",
"vcenterId":"vcenter-1",
"pscIp":"10.0.0.11",
"pscHostName":"psc-1",
"rootPscPassword":"VMware123!",
"embeddedPsc":false,
"adminUserSsoPassword":"VMware123!",
"pscSsoSpec":{
"ssoDomain":"vsphere.local",
"isJoinSsoDomain":false,
"ssoSiteName":"SSO-Site1"
}
},
{
"pscId":"psc-2",
"vcenterId":"vcenter-1",
"pscIp":"10.0.0.14",
"pscHostName":"psc-2",
"embeddedPsc":false,
"rootPscPassword":"VMware123!",
"adminUserSsoPassword":"VMware123!",
"pscSsoSpec":{
"ssoDomain":"vsphere.local",
"isJoinSsoDomain":false,
"ssoSiteName":"SSO-Site1",
"replicationPartner":"psc-1"
}
}
],
"clusterSpecs":[
{
"clusterId":"SDDC-Cluster1",
"vcenterId":"vcenter-1",
"clusterEvcMode":"",
"resourcePoolSpecs":[
{
"name":"Mgmt-ResourcePool",
"type":"management",
"cpuReservationPercentage":0,
"cpuLimit":-1,
"cpuReservationExpandable":true,
"cpuSharesLevel":"high",
"cpuSharesValue":0,
"memoryReservationMb":16384,
"memoryLimit":-1,
"memoryReservationExpandable":true,
"memorySharesLevel":"normal",
"memorySharesValue":0
},
{
"name":"Edge-ResourcePool",
"type":"network",
"cpuReservationPercentage":0,
"cpuLimit":-1,
"cpuReservationExpandable":true,
"cpuSharesLevel":"normal",
"cpuSharesValue":0,
"memoryReservationPercentage":0,
"memoryLimit":-1,
"memoryReservationExpandable":true,
"memorySharesLevel":"normal",
"memorySharesValue":0
},
{
"name":"User-Edge-ResourcePool",
"type":"compute",
"cpuReservationPercentage":0,
"cpuLimit":-1,
"cpuReservationExpandable":true,
"cpuSharesLevel":"normal",
"cpuSharesValue":0,
"memoryReservationPercentage":0,
"memoryLimit":-1,
"memoryReservationExpandable":true,
"memorySharesLevel":"normal",
"memorySharesValue":0
},
{
"name":"User-VM-ResourcePool",
"type":"compute",
"cpuReservationPercentage":0,
"cpuLimit":-1,
"cpuReservationExpandable":true,
"cpuSharesLevel":"normal",
"cpuSharesValue":0,
"memoryReservationPercentage":0,
"memoryLimit":-1,
"memoryReservationExpandable":true,
"memorySharesLevel":"normal",
"memorySharesValue":0
}
]
}
],
"vsanSpecs":[
{
"vsanId":"vsan-1",
"vcenterId":"vcenter-1",
"licenseFile":"XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"vsanDedup":"false",
"datastoreName":"VCF-VSAN"
}
],
"dvsSpecs":[
{
"dvsId":"SDDC-Dswitch-Private",
"vcenterId":"vcenter-1",
"vmnics":[
"vmnic0",
"vmnic1"
],
"mtu":8940,
"networks":[
"MANAGEMENT",
"VMOTION",
"VSAN",
"UPLINK01",
"UPLINK02"
],
"niocSpec":[
{
"trafficType":"VSAN",
"value":"HIGH"
},
{
"trafficType":"VMOTION",
"value":"LOW"
},
{
"trafficType":"VDP",
"value":"LOW"
},
{
"trafficType":"VIRTUALMACHINE",
"value":"HIGH"
},
{
"trafficType":"MANAGEMENT",
"value":"NORMAL"
},
{
"trafficType":"NFS",
"value":"LOW"
},
{
"trafficType":"HBR",
"value":"LOW"
},
{
"trafficType":"FAULTTOLERANCE",
"value":"LOW"
},
{
"trafficType":"ISCSI",
"value":"LOW"
}
]
},
{
"dvsId":"SDDC-Dswitch-Public",
"vcenterId":"vcenter-1",
"vmnics":[
"vmnic2",
"vmnic3"
],
"mtu":8940,
"networks":[
"PUBLIC"
]
}
],
"networkSpecs":[
{
"networkType":"PUBLIC",
"subnet":"10.0.9.0/24",
"gateway":"10.0.9.1",
"VLANId":"0",
"mtu":"1500",
"portGroupKey":"SDDC-DPortGroup-PUBLIC",
"standbyUplinks":[
],
"activeUplinks":[
"uplink1",
"uplink2"
]
},
{
"networkType":"MANAGEMENT",
"subnet":"10.0.0.0/24",
"gateway":"10.0.0.1",
"VLANId":"0",
"mtu":"1500",
"portGroupKey":"SDDC-DPortGroup-Mgmt",
"standbyUplinks":[
],
"activeUplinks":[
"uplink1",
"uplink2"
]
},
{
"networkType":"VMOTION",
"subnet":"10.0.8.0/24",
"gateway":"10.0.8.1",
"VLANId":"0",
"mtu":"8940",
"portGroupKey":"SDDC-DPortGroup-vMotion",
"association":"SDDC-Datacenter1",
"includeIpAddressRanges":[
{
"endIpAddress":"10.0.8.120",
"startIpAddress":"10.0.8.100"
}
],
"standbyUplinks":[
],
"activeUplinks":[
"uplink1",
"uplink2"
]
},
{
"networkType":"VSAN",
"subnet":"10.0.4.0/24",
"gateway":"10.0.4.1",
"VLANId":"0",
"mtu":"8940",
"portGroupKey":"SDDC-DPortGroup-VSAN",
"includeIpAddressRanges":[
{
"endIpAddress":"10.0.4.120",
"startIpAddress":"10.0.4.100"
}
],
"standbyUplinks":[
],
"activeUplinks":[
"uplink1",
"uplink2"
]
},
{
"networkType":"UPLINK01",
"subnet":"172.27.11.0/24",
"gateway":"172.27.11.1",
"VLANId":"0",
"mtu":"1500",
"portGroupKey":"mgmt01-uplink01",
"association":"SDDC-Datacenter1",
"standbyUplinks":[
]
},
{
"networkType":"UPLINK02",
"subnet":"172.27.12.0/24",
"gateway":"172.27.12.1",
"VLANId":"0",
"mtu":"1500",
"portGroupKey":"mgmt01-uplink02",
"association":"SDDC-Datacenter1",
"standbyUplinks":[
]
},
{
"networkType":"UDLR",
"subnet":"192.168.100.0/24",
"mtu":"9000",
"association":"SDDC-Datacenter1",
"VLANId":"0"
},
{
"networkType":"X_REGION",
"subnet":"192.168.11.0/24",
"gateway":"192.168.11.1",
"mtu":"9000",
"association":"SDDC-Datacenter1",
"VLANId":"0"
},
{
"networkType":"REGION_SPECIFIC",
"subnet":"192.168.31.0/24",
"gateway":"192.168.31.1",
"mtu":"9000",
"association":"SDDC-Datacenter1",
"VLANId":"0"
}
],
"nsxSpecs":[
{
"nsxId":"nsxmanager",
"vcenterId":"vcenter-1",
"nsxManagerIp":"10.0.0.13",
"nsxManagerHostname":"nsxmanager",
"nsxSegmentRangeBegin":"5000",
"nsxSegmentRangeEnd":"5200",
"multicastRangeBegin":"239.1.0.0",
"multicastRangeEnd":"239.1.255.255",
"universalSegmentRangeBegin":"30000",
"universalSegmentRangeEnd":"32000",
"universalMulticastRangeBegin":"239.2.0.0",
"universalMulticastRangeEnd":"239.2.255.255",
"license":"XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"adminNsxManagerPassword":"VMware123!",
"privilegedNsxManagerPassword":"VMware123!",
"adminNsxControllerPassword":"VMware123!VMware123!",
"nsxVXLANSpec":{
"VXLANVLANId":"0",
"VXLANMtu":"8940",
"subnetWithCidr":"",
"gateway":"",
"VXLANCluster":"SDDC-Cluster1",
"VXLANDvs":"SDDC-Dswitch-Private",
"poolStartAddress":"",
"poolEndAddress":"",
"teamingPolicy":"LOADBALANCE_SRCID",
"vtepVmknicCount":2
},
"controllerIPPoolSpec":{
"poolName":"SSO-Site1-mgmt01-nsxc01",
"startIp":"10.0.0.40",
"endIp":"10.0.0.42",
"prefix":"24",
"defaultGateway":"10.0.0.1"
},
"nsxEdgePassword":"VMware123!VMware123!",
"nsxEdgeSpecs":[
{
"edgeName":"esg-01",
"edgeSize":"compact",
"edgeType":"ESG",
"vnics":[
{
"name":"Uplink01",
"type":"uplink",
"networkType":"UPLINK01",
"primaryIpAddress":"172.27.11.2",
"useForDynamicRouterConfiguration":true
},
{
"name":"Uplink02",
"type":"uplink",
"networkType":"UPLINK02",
"primaryIpAddress":"172.27.12.2"
},
{
"type":"internal",
"networkType":"UDLR",
"attachToLogicalSwitch":true,
"primaryIpAddress":"192.168.100.1"
}
]
},
{
"edgeName":"esg-02",
"edgeSize":"compact",
"edgeType":"ESG",
"vnics":[
{
"name":"Uplink01",
"type":"uplink",
"networkType":"UPLINK01",
"primaryIpAddress":"172.27.11.3"
},
{
"name":"Uplink02",
"type":"uplink",
"networkType":"UPLINK02",
"primaryIpAddress":"172.27.12.3",
"useForDynamicRouterConfiguration":true
},
{
"type":"internal",
"networkType":"UDLR",
"attachToLogicalSwitch":true,
"primaryIpAddress":"192.168.100.2"
}
]
},
{
"edgeName":"udlr-1",
"edgeSize":"compact",
"edgeType":"UDLR",
"vnics":[
{
"name":"Uplink",
"type":"uplink",
"attachToLogicalSwitch":true,
"useForDynamicRouterConfiguration":true,
"networkType":"UDLR",
"primaryIpAddress":"192.168.100.3"
},
{
"type":"internal",
"attachToLogicalSwitch":true,
"networkType":"X_REGION"
},
{
"type":"internal",
"attachToLogicalSwitch":true,
"networkType":"REGION_SPECIFIC"
}
]
}
],
"nsxBgpSpec":{
"localAutonomousSystem":65003,
"defaultPassword":"",
"skipBgpValidation":false,
"nsxEdgeBgpSpecs":[
{
"edgeType":"ESG",
"neighbours":[
{
"neighbourIp":"172.27.11.1",
"autonomousSystem":65001,
"password":""
},
{
"neighbourIp":"172.27.12.1",
"autonomousSystem":65001,
"password":""
},
{
"neighbourIp":"192.168.100.4",
"keepAliveTimeSeconds":1,
"holdDownTimeSeconds":3
}
]
},
{
"edgeType":"UDLR",
"neighbours":[
{
"neighbourIp":"192.168.100.1",
"forwardingIp":"192.168.100.3",
"protocolIp":"192.168.100.4"
},
{
"neighbourIp":"192.168.100.2",
"forwardingIp":"192.168.100.3",
"protocolIp":"192.168.100.4"
}
]
}
]
},
"logicalSwitches":[
{
"name":"Universal Transit Network",
"controlPlaneMode":"HYBRID_MODE",
"networkType":"UDLR",
"transportZoneId":"universal-transport-zone"
},
{
"name":"Mgmt-xRegion01-VXLAN",
"controlPlaneMode":"HYBRID_MODE",
"networkType":"X_REGION",
"transportZoneId":"universal-transport-zone"
},
{
"name":"Mgmt-RegionA01-VXLAN",
"controlPlaneMode":"HYBRID_MODE",
"networkType":"REGION_SPECIFIC",
"transportZoneId":"universal-transport-zone"
}
],
"transportZones":[
{
"id":"universal-transport-zone",
"name":"Mgmt Universal Transport Zone",
"controlPlaneMode":"HYBRID_MODE",
"universal":true
}
],
"transportZoneName":"transportzone-1"
}
],
"logInsightSpecs":[
{
"vcenterId":"vcenter-1",
"adminPassword":"VMware123!",
"sshPassword":"VMware123!",
"loadBalancerHostname":"vrli",
"loadBalancerIpAddress":"192.168.31.10",
"vmSize":"xsmall",
"networkType":"REGION_SPECIFIC",
"license":"XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"logInsightNodeSpecs":[
{
"ipAddress":"192.168.31.11",
"hostname":"loginsight-node-1"
},
{
"ipAddress":"192.168.31.12",
"hostname":"loginsight-node-2"
},
{
"ipAddress":"192.168.31.13",
"hostname":"loginsight-node-3"
}
]
}
]
}Have fun!