Bring Your Own Network with VCF 3.0 Pre-Requisites

Switches, we don’t need no stinkin’ switches…. Well, yes we still need switches.  That being said we no longer require specific switches in VMware Cloud Foundation 3.0.  This article is not a replacement to the User Guide or the Release Notes.

WHAT DOES THAT MEAN???

In VMware Cloud Foundation 3.0 (VCF 3.0), we have moved to a Bring Your Own Network (BYON) model to allow customers to choose their favorite network vendor and switch model and provide the underlay network that VCF will use.  This will allows customers the flexibility to use the fabric manager of choice to control their underlay network.

THIS SOUNDS GREAT! MY NETWORKING GUY WILL BE SO HAPPY! WHAT DO I NEED TO TELL ASK HIM NICELY TO CONFIGURE?

Now that BYON is in place and you have selected your switching vendor of choice, or incorporated to your existing network topology there will be a few pre-requisites that are required from your networking team.  Those pre-requisites are as follows:

  1. Each port to be used by a physical server will need to be a trunk port, however that trunk port can be limited to the specific VLAN’s as required by VCF and your vSphere environment
  2. Creation of a management network to support the SDDC Manager, PSC’s, vCenters, NSX Managers, NSX Controllers, and vRealize LogInsight cluster, in addition to additional vCenters, NSX Managers, and NSX Controllers for additional workload domains, as well as any additional services to be deployed.  This will be the default network used for all of these services when Cloud Builder brings up the environment. (more on that in a later post)
  3. While not explicitly required an ESXi management network is also nice to have assuming it has a route to the above management network.  This way your physical hosts are logically separated on another vLAN in your environment.  Your hosts management (in band) IP’s can also reside on your management network above (optional)
  4. VCF 3.0 introduces a new construct called a Network Pool.  A Network Pool simply consists of two networks ensuring isolation of your vMotion and vSAN traffic in your environment.  In VCF 3.0 we also support multi cluster workload domains, therefore plan accordingly if those should have shared or segregated vMotion and vSAN traffic.
    • vMotion network per Workload Domain, this is due to ensure isolation within workload domains or clusters.  Please keep in mind this is not a requirement but rather a best practice in VCF.
    • vSAN network per Workload domain, this is due to ensure data isolation within workload domains or clusters.  Please keep in mind this is not a requirement but rather a best practice in VCF.
  5. VTEP network with DHCP provided.  Please keep in mind that in VCF 3.0 we configure a Load Based Teaming NIC policy.  This in turn requires two (2) IP addresses per physical host in the environment.
  6. While not explicitly required an network allowing out-of-band access to the BMC ports of the servers is also nice to have assuming it can route to the above management network. This allows you to access the Baseboard Management Controller (iLo, iDrac, IMC, ect) as needed remotely. (optional)
  7. When planning all of your IP requirements allocate a subnet with ample capacity and then use inclusion ranges to limit the use of that subnet.  Keep in mind when using network pools an overlapping subnet cannot be reutilized (i.e. 192.168.5.0/24 and 192.168.5.128/25)
  8. Ensure that your networking guy is not utilizing any ethernet trunking technology (LAG/VPC/LACP) for connection to ESXi servers.
  9. Example of this schema in action:
    As you can see in this example I would be sharing the management network for ESXi servers as well as PSC’s, vCenters, NSX Managers, NSX Controllers, and vRealize LogInsight.  My additional Workload Domain though would receive it’s own dedicated vMotion and vSAN network pool.  Based upon the VXLAN/VTEP DHCP network being a 192.168.5.0/24 I could in theory support up to 126 ESXi hosts in this environment after removing .1 for the gateway and .255 is utilized for broadcast.  Also one other thing to note is the management network can be MTU of 1500 or higher.  Jumbo Frames (1600+), are required for all other networks and is recommended to be 9000 as illustrated in this graphic.  All of the above networks also require gateways, and these need to be routable for the bringup portion of the environment.  Lastly VLAN Tagging through 802.1Q should be utilized, Jumbo Frames enabled, and IGMP Snooping Querier.

I HAVE COMPLETED ALL THAT AND GAVE MY NETWORKING GUY COPIOUS AMOUNTS OF COFFEE, WHAT’S NEXT?

Now that the networking has been plumbed to the environment and host ports are set appropriately the next topic is DNS.  VCF will now support custom naming of attributes, in fact it manages ZERO of the DNS going forward.  In order to ensure your environment is ready for VCF, forward and reverse DNS entries need to be established for every item to be spun up in VCF.  Also one note here is that the DNS server to be used needs to be an authoritative DNS server therefore services like unbound will not allow this to function properly.  Below is a table illustrating what is required in addition to the ESXi Hosts having appropriate forward and reverse DNS:

OK WE HAVE COMPLETED DNS AND ARE READY TO PROCEED WHAT IS NEXT!

In VCF 3.0 the support matrix for physical servers has been dramatically expanded.  In fact we support almost every vSAN ready node from this list.  A couple of things to keep in mind:

  1. Minimum Cache disk size requirement is 200Gb per disk
  2. Minimum Capacity disk size requirement is 1Tb per disk
  3. Minimum Two NIC’s of 10GbE, however VCF 3.0 supports 25GbE and 40GbE or any other NIC that has been qualified for ESXi and is IOVP certified.  This will allow for the use of 100GbE IOVP NIC’s as well.
  4. Configure BIOS and FW levels to VMware and manufacturer recommendations for vSAN (NICs are certified on the ESXi qualifying hardware and HBAs are the most important things to confirm and set before proceeding)
  5. Disable all ethernet ports except the two that will be used in VCF (currently VCF only supports two physical NICs, in addition to BMC, and they must be presented in vSphere as vmnic0 and vmnic1 respectively)
  6. Install a fresh copy of ESXi 6.5U2c/EP8 with a password defined.  Use the vendor build if possible if not patch the VIBs for NICs and disk controllers as stated above.
    * If you are using Dell Servers you are in luck as 6.5U2c/EP8 is directly installable with Dell’s branded ISO located here 
    **If you are using HP Server you are in luck as 6.5U2c/EP8 is directly installable with HP’s branded ISO located here
    *** Here is a link to additional custom ISO’s 
  7. Enable SSH and NTP to start/stop with host
  8. Configure DNS server IP on host.
  9. What you Can (and Cannot) Change in a vSAN Ready Node
  10. In an all flash configuration your hosts capacity SSD’s need to be marked as capacity.  This scrip would be run on each ESX host participating in VCF  Chris Mutchler @chrismutchler wrote this script to automate this:

    ** Please note the Size would need to be updated with the size of your capacity disk

ONE LAST STEP SHOULD I CHOOSE TO CREATE A NEW WORKLOAD DOMAIN OR USE THE NEW MULTI CLUSTER SUPPORT IN VCF 3.0?

In VCF 3.0 as mentioned above, has the ability to instantiate multiple clusters in a single workload domain.  To make this easier in the planning phase, coming soon, one should think if multi cluster or another workload domain makes more sense.  To discuss this I think we need to lay out the features and use cases for each one to compare and contrast.

Multi Cluster:

  1. Hardware
    • GPU’s for VDI
    • Storage dense cluster
    • High Memory cluster
    • Machine Learning is a great example that may need GPU’s for processing
  2. Licensing
    • Segregation of Web vs SQL components
    • Another licensing related component could be OS related, for example a Windows cluster and a Linux cluster (insert your distro of choice).
  3. Failover Zones
    • Ability to create disparate storage for a separate failover zone for an application.
  4. Patch Consistency
    • In multi cluster environment it will allow for patch conformity amongst all participating hosts.
  5. Vendor Requirements
    • Pivotal actually requires multiple clusters

New Workload Domain:

  1. Patching
    • Be able to run patches to test/dev before running them in production
  2. Multi Tenancy
    • Role Based access Control to a separate vCenter
  3. Isolation
    • Full isolation of NSX networks and policies
    • Full isolation of vSAN data controlled by a distinct vCenter server and storage policy based management
  4. Licensing
    • Oracle… Would be a great use case to give them their own vCenter to appease their licensing requests.

In-Rack Imaging for Cloud Foundation

So you have your VMware Cloud Foundation environment humming along and you need to add some additional capacity, how do you add that?  By following the VCF Administrators Guide and the VIA Guide it looks like you need to set this up on a laptop and plug into port 48 on the management switch, however I believe I have an easier option, that doesn’t require you to sit in the datacenter for hours on end.  Let’s look at how to setup In-Rack Imaging.

Pre-requisites:

  1. This applies to VCF 2.2.x and 2.3.x
  2. Download the VCF Bundle ISO as well as the MD5SUM.txt from your entitled MyVMware account
  3. Download the VIA OVA from your entitled MyVMware account
  4. You have completed the Jump VM setup detailed here (coming soon)

Now let’s get started:

  1. Ensure the servers to be imaged have been cabled to the wiremap (document available under downloads for Cloud Foundation)
  2. If Imaging with VCF 2.2: Ensure the servers to be imaged have iDRAC/BMC static IP information set to
    IP Range: 192.168.0.51-192.168.0.99
    Subnet Mask: 255.255.252.0
    Gateway: 192.168.3.254
    **Failure to set these IP addresses will result in a failure during the Add Host wizard later in this process.
  3. Ensure appropriate BIOS settings have been set and PXE boot is enabled on 10g NIC’s
  4. Deploy the VIA OVA in your Management Workload Domain
  5. Ensure vsanDatastore is selected for storage location
  6. Select the vRack-DPortGroup-NonRoutable portgroup
  7. Select defaults on all other screens and click finish
  8. Before booting the VIA Appliance we will need to edit settings of the VM
  9. Map the CD-ROM drive to Datastore ISO
  10. If the system was imaged at the same version it is now, you can point to bundle-iso folder and select the latest version.  If it was imaged at an older version and you want to bring the hosts in at a newer version, upload the ISO and select.
  11. Click OK and then Power On VIA
  12. If you are on version 2.2 or 2.2.1 and have All Flash R730 or R730xd follow this post before proceeding
  13. Now connect to your jump host that has access to the 192.168.0.0/22 and 192.168.100.0/22 subnets in your environment (post coming soon)
  14. Load the VIA interface http://192.168.100.2:8080/via
  15. You should see CD Mounted Successfully, if not click Refresh if VIA was booted before mounting CD-ROM/ISO
  16. Click Browse under ‘Bundle Hash’ Locate your MD5SUM.txt for the respective Imaging version and then click Upload Bundle

  17. Now that the bundle has been uploaded click Activate Bundle
  18. Select the Imaging Tab at the top
  19. Fill in an Appropriate Name in our case EUC Workload Domain
  20. Description is optional
  21. Change Deployment type to ‘Cloud Foundation Individual Deployment’
  22. Ensure Device Type is set to ESXI_SERVER
  23. Under Server ensure the ‘All Flash’ is Selected and the appropriate number of servers to image based upon the quantity you are adding
  24. Ensure the Vendor and Model are correct
  25. The default IP’s should be acceptable as this is considered a temporary range, however if you have two sets of servers you are imaging, make sure they don’t overlap before completing the ‘Add Host’ wizard towards the end of this post.
  26. Confirm all settings are correct before clicking Start Imaging
  27. Click Start Imaging
  28. One thing to note here is that if the servers had ESXi on them before, you will need to manually reboot them to get them to PXE boot.  Otherwise if they are new servers they should be looping through PXE boot
  29. As servers PXE boot, they will show up with a progress bar
  30. Looking at the console of a servers denoted to be in progress we can see that it is Loading ESXi
  31. Clicking on one of the servers denoted to be in progress will show the task list of what has been completed and what is remaining:
  32. Imaging process for all servers will take about 1 hour total
  33. As servers finish Imaging, their progress bar will turn into a green check
  34. Once Imaging has completed it will proceed onto Verify
  35. After which it will go to the Finish task, please wait until this screen shows completed
  36. Now Click the Inventory Tab, and then click download at the end of the Run ID that corresponds to the name given at step 19.  Save this file to location from which has access to the SDDC Manager Interface.  If this is the Jump VM saving on any local drive should be acceptable.
  37. If this is your only imaging operation it would be best practice to Shutdown the VIA appliance as it runs a PXE boot server while in the Imaging process.  Also to note, don’t reboot any other server in the rack during imaging to ensure it isn’t picked up by VIA.

The process of Imaging is now complete, however SDDC Manager does not know about the new inventory.  We will now need to login to your SDDC Manager UI to complete the task

  1. After logging into SDDC Manager click Settings on the left, then Add Host in the ribbon bar.
  2. Select Rack-1 if you are in a single rack configuration, or if you have multi-rack select the appropriate physical location
  3. Click Browse and point to the vcf-imaging-details manifest file that we downloaded above in step 36
  4. Confirm the details and click Add Host
  5. After a few minutes you should see something similar
  6. If the Continue button doesn’t light up blue after a 3 minutes, click Refresh on the browser and something like this should be displayed allowing you to click Continue
  7. Now we see the final steps in Add Host, this will take about 30 minutes to complete depending upon the number of hosts

  8. Once Host bring up completes the following should be displayed, you can click OK at this poing
  9. Clicking on dashboard shows that our environment now has 14 Hosts, when we started with 10 Hosts

VCF Imaging VIA Doesn’t Display My Server!

When going through imaging, or when adding capacity, on Cloud Foundation 2.2 or 2.2.1 you may notice that your server choice is unavailable.

Example: When selecting All Flash, the only Dell option is Dell R630

This is due to the JSON file having all flash set to false.  In order to resolve this and add for example Dell R730 or R730xd as an option we will need to follow a process to make a REST POST to add this as an option.  First, save the below code as: via-manifest-vcf-bundle-2.2.1-7236974.json

I will assume you know how to deploy the VIA appliance, or you can follow my post: coming soon to setup in rack imaging in Cloud Foundation 2.2/2.3 environments.

It is easiest to start with an ’empty’ VIA appliance, given that this is a small VM it would be worthwhile to delete the existing VIA appliance and re-deploy the OVA.  Once VIA has booted for the first time you will point your browser at the VIA interface http://192.168.100.2:8080/via

  1. Ensure that the VCF Bundle ISO has been mounted as the CD-ROM for the VIA VM.
  2. Ensure the MD5SUM.txt file for the specified bundle has been downloaded and is saved.
  3. By saving the above JSON file this will enable the ‘All Flash’ option for Dell R730 and R730xd
    **Important note, by editing this file and adding a server that does not appear on the VCF Hardware Compatibility List this will not allow it to work as the imaging process adds specific VIB’s for these servers.
  4. Now with the ISO mounted, the MD5SUM and JSON files saved on machine with Postman installed (or similar REST Client) we will need to perform the following operation:
    1. URL: http://192.168.100.2:8080/via/bundle/upload
    2. Request Type: POST
    3. Request Body Type: form-data
    4. Parameters:
    5. bundleHashType: <Text> MD5
    6. txtBundleHashFile: <FILE> C:\files\MD5SUM.txt
    7. txtInventoryJson: <FILE> C:\files\via-manifest-vcf-bundle-2.2.1-7236974.json
  5. Click the send button and you should receive a “status”: “Success”, message as seen below.
  6. Switching back to the VIA interface at this point you will see the upload is in progress
  7. Once upload is complete R730/R730xd is now available to image:

NSX 6.1.3/6.1.4 API Changes and Other Fun Registration Knowledge

regnow

While working on a project I discovered that previous powershell/curl and various rest client REST requests that would register the NSX manager with vCenter and SSO server were no longer working.

For example, against NSX 6.1.2 the following code worked fine Returning a 200:

With 6.1.3 and 6.1.4 it would return a 403 error with a cryptic error:

With help from @voltmer we were able to figure out that the returned error was the certificate thumbprint of the vCenter server.  Turns out you need to pass the thumbprint along with the rest of the payload starting with version 6.1.3.  With the above example, it would look like this:

Looking at the API doc’s for NSX this requirement is not noted but this is being addressed.

While I’m at it, there was a additional step required to fully integrate NSX into the WebClient that I didn’t have to do before.  This would be the step of adding a SSO domain user or group and setting a role in NSX.  In vCenter 6.0 if you’ve installed you know that logging in as root the first time get’s you nowhere special.  The administrator@<the sso domain you created on install> has all the power nowadays.   When you register the NSX manager with the vCenter it does not give the user used to register and kind of role within NSX.  When you login to vCenter after registering with the API you can see the Networking and Security Icon, but are unable to see any NSX managers.  Thankfully this is easily rectified by using an additional NSX API call after SSO and vSphere registration:

Make sure you logout of the webclient and back in to be able to see the NSX manager inside of the Networking and Security -> NSX Managers menu.

FYI, the curl in this article will most likely need some modifying.. I “adjusted” it so it would read better, but don’t know if it will run as is.  If you need the original drop me a line.

Hope this helps!

Links of thanks:

@voltmer (fyi, he hasn’t been active on twitter for some time)

Apple Watch Quick Review

Some would call me an Apple Fan Boy, others merely call me Kevin.  While the myriad of Apple devices in my house is symbolic to walking into the Apple Store at your local shopping mall or electronics store of choice, I have to say that I love aesthetically pleasing design.  I remember in high school walking through and seeing rows and rows of the iToys I did have to chuckle:

appleimac1998

 

The computer that I just got at home was screaming fast in comparison to these, plus I had Windows!  I could install my own games browse the internet and we were one of the first residential areas to receive 3mb cable internet!

Now I know what you are thinking what does this have anything to do with the Apple Watch.  Without telling how much I had a disdain for Apple in high school makes it hard to see how I have come full circle since that point.  I have been an Apple Loyalist on the phone platform since the iPhone 3gs, even though I have tried several of the Android offering I found myself rebooting and thinking that I just want my phone to work at the end of the day.  Since this time I preordered the iPad 2, the iPhone 4s, 5s, and 6 Plus as well as picking up an iMac, two Apple TV’s, another iPad Mini, and finally a MacBook Pro.  Seeing the lay of the land in smart watches with the Pebble, Moto 360, and Samsung Gear I  have felt something is missing with all of them, and the lack of iOS support amongst some has left me hanging.  Hearing rumors of an Apple Watch over a year ago left me waiting idle until the new iWhatever would be released.  Also knowing that Apple’s attention to detail would leave me satisfied aesthetically I decided to wait.  That is until Apple announced the  Apple Watch September 9, 2014…

Feverishly I spent the last few hours before preorder on April 10th ensuring my alarm clock was set for 2:45am as I was on Spring Break with my wife and children in Louisville.  I woke up several times before 2:45am figuring that I had slept in past 3:00am and missed the alarm.  Finally the alarm went off and I preordered and received a confirmation email by 3:05am, although bummed that I landed in the May 13-27 shipping window.  While I patiently waited for what I thought was going to be week, I got a preparing to ship on April 27th!  Although I was not in town for its arrival on April 29th I had it sitting there waiting for me when I did make it home.

May 6th 2015:
I arrived homes42sg and tried on my 42mm Space Gray Apple Watch Sport Edition.  It is much lighter than I expected!  Packaging however is top notch in true apple style with a wonderful plastic case to house the watch if you wanted to store it, or carry it in your laptop bag, gym bag, however both of those defeat the purpose.  I was able to very easily and very quickly pair my watch to my phone.  Also quickly able to send/receive text messages and take phone calls on my watch ‘Dick Tracy’ style.  Battery came in about 70% charged and I had the watch on around 2:00pm.  I proceeded to coach my sons soccer game pickup an after soccer dinner and bring it home, put the kids to bed, play with the watch settings and put it on the charger about 11:00pm still with 40% battery remaining.

 

May 7th 2015:
First full day wearing a smart watch and I am not sure what to expect.  This day I needed to run to a few customer sites for appointments and discussions.  I did experience something I was not expecting constant bluetooth issues in the car and in the home office while trying to utilize a bluetooth headset.  While the devices appeared to be paired with the phone, they would constantly revert to using the iPhone as the audio device.  Annoyed with this I plugged in a wired headset and continued on with my day driving and taking conference calls and fussing with this wired headset.  I did find another weird thing this day, iPhone battery dropped to 30% remaining before 4:00pm and that is very unusual based upon my driving, talking, texting, and emailing habits, of course not at the same time.  Again fiddled with some of the settings on the iPhone for the Apple Watch looked for some more Apple Watch enabled apps and hung it on the charger just after 10:15pm with 46% charge remaining.  I was a bit surprised by how much battery life I am getting on the watch and how little I received on my phone.

May 8th 2015:
Annoyed with bluetooth and battery issues I reboot my phone this a.m. before taking the watch off the charger.  This appears to have resolved the bluetooth issues as I was able to accept a phone call ‘Dick Tracy’ style and immediately move it to the bluetooth headset at home without being disappointed.  Relieved with my success of the morning I have also found the battery life to be better on the iPhone as well.  Now that we have the major sticking points resolved or somewhat resolved I am feeling better about my purchase.  Now let’s test some of the apps and notifications!  You think having a smart watch it will make your life easier, however I must say there is an adjustment period where you forget you have a smart watch, feel a tap on your wrist and automatically take out your iPhone to check your notification.  Not sure how this happens but I am adjusting to it.

Text messaging is absolutely rock solid and the Siri integration on this component I feel is possibly better than on the iPhone itself.  I am not sure how this is possible, but it just seems like Watch Siri listens to me better than iPhone Siri.  Other integrations I am finding useful are the notifications for Twitter as well as calendar reminders and phone calls when my phone isn’t right next to me, or say I run upstairs for a cup of coffee and don’t bring my phone.

May 9th 2015:
Today we are busy busy busy with my daughters soccer game, sons karate, daughters birthday, and oh shoot I need a mothers day present.  I setup a list on Clear and let it sync to the Watch so I can stay on task and figured I wouldn’t have to get out my phone.  With 3rd party apps I am seeing a little bit of a delay loading their content, however this is a 1.0 product with 1.0 software I am sure it will be O.K. with a simple software update.  Clear worked fantastic, I was able to update my list in real time after soccer, karate, and picking up those Mother’s Day presents as well as picking up  dinner for my daughter’s birthday party.  Just to be sure I can use this going forward I verified on the iPhone clearing the tasks actually took affect on the other side.  Wonderful this may be a time saver!  Hung the watch up around 11:30pm after moderate use all day with 42% battery remaining.

May 10th 2015:
Today is Mother’s Day and in tradition I take care of the kids all day and we serve breakfast in bed to my wife.  Kids were up at 7:15am so I strapped on the watch and worked with my youngest getting a show on the Apple TV.  The wonderful thing about this is that I was able to browse into Netflix and other movies purchased from iTunes directly from the watch!  While I didn’t have to configure anything and reading further into it, it appears as though the watch can be paired directly so that the phone is not required.  While I haven’t tried this it is good to know it is possible, however I usually use the remote.  I tried a couple more things today on the watch such as the timer so that I could cook perfect over medium eggs for my wife as well as timing my kids screen time.  Family photo time after lunch without a selfie stick or another person to take the picture, not a problem with Apple Watch just open the camera app and use the watch as a view finder and take a picture myself, hidden use case here for sure!

Also tried the App ’10’ this is actually one of the main reasons why I bought the watch, to use it as a remote for a GoPro camera while I am fly fishing.  While I thought this was going to be absolutely fantastic to use as a live view while recording I am a little disappointed with the first iteration of this app and hope that GoPro make their app work with the Apple Watch.  The lag is terrible, about 1-2 seconds, no live view, and mediocre results when hitting record or stopping record.

May 11th 2015:
While I haven’t really run this watch through its paces yet, I am very impressed with the seamless implementation of Apples first wearable.  Again today I was notified of meetings with the same 15 minute window I have come to expect, and haven’t been required to dig my phone out of my pocket.  It is 10:02pm and have had the watch on since 7:03am and I still have 56% battery remaining even after a 15 minute phone call ‘Dick Tracy’ style and checking stocks, email, texts, and Google Voice notifications (Yes I do use this even though it isn’t Apple). I could see this with a few more software iterations and potentially Apple Watch 2 hardware running well into 2-3 days of use.

Pros:
Fantastic battery life considering the size of battery (205mah)
Seamless Integration with the iPhone
Mirroring notification from iPhone is a breeze for setup
Software make this device
Text Messaging
‘Dick Tracy’ style phone calls
Sync last up to 500 photos!
Sync up to 2gb of Music
Pair bluetooth headphones with watch

Cons:
No reply to email function (this would be painful with Siri anyway, but for quick replies, or forwards that would be excellent)
Google Voice – repeat the last description
Slow Loading 3rd party Apps – I am sure this will improve over the next software release
Haven’t found a way for Podcasts to sync over automatically

Conclusion:
As this is the first iteration of the wearable market for Apple and the only shining star being the Pebble it is my opinion the Apple Watch is a shining star.  Although there are a few things in the Cons category (mostly 3rd party apps) I think there is enough in the Pros category to warrant a purchase for anyone that relies heavily on the iPhone as their single information source during the day.  Also this is light enough to wear while exercising, I have only been testing while coaching soccer followed by one on one instructional with mild scrimmage play after games.  While you may read this and think it is just another Apple Fan Boy, I would encourage you to call me Kevin.

Links:

Tying into the vRA theme from prior posts here is an excellent article by coworker and fellow blogger deploying workloads from his Apple Watch:

http://www.vaficionado.com/2015/04/deploying-vrealize-automation-workloads-from-apple-watch/

Remove/Reset NSX configuration in a vRA vCenter Endpoint

A couple months ago I was working with a customer that was in the midst of deploying vRA 6.1 integrated with NSX.  Once they had everything up and running there were having problems with the NSX data collection.  There was an error in the vRA logs stating something about the security groups not being able to be enumerated.. my apologies for not having the exact error handy.

cyanide_and_happiness_any_IT_job_google_troubleshooting

After some additional troubleshooting it was found that the NSX 1.0.1 plugin had been installed on the external vCO server.  Unfortunately, this plugin only supports vRA 6.2 and above:

VMware vCenter Orchestrator Plugin 1.0.1 for NSX
This plug-in can be utilized by vRA 6.2.0, vRO 5.5.2, vRO 6.0.0, vCNS 5.5.2, vCNS 5.5.3.x, vCNS 5.5.4, NSX-vSphere 6.1.0, NSX-vSphere 6.1.1, NSX-vSphere 6.1.2, NSX-vSphere 6.1.3.

Once that was found, the path to rectify went like this:

  • ​Remove the NSX 1.0.1 plugin from vCO
  • Reset the Plugin versions in vCO, this step might not be necessary but just to be safe…
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click the Troubleshooting tab.
    3. Click Reset current version.
  • Restart vCO Server/Config services
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click the Startup Options tab.
    3. Restart both the Server and the configuration server services.
  • Install NSX 1.0.0 plugin
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click on the Plugins tab.
    3. Locate the plugin file and click Upload and Install.
  • Restart vCO Server service
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click the Startup Options tab.
    3. Restart the Server service.
  • Remove/Reset the NSX config info for the vCenter Endpoint
    • This was the most time consuming annoying pieces of this fix…  I loathe form elements that are greyed out when it’s something I need to modify.  With the help of a seasoned colleague we arrived at this task:
        • Clear out all vRA tables of DynamicOps.VCNS.*  So, basically we needed to remove all the data that was collected and then, and only then could we actually remove the endpoint.  This all took short of 40 minutes but was tedious enough that I build a SQL script to tackle the job, *Please read the comments in the script!*:

  • Add the Networking and Security manager to the vCenter endpoint and run a data collection… since we just removed this, obviously you know how to add one 🙂  If not head over to http://dailyhypervisor.com/vmware-nsx-6-1-vcac-6-1-connecting-nsx-to-vcac/ to see how.

Hope this helps someone!

Links of thanks:

http://dailyhypervisor.com

PowerCLI and your VM’s GuestID

1364677641-01-bacon_egg

Happy Easter!

Easter eggs can be dyed in many different colors…  just like the many OS’s you can run in a vSphere VM!  (Which is 93 if you count everything you can set New-VM’s GuestID!)

Interestingly, when you deploy a VM using PowerCLI and don’t specify an OS with GuestID it defaults to Windows XP (32 bit):

PowerCLI-DefaultOS

As I found out this can lead to some challenges.  I was running into a problem where it seemed like the VMtools were failing.  This all started occurring when testing an existing project that was working fine in vSphere 5.5 on a new vSphere 6.0 install.  The VM was being deployed by PowerCLI and mounting a OpenSUSE Live ISO to boot which also had open-vm-tools on it so you can run scripts, copy files, etc…

Lets try running a script against our newly created “JustANewVM” which was created without specifying a GuestID and is running OpenSUSE with open-vm-tools.PowerCLI-RunDefaultScriptTypeWhile red is my favorite color, I don’t like it at all in this context!


The corresponding host’s hostd.log file didn’t shed much light on this either, lines 4 and 5 do let us know something’s gone awry:

Now I think there might be a problem with the host.. so I restart hostd and try again, same result.  Next, I vMotion the VM over to another host and run the same command and get these logs:

A completely different error came back, which validates my assumption about the host… OR NOT!  The error in the PowerCLI windows was different as well, telling me that there was not a PowerShell interpreter that could be run.  Running the same command subsequent times still produced the original error.. Now, why vMotioning it seems to change the verbosity level I do not know?  I was glad it did however, as it provided me a the clue into what was going on!  If you look at the logs you can see it’s trying to run cmd.exe on the VM and pass it a powershell “EncodedCommand”.  Cool, but that’s not going to work on a *nix box!  At this point I started looking at why vSphere thought this was a Windows machine and found that the default behavior of New-VM without -GuestID is to make it a Windows XP 32 bit VM as discussed above.

As I could see that it was listed as a Windows XP VM, I powered it off, set the OS to the correct one (otherLinux64Guest in this case) and rebooted.  When I tried to run the command, it failed yet again!  The host logs showed the “VIX_E_FAIL” error initially seen in the host logs.  I decided to restart hostd on the host where the VM was running and after that the PowerCLI command ran fine.

An additional anomaly that I found was that when the VM was powered on and the tools started the C# (fat, and going away someday) client would update the Guest OS: field on the summary page for the VM to be the correct one.  The web-client however did not.  I believe that the different clients are looking at different information, the C# client is most likely looking at the properties in ExtensionData.Summary.Guest where the Web-Client is looking at the ExtensionData.Config properties.  In this case the C# client is more accurate/dynamic, not sure of the reasoning behind this.

* I removed a bunch of bits from the above that weren’t relevant, but it’s clear that these are different sets of data and the different clients look at one or the other to get the OS info.

I hope this helps someone! (perhaps it will help me when I forget what happened here!)

As a follow on, long long ago Frank Denneman and Alan Renouf posted about the dangers of mismatched OS’s and VM settings and a really cool one liner for finding them here:

http://frankdenneman.nl/2009/12/15/impact-of-mismatch-guest-os-type/

Links of thanks:

http://vtagion.com
http://frankdenneman.nl/
http://www.virtu-al.net/