All posts by Ben Sier

NSX 6.1.3/6.1.4 API Changes and Other Fun Registration Knowledge

regnow

While working on a project I discovered that previous powershell/curl and various rest client REST requests that would register the NSX manager with vCenter and SSO server were no longer working.

For example, against NSX 6.1.2 the following code worked fine Returning a 200:

curl -k -u admin:VMware1! -H 'Accept:application/xml' \
-H 'Content-Type:application/xml' \
-X PUT https://10.0.0.80/api/2.0/services/vcconfig \
-d '<vcInfo> \
     <ipAddress>10.0.0.30</ipAddress> \
     <userName>administrator@sierlab.local</userName> \
     <password>VMware1!</password> \
     <assignRoleToUser>true</assignRoleToUser> \
    </vcInfo>'

With 6.1.3 and 6.1.4 it would return a 403 error with a cryptic error:

<?xml version="1.0" encoding="UTF-8"?>
<error>
  <details>92:4D:D6:A4:C2:C2:39:EE:81:11:AA:A9:8D:0D:1F:17:D0:33:C2:C1</details>
  <errorCode>226</errorCode>
</error>

With help from @voltmer we were able to figure out that the returned error was the certificate thumbprint of the vCenter server.  Turns out you need to pass the thumbprint along with the rest of the payload starting with version 6.1.3.  With the above example, it would look like this:

curl -k -u admin:VMware1! -H 'Accept:application/xml' \
-H 'Content-Type:application/xml' \
-X PUT https://10.0.0.80/api/2.0/services/vcconfig \
-d '<vcInfo> \
     <ipAddress>10.0.0.30</ipAddress> \
     <userName>administrator@sierlab.local</userName> \
     <password>VMware1!</password> \
     <assignRoleToUser>true</assignRoleToUser> \
     <certificateThumbprint>92:4D:D6:A4:C2:C2:39:EE:81:11:AA:A9:8D:0D:1F:17:D0:33:C2:C1</certificateThumbprint> \
    </vcInfo>'

Looking at the API doc’s for NSX this requirement is not noted but this is being addressed.

While I’m at it, there was a additional step required to fully integrate NSX into the WebClient that I didn’t have to do before.  This would be the step of adding a SSO domain user or group and setting a role in NSX.  In vCenter 6.0 if you’ve installed you know that logging in as root the first time get’s you nowhere special.  The administrator@<the sso domain you created on install> has all the power nowadays.   When you register the NSX manager with the vCenter it does not give the user used to register and kind of role within NSX.  When you login to vCenter after registering with the API you can see the Networking and Security Icon, but are unable to see any NSX managers.  Thankfully this is easily rectified by using an additional NSX API call after SSO and vSphere registration:

curl -k -u admin:VMware1! -H 'Accept:application/xml' \
-H 'Content-Type:application/xml' \
-X POST https://10.0.0.80/api/2.0/services/usermgmt/role/administrator@sierlab.local??isGroup:false \
-d '<accessControlEntry> \
     <role>super_user</role> \
    </accessControlEntry>'<br>

Make sure you logout of the webclient and back in to be able to see the NSX manager inside of the Networking and Security -> NSX Managers menu.

FYI, the curl in this article will most likely need some modifying.. I “adjusted” it so it would read better, but don’t know if it will run as is.  If you need the original drop me a line.

Hope this helps!

Links of thanks:

@voltmer (fyi, he hasn’t been active on twitter for some time)

Remove/Reset NSX configuration in a vRA vCenter Endpoint

A couple months ago I was working with a customer that was in the midst of deploying vRA 6.1 integrated with NSX.  Once they had everything up and running there were having problems with the NSX data collection.  There was an error in the vRA logs stating something about the security groups not being able to be enumerated.. my apologies for not having the exact error handy.

cyanide_and_happiness_any_IT_job_google_troubleshooting

After some additional troubleshooting it was found that the NSX 1.0.1 plugin had been installed on the external vCO server.  Unfortunately, this plugin only supports vRA 6.2 and above:

VMware vCenter Orchestrator Plugin 1.0.1 for NSX
This plug-in can be utilized by vRA 6.2.0, vRO 5.5.2, vRO 6.0.0, vCNS 5.5.2, vCNS 5.5.3.x, vCNS 5.5.4, NSX-vSphere 6.1.0, NSX-vSphere 6.1.1, NSX-vSphere 6.1.2, NSX-vSphere 6.1.3.

Once that was found, the path to rectify went like this:

  • ​Remove the NSX 1.0.1 plugin from vCO
  • Reset the Plugin versions in vCO, this step might not be necessary but just to be safe…
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click the Troubleshooting tab.
    3. Click Reset current version.
  • Restart vCO Server/Config services
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click the Startup Options tab.
    3. Restart both the Server and the configuration server services.
  • Install NSX 1.0.0 plugin
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click on the Plugins tab.
    3. Locate the plugin file and click Upload and Install.
  • Restart vCO Server service
    1. Log in to the vRealize Orchestrator configuration interface as vmware.
    2. Click the Startup Options tab.
    3. Restart the Server service.
  • Remove/Reset the NSX config info for the vCenter Endpoint
    • This was the most time consuming annoying pieces of this fix…  I loathe form elements that are greyed out when it’s something I need to modify.  With the help of a seasoned colleague we arrived at this task:
        • Clear out all vRA tables of DynamicOps.VCNS.*  So, basically we needed to remove all the data that was collected and then, and only then could we actually remove the endpoint.  This all took short of 40 minutes but was tedious enough that I build a SQL script to tackle the job, *Please read the comments in the script!*:
      /****BE CAREFUL WITH THIS*** 
      To see what tables are populated in the DynamicOps.VCNSModel schema simply change the database name and execute.
      To also clean out all the DynamicOps.VCNSModel tables except VCNSEndpoints, uncomment the last line at the end of the script 
      By: Ben Sier bsier@vmware.com
      Date: 2/20/2014
      Tested against vRA 6.2 / SQL 2012 R2 SP2
      */ 
      
      use vcac;  --May need to change DB name.
      declare c1 cursor for 
      select QUOTENAME(SCHEMA_NAME(schema_id)) + '.' + QUOTENAME(name) from sys.tables where SCHEMA_NAME(schema_id) LIKE 'DynamicOps.VCNSModel' AND name NOT LIKE 'VCNSEndpoints';
      
      declare @ITABLE as nvarchar(500)
      declare @FTABLE as nvarchar(500)
      declare @SQL as nvarchar(200)
      declare @SQL2 as nvarchar(200)
      declare @DelSQL as nvarchar(200)
      declare @EPDelSQL as nvarchar(200)
      set @EPDelSQL = 'delete from [DynamicOps.VCNSModel].[VCNSEndpoints]'
      
      open c1 
      fetch next from c1 into @FTABLE; 
      while (@@FETCH_STATUS =0)
      
      	begin 
              set @SQL = 'declare allVCNSTables cursor for '
              set @SQL2 = 'Select COUNT(*) from ' + @FTABLE
              set @SQL = @SQL + @SQL2
              
              exec sp_executesql @SQL
                      
              open allVCNSTables
              fetch next from allVCNSTables into @ITABLE
              while (@@FETCH_STATUS =0)
      			
      			begin
      				IF @ITABLE > 0 
      				begin
      					PRINT @FTABLE + ' has ' + @ITABLE + ' entries.'
      					PRINT 'Clearing table - ' + @FTABLE
      					set @DelSQL = 'delete from ' + @FTABLE
      					PRINT @DelSQL
      /**** Uncomment the line below to clear out all tables except [DynamicOps.VCNSModel].[VCNSEndpoints] ****/
      					--exec sp_executesql @DelSQL
      				end				
      				fetch next from allVCNSTables into @ITABLE
      			end
              
              close allVCNSTables
              deallocate allVCNSTables   
      	fetch next from c1 into @FTABLE 
      	end 
      close c1 
      deallocate c1
      
      /**** Uncomment the line below to clear out the [DynamicOps.VCNSModel].[VCNSEndpoints] table ****/
      --exec sp_executesql @EPDelSQL
      
      
  • Add the Networking and Security manager to the vCenter endpoint and run a data collection… since we just removed this, obviously you know how to add one 🙂  If not head over to http://dailyhypervisor.com/vmware-nsx-6-1-vcac-6-1-connecting-nsx-to-vcac/ to see how.

Hope this helps someone!

Links of thanks:

http://dailyhypervisor.com

PowerCLI and your VM’s GuestID

1364677641-01-bacon_egg

Happy Easter!

Easter eggs can be dyed in many different colors…  just like the many OS’s you can run in a vSphere VM!  (Which is 93 if you count everything you can set New-VM’s GuestID!)

Interestingly, when you deploy a VM using PowerCLI and don’t specify an OS with GuestID it defaults to Windows XP (32 bit):

PowerCLI-DefaultOS

As I found out this can lead to some challenges.  I was running into a problem where it seemed like the VMtools were failing.  This all started occurring when testing an existing project that was working fine in vSphere 5.5 on a new vSphere 6.0 install.  The VM was being deployed by PowerCLI and mounting a OpenSUSE Live ISO to boot which also had open-vm-tools on it so you can run scripts, copy files, etc…

Lets try running a script against our newly created “JustANewVM” which was created without specifying a GuestID and is running OpenSUSE with open-vm-tools.PowerCLI-RunDefaultScriptTypeWhile red is my favorite color, I don’t like it at all in this context!

invoke-vmscript : 4/7/2015 12:11:42 PM Invoke-VMScript The remote server returned an error: (500) Internal Server Error.
At line:1 char:1
+ invoke-vmscript -VM $new2 -guestuser root -guestpass vmware -ScriptText "ls /etc ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-VMScript], ViError
+ FullyQualifiedErrorId:Client20_VmGuestServiceImpl_DownloadFileFromGuest_DownloadError,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript

The corresponding host’s hostd.log file didn’t shed much light on this either, lines 4 and 5 do let us know something’s gone awry:

2015-04-07T17:42:16.496Z info hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GetGuestFileTransferParameters, token is 5250241e-bbc5-ffa9-cda3-50a00349485d92
2015-04-07T17:42:16.496Z info hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GetGuestFileTransferParameters: vmmoid is 92
2015-04-07T17:42:16.496Z info hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] Guest File Path is /tmp/vmware-root/powerclivmware11
2015-04-07T17:42:16.497Z info hostd[6D9F2B70] [Originator@6876 sub=Libs] HGFileCopy_TransferFileUsingReader: Error, cannot convert /tmp/vmware-root/powerclivmware11 to CPName
2015-04-07T17:42:16.497Z info hostd[6D9F2B70] [Originator@6876 sub=Libs] Vix: [440716 vmGuestOps.cpp:5377]: Error VIX_E_FAIL in CopyFileDoneCallback(): Unknown error
2015-04-07T17:42:16.497Z error hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GuestFileCompleteFunc() job failed with 22
2015-04-07T17:42:16.497Z error hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GuestFileCompleteFunc() throwing InternalServerError

Now I think there might be a problem with the host.. so I restart hostd and try again, same result.  Next, I vMotion the VM over to another host and run the same command and get these logs:

2015-04-07T19:45:57.465Z info hostd[52281B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52088dbc897dbeb9-4ba67b7615492055/411f2455-5c25-d1cc-8ef7-002481fd70f6/JustANewVM.vmx] [N8Guestsvc23StartProgramRequestImplE:0x5232a938] opCode=4 auth=<hidden> programPath=cmd.exe arguments=/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEAdAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsAbABzACAALwBlAHQAYwB9ACcAIAA+ACAAIgAvAHQAbQBwAC8AdgBtAHcAYQByAGUALQByAG8AbwB0AC8AcABvAHcAZQByAGMAbABpAHYAbQB3AGEAcgBlADEANgA1ACIAOwAgAGUAeABpAHQAIAAkAGwAYQBzAHQAZQB4AGkAdABjAG8AZABlAA== failed
2015-04-07T19:45:57.465Z info hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Activation [N5Vmomi10ActivationE:0x5232a8d8] : Invoke done [startProgram] on [vim.vm.guest.ProcessManager:ha-guest-operations-process-manager]
2015-04-07T19:45:57.465Z verbose hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Arg vm:
--> 'vim.VirtualMachine:14'
2015-04-07T19:45:57.465Z verbose hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Arg auth:
--> (vim.vm.guest.NamePasswordAuthentication) {
--> interactiveSession = false,
--> username = "root",
--> password = (not shown)
--> }
2015-04-07T19:45:57.465Z verbose hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Arg spec:
--> (vim.vm.guest.ProcessManager.ProgramSpec) {
--> programPath = "cmd.exe",
--> arguments = "/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEAdAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsAbABzACAALwBlAHQAYwB9ACcAIAA+ACAAIgAvAHQAbQBwAC8AdgBtAHcAYQByAGUALQByAG8AbwB0AC8AcABvAHcAZQByAGMAbABpAHYAbQB3AGEAcgBlADEANgA1ACIAOwAgAGUAeABpAHQAIAAkAGwAYQBzAHQAZQB4AGkAdABjAG8AZABlAA==",
--> workingDirectory = <unset>,
--> }
2015-04-07T19:45:57.466Z info hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.FileNotFound
2015-04-07T19:45:57.466Z info hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Result:
--> (vim.fault.FileNotFound) {
--> faultCause = (vmodl.MethodFault) null,
--> file = "cmd.exe",
--> msg = ""
--> }

A completely different error came back, which validates my assumption about the host… OR NOT!  The error in the PowerCLI windows was different as well, telling me that there was not a PowerShell interpreter that could be run.  Running the same command subsequent times still produced the original error.. Now, why vMotioning it seems to change the verbosity level I do not know?  I was glad it did however, as it provided me a the clue into what was going on!  If you look at the logs you can see it’s trying to run cmd.exe on the VM and pass it a powershell “EncodedCommand”.  Cool, but that’s not going to work on a *nix box!  At this point I started looking at why vSphere thought this was a Windows machine and found that the default behavior of New-VM without -GuestID is to make it a Windows XP 32 bit VM as discussed above.

As I could see that it was listed as a Windows XP VM, I powered it off, set the OS to the correct one (otherLinux64Guest in this case) and rebooted.  When I tried to run the command, it failed yet again!  The host logs showed the “VIX_E_FAIL” error initially seen in the host logs.  I decided to restart hostd on the host where the VM was running and after that the PowerCLI command ran fine.

An additional anomaly that I found was that when the VM was powered on and the tools started the C# (fat, and going away someday) client would update the Guest OS: field on the summary page for the VM to be the correct one.  The web-client however did not.  I believe that the different clients are looking at different information, the C# client is most likely looking at the properties in ExtensionData.Summary.Guest where the Web-Client is looking at the ExtensionData.Config properties.  In this case the C# client is more accurate/dynamic, not sure of the reasoning behind this.

PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> $newVM.ExtensionData.Config

ChangeVersion : 2015-03-26T19:59:59.952422Z
Modified : 1/1/1970 12:00:00 AM
Name :&nbsp;JustANewVM
GuestFullName : Microsoft Windows XP Professional (32-bit)
GuestId : winXPProGuest
PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> $newVM.ExtensionData.Summary.Guest

GuestId : otherGuest
GuestFullName : Linux 3.11.10-25-default openSUSE 13.1 (x86_64)
ToolsStatus : toolsOk
ToolsVersionStatus : guestToolsUnmanaged
ToolsVersionStatus2 : guestToolsUnmanaged
ToolsRunningStatus : guestToolsRunning
HostName : linux-e2zm
IpAddress : 192.168.2.70

* I removed a bunch of bits from the above that weren’t relevant, but it’s clear that these are different sets of data and the different clients look at one or the other to get the OS info.

I hope this helps someone! (perhaps it will help me when I forget what happened here!)

As a follow on, long long ago Frank Denneman and Alan Renouf posted about the dangers of mismatched OS’s and VM settings and a really cool one liner for finding them here:

http://frankdenneman.nl/2009/12/15/impact-of-mismatch-guest-os-type/

Links of thanks:

http://vtagion.com
http://frankdenneman.nl/
http://www.virtu-al.net/