Happy Easter!
Easter eggs can be dyed in many different colors… just like the many OS’s you can run in a vSphere VM! (Which is 93 if you count everything you can set New-VM’s GuestID!)
Interestingly, when you deploy a VM using PowerCLI and don’t specify an OS with GuestID it defaults to Windows XP (32 bit):
As I found out this can lead to some challenges. I was running into a problem where it seemed like the VMtools were failing. This all started occurring when testing an existing project that was working fine in vSphere 5.5 on a new vSphere 6.0 install. The VM was being deployed by PowerCLI and mounting a OpenSUSE Live ISO to boot which also had open-vm-tools on it so you can run scripts, copy files, etc…
Lets try running a script against our newly created “JustANewVM” which was created without specifying a GuestID and is running OpenSUSE with open-vm-tools.
While red is my favorite color, I don’t like it at all in this context!
invoke-vmscript : 4/7/2015 12:11:42 PM Invoke-VMScript The remote server returned an error: (500) Internal Server Error. At line:1 char:1 + invoke-vmscript -VM $new2 -guestuser root -guestpass vmware -ScriptText "ls /etc ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Invoke-VMScript], ViError + FullyQualifiedErrorId:Client20_VmGuestServiceImpl_DownloadFileFromGuest_DownloadError,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript
The corresponding host’s hostd.log file didn’t shed much light on this either, lines 4 and 5 do let us know something’s gone awry:
2015-04-07T17:42:16.496Z info hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GetGuestFileTransferParameters, token is 5250241e-bbc5-ffa9-cda3-50a00349485d92 2015-04-07T17:42:16.496Z info hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GetGuestFileTransferParameters: vmmoid is 92 2015-04-07T17:42:16.496Z info hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] Guest File Path is /tmp/vmware-root/powerclivmware11 2015-04-07T17:42:16.497Z info hostd[6D9F2B70] [Originator@6876 sub=Libs] HGFileCopy_TransferFileUsingReader: Error, cannot convert /tmp/vmware-root/powerclivmware11 to CPName 2015-04-07T17:42:16.497Z info hostd[6D9F2B70] [Originator@6876 sub=Libs] Vix: [440716 vmGuestOps.cpp:5377]: Error VIX_E_FAIL in CopyFileDoneCallback(): Unknown error 2015-04-07T17:42:16.497Z error hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GuestFileCompleteFunc() job failed with 22 2015-04-07T17:42:16.497Z error hostd[6D9F2B70] [Originator@6876 sub=Guestsvc.GuestFileHandler] GuestFileCompleteFunc() throwing InternalServerError
Now I think there might be a problem with the host.. so I restart hostd and try again, same result. Next, I vMotion the VM over to another host and run the same command and get these logs:
2015-04-07T19:45:57.465Z info hostd[52281B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52088dbc897dbeb9-4ba67b7615492055/411f2455-5c25-d1cc-8ef7-002481fd70f6/JustANewVM.vmx] [N8Guestsvc23StartProgramRequestImplE:0x5232a938] opCode=4 auth=<hidden> programPath=cmd.exe arguments=/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEAdAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsAbABzACAALwBlAHQAYwB9ACcAIAA+ACAAIgAvAHQAbQBwAC8AdgBtAHcAYQByAGUALQByAG8AbwB0AC8AcABvAHcAZQByAGMAbABpAHYAbQB3AGEAcgBlADEANgA1ACIAOwAgAGUAeABpAHQAIAAkAGwAYQBzAHQAZQB4AGkAdABjAG8AZABlAA== failed
2015-04-07T19:45:57.465Z info hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Activation [N5Vmomi10ActivationE:0x5232a8d8] : Invoke done [startProgram] on [vim.vm.guest.ProcessManager:ha-guest-operations-process-manager]
2015-04-07T19:45:57.465Z verbose hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Arg vm:
--> 'vim.VirtualMachine:14'
2015-04-07T19:45:57.465Z verbose hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Arg auth:
--> (vim.vm.guest.NamePasswordAuthentication) {
--> interactiveSession = false,
--> username = "root",
--> password = (not shown)
--> }
2015-04-07T19:45:57.465Z verbose hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Arg spec:
--> (vim.vm.guest.ProcessManager.ProgramSpec) {
--> programPath = "cmd.exe",
--> arguments = "/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEAdAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsAbABzACAALwBlAHQAYwB9ACcAIAA+ACAAIgAvAHQAbQBwAC8AdgBtAHcAYQByAGUALQByAG8AbwB0AC8AcABvAHcAZQByAGMAbABpAHYAbQB3AGEAcgBlADEANgA1ACIAOwAgAGUAeABpAHQAIAAkAGwAYQBzAHQAZQB4AGkAdABjAG8AZABlAA==",
--> workingDirectory = <unset>,
--> }
2015-04-07T19:45:57.466Z info hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.FileNotFound
2015-04-07T19:45:57.466Z info hostd[52281B70] [Originator@6876 sub=Solo.Vmomi] Result:
--> (vim.fault.FileNotFound) {
--> faultCause = (vmodl.MethodFault) null,
--> file = "cmd.exe",
--> msg = ""
--> }
A completely different error came back, which validates my assumption about the host… OR NOT! The error in the PowerCLI windows was different as well, telling me that there was not a PowerShell interpreter that could be run. Running the same command subsequent times still produced the original error.. Now, why vMotioning it seems to change the verbosity level I do not know? I was glad it did however, as it provided me a the clue into what was going on! If you look at the logs you can see it’s trying to run cmd.exe on the VM and pass it a powershell “EncodedCommand”. Cool, but that’s not going to work on a *nix box! At this point I started looking at why vSphere thought this was a Windows machine and found that the default behavior of New-VM without -GuestID is to make it a Windows XP 32 bit VM as discussed above.
As I could see that it was listed as a Windows XP VM, I powered it off, set the OS to the correct one (otherLinux64Guest in this case) and rebooted. When I tried to run the command, it failed yet again! The host logs showed the “VIX_E_FAIL” error initially seen in the host logs. I decided to restart hostd on the host where the VM was running and after that the PowerCLI command ran fine.
An additional anomaly that I found was that when the VM was powered on and the tools started the C# (fat, and going away someday) client would update the Guest OS: field on the summary page for the VM to be the correct one. The web-client however did not. I believe that the different clients are looking at different information, the C# client is most likely looking at the properties in ExtensionData.Summary.Guest where the Web-Client is looking at the ExtensionData.Config properties. In this case the C# client is more accurate/dynamic, not sure of the reasoning behind this.
PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> $newVM.ExtensionData.Config ChangeVersion : 2015-03-26T19:59:59.952422Z Modified : 1/1/1970 12:00:00 AM Name : JustANewVM GuestFullName : Microsoft Windows XP Professional (32-bit) GuestId : winXPProGuest
PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> $newVM.ExtensionData.Summary.Guest GuestId : otherGuest GuestFullName : Linux 3.11.10-25-default openSUSE 13.1 (x86_64) ToolsStatus : toolsOk ToolsVersionStatus : guestToolsUnmanaged ToolsVersionStatus2 : guestToolsUnmanaged ToolsRunningStatus : guestToolsRunning HostName : linux-e2zm IpAddress : 192.168.2.70
* I removed a bunch of bits from the above that weren’t relevant, but it’s clear that these are different sets of data and the different clients look at one or the other to get the OS info.
I hope this helps someone! (perhaps it will help me when I forget what happened here!)
As a follow on, long long ago Frank Denneman and Alan Renouf posted about the dangers of mismatched OS’s and VM settings and a really cool one liner for finding them here:
http://frankdenneman.nl/2009/12/15/impact-of-mismatch-guest-os-type/
Links of thanks:
http://vtagion.com
http://frankdenneman.nl/
http://www.virtu-al.net/