First, let me say that I failed to troubleshoot this alone and ended up making call for assistance.
Environment: vCenter 4.0 U1 with ESX 4.0 U1, two locations.
I encountered an odd error that occurred when I had detached an ESX server from an existing cluster and attempted to add it back in. The host had come down for hardware repairs, I removed it from the cluster so it wouldn’t show up on my vCheck reports. Upon attempting to add the server back I received a pop-up from the vCenter client with the message “vCenter does not support hosts of this type (VMware ESX 4.0.0).” I spent several hours, far too many maybe, troubleshooting this as a host issue but was not finding anything of interest in the logs. I could see that the vpxuser account was being created, key exchanges occurring but then the vpxa scripts were never installed. In frustration I rebuilt the ESX server with exactly the same outcome. I found logging in the ESX server ‘Authd error: 551 There is no VMware process running for config file vmware-vpxa’. This lead to VMware KB Article 1007337 which indicated that there was insufficient free space on the ESX server file system. This was not my issue.
Next I pulled another ESX server out of the cluster and attempted to add back without the same result. Adding these hosts to my other vCenter server was successful.
Finally on the right track I started troubleshooting my vCenter server and found the following relevant logging.
[2010-07-26 16:35:01.592 02728 error ‘App’] [VpxdHostUpgrader] Unrecognized bundle version 15. The bundleversion.xml file may be out of date
[2010-07-26 16:35:01.592 02728 info ‘App’] [VpxdHostUpgrader] vpxa not reporting correct version. bundleVersion = 15, build = 208111, expected bundleVersion = -1
[2010-07-26 16:35:01.715 02728 error ‘App’] [VpxdHostUpgrader] Unrecognized bundle version 15. The bundleversion.xml file may be out of date
[2010-07-26 16:35:01.715 02728 info ‘App’] [VpxdHostUpgrader] vpxa not reporting correct version. bundleVersion = 15, build = 208111, expected bundleVersion = -1
[2010-07-26 16:35:01.776 02728 info ‘App’] [VpxLRO] — FINISH task-18739 — domain-c34 — vim.ClusterComputeResource.addHost — 07063040-6EC0-44DE-9F79-2E1652BFA653(371258EB-2C3F-49F2-ACB8-147A610F6F0A)
[2010-07-26 16:35:01.776 02728 info ‘App’] [VpxLRO] — ERROR task-18739 — domain-c34 — vim.ClusterComputeResource.addHost: vim.fault.NotSupportedHost:
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
productName = “VMware ESX”,
productVersion = “4.0.0”,
msg = “”,
So the error is stating that vCenter cannot compare vpx bundle version but what turned out to really be the case is that vCenter could not find it’s own bundleversion.xml file. In fact, the entire contents of ‘C:Program Files (x86)VMwareInfrastructureVirtualCenter Serverupgrade’ was missing.
Quick fix, since I was fortunate enough to have another vCenter server, was to copy the contents of the directory to my problematic server. At this point I was able to add the ESX servers into the cluster without incident. In fact, all seems to be well now and I didn’t even have to restart services. Now, the real question I need to be asking is “How did the contents of my directory go missing?”
I have asked VMware to review, clean up and publish an internal KB article that lead me to the ultimate solution.