Category Archives: upgrade

View Admin dashboard for vCenter Server 5.1 displays the message: VC service is not working properly

Regarding VMware KB: View Admin dashboard for vCenter Server 5.1 displays the message: VC service is not working properly, I encountered this when upgrading a vCenter 5.0 host to 5.1.

When it was all said and done, I checked the View 5.1.3 console and found the vCenter objects in a red error state, saying the service was not working properly.

I edited the vCenter server in View and it gave me the incorrect username/password. Since that is one of the issues in the KB, I followed the steps to move the View domain to be above the System domain in SSO.

Seems like an obvious thing to do for a new SSO installation – wish the SSO installer would have done this.

Once moved, the vCenter objects turned green and provisioning was able to run.

Updating VM tools without rebooting

Whenever the ESX hosts in a cluster are upgraded, a newer version of VMware Tools becomes available which should be applied to each VM in the cluster. The Tools are updated drivers and services that enable the VM OS to run more efficiently in a virtualized environment. A normal “automatic” tools upgrade via the vCenter client will cause the VM to reboot when the tools upgrade is finished. If this is OK, then go for it. Otherwise, use this process to suppress the reboot and let things take effect during the next scheduled reboot.

Note: most things to take effect after the update, such as newer services, and drivers that can be safely unloaded/loaded without requiring a reboot.

Note: the NIC driver may be updated causing a brief drop in traffic.

To upgrade the VMware Tools on a Windows guest without rebooting, use this bit of PowerShell code with the vSphere PowerCLI.

First run this to set things up (connects to the specific vCenter server using your current Windows credentials and sets the installation parameters to pass to the installer):

Connect-VIServer
$insParm = '/S /v"/qn REBOOT=ReallySuppress"'

Then run this for each VM, changing with the name of the VM to update tools on, without rebooting:

$vmView = Get-VM | Get-View; $vmView.UpgradeTools_Task($insParm)

Example:

$vmView = Get-VM Babyruth | Get-View; $vmView.UpgradeTools_Task($insParm)

A task will start in vCenter and the tools should be updated on the VM. It takes about 2-3 minutes. If it seems to be taking longer than that to complete, check the VM console to make sure no errors have popped up that require attention, and also check if setup.exe and one or more msiexec.exe processes are running.

The task should finish successfully and the VMware Tools Status for the VM should change to “OK”

There is plenty of discussion online about various ways to accomplish this (“How to install VMware tools without a reboot?“), but what I described above is what works for me. It is definitely possible to write a PowerShell script to traverse through a folder or cluster and upgrade Tools on all the VMs contained within, but I prefer to take it one at a time so the process doesn’t get out of hand. Tools upgardes are generally painless but sometimes there are issues.

vCenter

Upgrading to vCenter 4.1

Like a lot of folks, I was excited with the release of vCenter 4.1 and ESX(i) 4.1. Lots of great new features, some bugfixes, and the overall feeling of “ooh, new shiny things to play with!”

Every new release of vCenter/ESX is like Christmas morning as there are plenty of new features to be discovered and bugs fixed that streamline previous operations.

I was dismayed at first to read that vCenter required a 64-bit OS as not too long ago I had built up a nice pair of vCenter 4.0 servers on Windows Server 2003 Standard (32-bit) and liked the way they were working. I was running SRM 4.0 along with Update Manager and everything was working great.

I knew that with the 64-bit requirement, I might as well go to Windows Server 2008 R2 64-bit for my two vCenter servers, but I had some hurdles to jump over. Specifically, we aren’t really running any Server 2008 machines in-house, save for those that we have to experiment and kick the tires with. So I decided to build a Server 2008 template and then deploy a pair of new vCenter servers from that.

The process of building a new template VM has been documented widely across the virtualization blogosphere, but I shall throw my two cents in the ring with a post describing my own win2k8 template. But as usual, that is for a later time.

The other hurdle to leap over, in addition to getting Server 2008 set up for production use, was the fact that I was going to have to relocate my vCenter servers to brand new machines. I tried this once awhile ago with VirtualCenter 2.5 and the results were diastrous. So I was not encouraged with the fact that I was going to have to try to move again.

Luckily, the VMware team understood that a lot of people would be in the same boat due to their 64-bit-only requirement so they put together a set of python scripts to aid in the moving of vCenter to a new host.

The guides that I referred to during this upgrade are:

I started out by stopping my vCenter services, backing up the Microsoft SQL 2008 Standard databases for VC and SRM, and then detaching and copying them to the new vCenter servers where I already had Microsoft SQL 2008 Standard  64-bit installed. I attached the DBs and made sure my SQL “vpxuser” account was set to owner.

Then I created a 64-bit System DSN for the vCenter database by running “odbcad32” at the Run command. I also created a 32-bit System DSN for SRM by running c:\windows\sysWOW64\odbcad32 at the Run command (thanks to boche.net for that tip)

A side note: my VUM database is part of my vCenter database, so I didn’t have to create a DSN for that – it will just piggyback on top of the vCenter DSN as it did on my Server 2003 vCenter servers.

After my DSNs were created, I followed the instructions in the Upgrade Guide to copy the datamigration folder off the vCenter 4.1 ISO on to my 32-bit server and ran the “backup.bat” to backup my vCenter certificates and other configuration that lives outside the database. It also backed up my VUM and Orchestrator configuration (even though I don’t use Orchestrator).

I copied the resulting datamigration folder to the new host. It now has “data” and “log” folders that hold the backed up configuration. I ran “install.bat” on the new host and pointed it to my vCenter 4.1 media. One thing that got me at first was it prompted for the Update Manager media path and I gave it what I thought was the path – the Update Manager folder inside the vCenter installer, but that was incorrect and I had to end up giving it the same path that I gave for vCenter. It would be nice if it would just check there first for the Update Manager installer files and then only prompt if it can’t find them.

Then the datamigration install.bat file launched the vCenter installer and I stepped through that, changing the install path to the E:\ drive that I had set up for it and pointing it to the DSN that I had configured. I also allowed it to automatically update the ESXi/vCenter host agents. Soon the installer finished and then also launched the VUM one. I stepped through that as well, again changing the path to E: and pointing it to the DSN that it needed.

The VUM install was also successful and then datamigration install.bat utility ended. I didn’t even get to see any of the host agents being updated because when I got the client installed and logged in, everything looked great. I had a few servers that needed to be reconfigured for HA but other than that, the install went great.

I repeated this process on my 2nd vCenter host and it went well too.

I did have to break up my linked-mode config as it sort of stopped working – I could log in to one server and see both, but if I logged in to the other one, it would only show that one and an authentication failed message for the other one. I had to “force” it to isolate itself from linked-mode, but I think that was because the vCenter service hadn’t stopped in time. But 2nd time was a charm and after joining them back together, all was well.

I then installed SRM 4.1, paying careful attention to the release notes (linked earlier) describing how to relocate SRM to a new server with a new name. I installed the HP EVA Storage adapter and visited my SRM config to make sure it was OK. It was but I did have to reconfigure the EVA credentials. I don’t know if they were lost or what but it was no big deal. I ran a test recovery plan and that worked perfectly.

I then reconfigured VUM to not download VM patches for Windows or Linux and none for ESX3. I also added the ESX4.0 to 4.1 upgrade zip and created a baseline for upgrading all my ESXi 4.0 hosts to 4.1.

Finally I remembered to copy the sysprep files off my 2k3 vcenter servers so that we can continue deploying/customizing 2k3 and XP server VMs.

At this point, everything looks great and I’m heading home. ESXi upgrade next week!