Author Archives: chouse

HP Virtual Connect firmware 3.01 available

I’ve been tearing my hear out over the last couple of days trying to solve a problem that appeared after I upgraded a few of my HP c-class enclosures to the 3.00 version of the HP Virtual Connect firmware.

The problem was whenever viewing a server profile or a defined network, the Flex UI would just sit there with a box entitled “Loading” that said “Please wait…” with a neverending progress bar. One of those progress bars that just is infinite and pretty worthless. If you’re going to include a progress bar in your display, it should actually indicate the progress of the current action and stop at some point. But this one just sat there all animated and infinite.

I power cycled one and then both VC Ethernet modules. I tried restoring a backup of the config that I just made. I power cycled the entire enclosure. Nothing would solve this problem for me. Right before I was about to call HP, I visited the HP 1/10GB Virtual Connect Ethernet Module download page and was pleasantly surprised (ecstatic really) to see that version 3.01 had been released (with no sign of 3.00 even having ever existed) and one of the purported fixes was for the very issue I was experiencing!

I quickly downloaded the firmware and applied it to the two enclosures where I had previously applied the 3.00 version. Problem solved!

Moral of the story: check for newer firmware before trying anything else. My problem was that 3.00 had come out not too long ago and I knew that VirtualConnect firmware is not updated too often so I subconsciously assumed there wouldn’t be anything new.

Fixing vCenter Linked Mode

I recently had a problem with one of my two vCenter servers in linked mode. When using the vSphere client to connect to this particular server, the client would not show the hosts/VMs or anything about the 2nd vCenter server.

But when connecting with the client to the 2nd vCenter server, it would show both vCenter servers and their hosts/VMs.

So obviously something is up with the first vCenter server. I ran the Linked Mode setup again and it acted like it wasn’t linked, which was how it appeared. So I went through the prompts to set up the linked mode and when it came down to actually setting it up, it would fail. It did this twice. Looking at the log that it mentioned, it was complaining that the vCenter Server service was not stopped. It had successfully stopped the Web Access service, but not vCenter Server.

I loaded up services.msc and went to stop the vCenter server service but was prompted that our Site Recovery Manager service would also need to be stopped. After agreeing and stopping both services, the Linked Mode configuration completed successfully and started the vCenter Server and Web Access services. It did not however start the Site Recovery Server service.

At this point it looks like the Linked Mode configurator needs to understand about other services that depend on vCenter Server and track them to stop/start before/after the linked mode is configured.

I don’t know why the vCenter server became unlinked from the 2nd one, but it works good now!

Disable cluster HA before VUM remediation of hosts

I have discovered that during the intense process of host remediation using VMware Update Manager, VMware HA on a cluster may become misconfigured and no longer function correctly. This can cause issues when VUM commands a host to enter Maintenance Mode and all the VMs need to VMotion away, but all eligible hosts are reporting errors with their HA configuration (usually because all the primary HA agents are no longer responding and hosts can’t reconfigure themselves after exiting maintenance mode).

Without eligible hosts to VMotion VMs to, the enter-maintenance-mode task will fail. VUM handles this failure depending on how it is configured when the update task was created. I usually set it to Retry with 1 minute delay, a maximum of 5 times. This allows me to notice the failure if I happen to glance at the client and figure out what to do to help it along. VUM could also Fail the update task but this is annoying if the task hasn’t gotten very far with other hosts in the cluster. It would be nice if the host could be skipped after it couldn’t enter maintenance mode and VUM could focus on the rest of the hosts.

It seems the best thing to do is to just disable HA on the cluster before starting VUM remediation. Instead of HA being disabled/enabled for each host during the process, they are all disabled. This saves time on the overall process and now VUM will not fail the entire task.

With that being said, in addition to VUM skipping hosts that can’t enter maintenance mode, it would also bee nice if the cluster itself could recognize the total HA misconfiguration and globally disable/enable it accordingly. That’s what the Administrator would have to do anyway so automating this would be nice.

Reinstalling Update Manager

One of my monthly health checks for our vSphere environment (which I’ll detail later) is to run a VUM (VMware Update Manager) scan of our ESXi hosts to see if they’re missing any patches. One of the things I love about ESXi is the fact that it does not have a full-blown Service Console like the “full” or “classic” ESX. This service console is based on redhat so any RedHat patches that are released by them are also needed on the ESX service console. But ESXi’s minimal service console isn’t based on RedHat and therefore doesn’t have all those extra pieces of software that need to be patched.

So ESXi patches don’t come out very often but I do make it a point to scan my hosts monthly and apply any updates that have come out. Usually these updates consist of updates to VMware Tools and then a new firmware image for ESXi.

This is the first month that I’ve implemented these monthly health checks and I’ll start by running that VUM scan.

We have two vCenter servers (which I’ll detail later) and they both have their own installation of VUM and their own patch repository. I do want to get to having a combined repository to save space, but they’re not very big so it’s not a big priority at the moment. The first vCenter server, Aero, which manages the ESXi hosts/clusters in our primary datacenter showed the “Update Manager” tab as expected when I clicked on the top-level vCenter server object in vCenter client, but when I went to check on Extra, our other vCenter server which manages the ESXi hosts/clusters in our secondary datacenter, the Update Manager tab was not present! It wasn’t there on any of the expected levels – vCenter server, Datacenter, Cluster, or Host.

I checked that the VUM plugin was installed and enabled on both my PC where I have the vSphere Client installed, and I also checked that it was installed and enabled on the vSphere Client which I’ve installed on the vCenter server Extra. It looked good but just to be safe, I disabled it and relaunched the client. No change. Then I uninstalled it from Add/Remove programs and installed it again via the client. No change. At this point, since I was seeing the behavior on both my PC and the vCenter server itself, I decided the problem must be with the VUM service itself. I restarted the VUM service on the vCenter server but that did not make any difference either.

I did a quick google search [missing “update manager” tab] and ended up at this post on the VMware Communities forum for vCenter Server: missing update manager tab where others had experienced the same problem. I knew that my issue was not that I hadn’t scrolled the tabs over far enough, and I didn’t understand the PATH issue that others had mentioned. I did check my PATH variable but it looked OK and we don’t use Norton products. We are running a 32-bit OS though, but we aren’t getting any messages about a lack of drive space. Either way, I decided to reinstall the VUM service. I visited the vSphere download page to see if there was a newer release than what I had previous downloaded, but there wasn’t.

I extracted the zip file for the VUM service which I had downloaded a few months ago when this latest release came out and copied the files up to my vCenter server. The structure inside the zip file containing the VUM install files is always a bit of a mystery. I navigated to the “bin” folder and found the VMware vCenter Update Manager.msi file which I right-clicked on and chose “Repair” just for fun. This failed saying it couldn’t register with the vCenter server or something, so then I right-clicked again and chose “Uninstall”. After verifying the service was no longer listed in Services, I doubleclicked on the VMware-UpdateManager.exe file to launch the installer.

I gave it credentials to connect to the vCenter server and reused the existing VUM tables inside my vCenter database. After the service was up and running, I launched vSphere Client and found that the Update Manager tab was back! I doublechecked the settings in “Admin View” and found that the Patch download schedule wasn’t really configured correctly. I set it to run every day and email me when new patches are found. I also made sure to set the time to be within a few minutes. Soon the task started and apparently didn’t have to download much because it ended pretty quickly.

I was then able to rescan the datacenter and find that my ESXi hosts are missing 1-2 patches.

I will write about how we use Update Manager at a later point but just wanted to point out that sometimes it just comes down to reinstalling a product to get it to work correctly. Luckily the configuration is stored in a database and that was able to be reused so reinstalling it wasn’t any big deal. There’s not much to configure anyway with VUM but it’s nice to be able to reuse the database.

Virtualization stack

Welcome to VirtIRL!

Welcome to VirtIRL – Virtualization: In Real Life – a virtualization blog dedicated to the usage of different kinds of virtualization technology in real life. Not necessarily limited to server or desktop virtualization, I cover any sort of virtual topic as it comes up in my daily work life.

A lot of virtualization blogs cover the in-depth configuration of the various products that are out there, but not many are dedicated to the day to day usage of the technology and its real-life implementation.

My name is Chris and I work for a 200-bed health care provider in the West Michigan area. I’m not a virtualization guru but I plan to outline how we use virtualization to reduce our physical server count and increase the availability of our applications. My configurations may give you a good jumping off point for building your own infrastructure, or they may not make any sense at all for your organization.