In the vSphere Client, Datastore inventory view (Ctrl+Shift+D), VMware kindly gives us datastore Capacity and Free space values, but there is no column for  Provisioned.

If you open a datastore, the Provisioned amount is displayed:

In my (humble) opinion, besides knowing how much Free space is left on the volume, Provisioned is important too so you know just how far in the hole you’re digging yourself by over-provisioning datastores, and it would be nice to see this in the list view of all Datastores as a way of comparison.

Since we don’t have that column available to us (VMware, pretty please?), a bit of Powershell can give us what we need.

connect-viserver your_vcenter_server
$datastores = get-datastore | where-object {$_.name -match "Servers"} | get-view
$datastores | select -expandproperty summary | select name, @{N="Capacity (GB)"; E={[math]::round($_.Capacity/1GB,2)}}, @{N="FreeSpace (GB)"; E={[math]::round($_.FreeSpace/1GB,2)}}, @{N="Provisioned (GB)"; E={[math]::round(($_.Capacity - $_.FreeSpace + $_.Uncommitted)/1GB,2) }}| sort -Property Name

In my example above, I am using where-object to filter only for datastores that have “Servers” in the name. Remove it or customize it as needed. The snippet above produces the following output:

.. you could even append “Export-CSV c:\path\to\output.csv -NoTypeInformation” to the end to write it to a CSV file, useful for Excel or other things.

Based on the following pages:

Tags:

Recently we needed to expand our Virtual Desktop VMs from 10GB C: drives to 15GB to accommodate some updates for one of our primary applications. The VMs are chronically low on free space so the decision was made to expand the drive.

The first step was to grow each VM’s VMDK from 10GB to 15GB. This was easily accomplished with Powershell:

Get-Folder Desktops | Get-VM | Get-HardDisk | Where {$_.CapacityKB -eq 10485760} | Set-HardDisk -CapacityKB 15728640 -Confirm:$false

Simply put, this one-liner acts against all VMs in the “Desktops” folder, and if it currently has a 10GB drive (10,485,760 KB), it uses the Set-HardDisk command to expand it to 15GB (15,728,640 KB). This worked incredibly well. I did run in to one or two VMs that had trouble expanding because they could not be “stunned” while currently in the process of VMotioning to another host. They weren’t actually migrating, so I quickly powercycled them and ran the one-liner again, with success.

The VMDKs are thin-provisioned so no additional space was consumed on the datastores, but now the disks can grow beyond 10GB, up to 15GB.

The next step which fortunately is “not my problem” is to expand the C: partition from 10GB to 15GB. I advised our desktop manager to try Dell’s “extpart” utility which can expand the C: partition while the VM is booted. I have used it successfully with Windows Server 2003, but not tried it with XP. I believe it will work though. Our desktop manager will probably make use of Altiris and some batch scripts to run extpart on all the desktops.

We have about 55 ESXi 4.1 hosts in our environment, all configured to send their syslog data to a standalone linux (Ubuntu) blade server (HP) which also runs our Nagios server monitoring system. Since pointing all the ESXi hosts to the Nagios server for their syslog data, the server is constantly drowning under a massive amount of I/O that needs to be written to the filesystem.

Using iostat -x 1 I could see that the utilization of the disks was always around 100% and the average wait could explode up to 3000+msec which is not that great. I assumed it was syslog causing the problem but I’m not quite sure. Either way, I don’t need all the verbose data from the ESXi hosts in syslog and clogging up the I/O subsystem so here is a way to change verbose logging to something else (none, error, warning, information) (based on http://communities.vmware.com/thread/285254)

Here is what I did:

  1. Enable Remote Tech Support Mode (SSH) on all hosts seen by the vCenter server “serverName” using Powershell:
    connect-viserver serverName
    get-vmhost | foreach-object { get-vmhostservice -vmhost $_ | where {$_.Key -eq 'TSM-SSH'} | start-vmhostservice -confirm:$false }
    
  2. Create a script that SSH’s to a given server to run the commands:
    #!/bin/bash
    
    ssh -l root $1 "mv /etc/vmware/hostd/config.xml /etc/vmware/hostd/config.xml.orig && sed -e 's/<level>verbose<\/level>/<level>warning<\/level>/' /etc/vmware/hostd/config.xml.orig > /etc/vmware/hostd/config.xml && mv /etc/opt/vmware/vpxa/vpxa.cfg /etc/opt/vmware/vpxa/vpxa.cfg.orig && sed -e 's/<level>verbose<\/level>/<level>warning<\/level>/' /etc/opt/vmware/vpxa/vpxa.cfg.orig > /etc/opt/vmware/vpxa/vpxa.cfg && services.sh restart hostd && /sbin/auto-backup.sh"
    

    Using sed, it updates /etc/vmware/hostd/config.xml and /etc/opt/vmware/vpxa/vxpa.cfg to replace the default “verbose” logging levels with “warning” (which could be any of the levels mentioned earlier)

  3. Run this script on a separate linux host using the ESXi hostname as the first and only argument and then accept the root key and provide the root password. The script will update the files, restart all the services, and then backup the changes so they are saved.
  4. Once all the hosts have been changed, run this powershell code to stop the SSH service:
    connect-viserver serverName
    get-vmhost | foreach-object { get-vmhostservice -vmhost $_ | where {$_.Key -eq 'TSM-SSH'} | stop-vmhostservice -confirm:$false }
    

We are testing out the latest VMware View release, 4.5, and during an uninstall of an older release, somehow the vCenter service lost its connection to the SQL Server 2005 Express database that we are using and it would not start.

After some poking around, the vCenter log (C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vpxd-*.log) contained these telling lines:

[2010-09-15 09:14:55.354 03728 info 'App'] [Vpxd::ServerApp::Init:759] Calling: VpxdVdb::Init(Vdb::GetInstance(), false, false)
[2010-09-15 09:14:55.354 03728 error 'App'] ODBC error: (IM002) – [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
[2010-09-15 09:14:55.354 03728 error 'App'] Error getting configuration info from the database
[2010-09-15 09:14:55.354 03728 error 'App'] [Vpxd::ServerApp::Init] Init failed: VpxdVdb::Init(Vdb::GetInstance(), false, false)
[2010-09-15 09:14:55.354 03728 error 'App'] Failed to intialize VMware VirtualCenter. Shutting down…
[2010-09-15 09:14:55.354 03728 info 'App'] Forcing shutdown of VMware VirtualCenter now

Poking around the VMware Knowledge Base, I found the article 1003928: Troubleshooting the database data source used by VirtualCenter Server which mentions the registry key “HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter\DB”.  The value for the key “1″ was blank, meaning no DSN was specified. I changed the value to our System DSN “vcenter”.

Next I opened up odbcad32 to actually look at the “vcenter” System DSN but could not open it because the SQL Native Client 10.0 driver was missing! I downloaded it from the Microsoft SQL Server 2008 Feature Pack, August 2008 and installed it and then was able to open the DSN. I found that the “Server” field was simply set to “localhost” and when I tried to test the connection, it timed out. I looked at the old ViewComposer DSN and it had a server and instance specified – “VIEW45-VC\SQLEXP_VIM”. I put this in the “vcenter” DSN and then with integrated Windows authentication, I was able to change the default database to “VIM_VCDB” and test the connection and it succeeded. I saved the DSN and attempted to start the vCenter service. This time it started and stayed running and the logs looked good.

VMware KB wins again!

VMware backup

At work, we have been running a virtualized server environment for about 5 years. Currently we have 8 ESXi 4.1 HP blade servers hosting about 165 guest VMs.

All our servers including our VMs are backed up using Tivoli Storage Manager (TSM) which does an incremental backup every night to a disk pool and then is offloaded to a tape pool during the day.

We can use TSM to restore individual files inside a guest VM but we can’t easily (or at least have never tried) to restore an entire VM if it were to be lost.

A few months ago during a fresh installation of ESXi 4.0 (prior to the release of 4.1), I accidentally selected the wrong SCSI disk to install ESXi on and ended up nuking an entire VMFS datastore. Luckily it was only hosting about 25 virtual desktops and those were easily recreated. But it got me thinking about what would happen if we accidentally lost an entire VMFS datastore hosting server VMs, or even just lost one server VM for any reason.

Therefore I decided to look in to backing up the entire VM and not just the files inside it. I asked around and unequivocally heard that the best solution was the Veeam Backup & Replication suite. My only experience so far with Veeam has been their FastSCP product and that was a nice piece of software, though I did not need to use it that much. I don’t think I even have it installed.

I plan to ask Veeam for a 16-socket trial license so I can get an idea of what backing up my entire environment will be like. I will use changed-block tracking to enable quick incremental backups after the large initial one.

My backup target will be an old EVA8000 array that we purchased and has about 15TB of FC and FATA disk. I only need about 6TB so I plan to carve out a large Vdisk and present it to a Windows Server 2008 Standard R2 64-bit HP blade server where I will install Veeam Backup & Replication. I will also assign the existing VMFS datastores to the Veeam Backup & Replication server in order to do LAN-free (SAN only) backups going from the VMFS datastores, through the backup server, to the destination Vdisk.

I am very excited about this and hope the license-purchase quote that I also asked for will not be prohibitively expensive. There are several local resellers as well as CDW that I could purchase it from, so I will have to shop around and see who can offer the best deal.

I think it’s great what Veeam has put together and I look forward to checking it out.

Tags: ,

« Older entries