We have about 55 ESXi 4.1 hosts in our environment, all configured to send their syslog data to a standalone linux (Ubuntu) blade server (HP) which also runs our Nagios server monitoring system. Since pointing all the ESXi hosts to the Nagios server for their syslog data, the server is constantly drowning under a massive amount of I/O that needs to be written to the filesystem.

Using iostat -x 1 I could see that the utilization of the disks was always around 100% and the average wait could explode up to 3000+msec which is not that great. I assumed it was syslog causing the problem but I’m not quite sure. Either way, I don’t need all the verbose data from the ESXi hosts in syslog and clogging up the I/O subsystem so here is a way to change verbose logging to something else (none, error, warning, information) (based on http://communities.vmware.com/thread/285254)

Here is what I did:

1. Enable Remote Tech Support Mode (SSH) on all hosts seen by the vCenter server “serverName” using Powershell:

   connect-viserver serverName
   get-vmhost | foreach-object { get-vmhostservice -vmhost $_ | where {$_.Key -eq 'TSM-SSH'} | start-vmhostservice -confirm:$false }
   

2. Create a script that SSH’s to a given server to run the commands:

   #!/bin/bash
   
   ssh -l root $1 "mv /etc/vmware/hostd/config.xml /etc/vmware/hostd/config.xml.orig && sed -e 's/<level>verbose<\/level>/<level>warning<\/level>/' /etc/vmware/hostd/config.xml.orig > /etc/vmware/hostd/config.xml && mv /etc/opt/vmware/vpxa/vpxa.cfg /etc/opt/vmware/vpxa/vpxa.cfg.orig && sed -e 's/<level>verbose<\/level>/<level>warning<\/level>/' /etc/opt/vmware/vpxa/vpxa.cfg.orig > /etc/opt/vmware/vpxa/vpxa.cfg && services.sh restart hostd && /sbin/auto-backup.sh"
   

   Using sed, it updates /etc/vmware/hostd/config.xml and /etc/opt/vmware/vpxa/vxpa.cfg to replace the default “verbose” logging levels with “warning” (which could be any of the levels mentioned earlier)

3. Run this script on a separate linux host using the ESXi hostname as the first and only argument and then accept the root key and provide the root password. The script will update the files, restart all the services, and then backup the changes so they are saved.
4. Once all the hosts have been changed, run this powershell code to stop the SSH service:

   connect-viserver serverName
   get-vmhost | foreach-object { get-vmhostservice -vmhost $_ | where {$_.Key -eq 'TSM-SSH'} | stop-vmhostservice -confirm:$false }