Thursday, October 8, 2015

vCenter server service vpxd.exe crashing and high CPU usage

So you can't connect to vcenter (or the connection drops after a minute or so), and you look on the vCenter server and see vpxd.exe and java.exe using as much CPU processing power as you can throw at them. You also notice that the Process ID (PID) changes every few minutes. Oh, and every time you kill the VMware VirtualCenter Server service it starts back up again (unless you disable the service).

Quick Answer:
Check the logs -- Specifically C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd-####.log (where ### was the PID).

In our case it was the fact that we were using SQL express 2008R2 (which has a database size limit of 10GB)... and the database was full:

"
2015-10-08T02:46:03.566-04:00 [06944 error 'Default' opID=SWI-4468da36] [Vdb::IsRecoverableErrorCode] Unable to recover from 42000:1105
2015-10-08T02:46:03.566-04:00 [06944 error 'Default' opID=SWI-4468da36] [VdbStatement] SQLError was thrown: "ODBC error: (42000) - [Microsoft][SQL Server Native Client 10.0][SQL Server]Could not allocate space for object 'dbo.VPX_EVENT_ARG'.'PK_VPX_EVENT_ARG' in database 'VIM_VCDB' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "INSERT INTO VPX_EVENT_ARG WITH (ROWLOCK) (EVENT_ID, ARG_ID, ARG_TYPE, ARG_DATA, OBJ_TYPE, OBJ_NAME, VM_ID, HOST_ID, COMPUTERESOURCE_ID, DATASTORE_ID, NETWORK_ID, NETWORK_TYPE, DVS_ID, DATACENTER_ID, RESOURCEPOOL_ID, FOLDER_ID, ALARM_ID, SCHEDULEDTASK_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
--> backtrace[00] rip 000000018018b95a
"

We have an enterprise license of SQL, so we:
  1. Connected to the esxi host that was running vCenter using VMware vSphere Client
  2. Mounted the SQL server enterprise ISO to that VM
  3. Logged into that system
  4. Disabled the "VMware VirtualCenter Server" service
  5. Killed the vpxd.exe process
  6. Upgraded the SQL edition
    • Launched setup.exe
    • Maintenance
    • Edition Upgrade
    • Ok... Next... Next... Accept License... Select Instance (if there is more than 1 instance)... Next... Next... Upgrade
    • Restart the SQL service
  7. Grow the database past 10GB... to 11GB or similar.
    1. USE [master]
      GO
      ALTER DATABASE [VIM_VCDB] MODIFY FILE ( NAME = N'VIM_VCDB', SIZE = 11509760KB )
      GO
  8.  Enable and start the "VMware VirtualCenter Server" service
  9.   Now make sure there was not anything unusual going on to cause the db to grow.
Bored yet? Make even more work for yourself... upgrade vCenter - Download here https://my.vmware.com/web/vmware/details?downloadGroup=VC55U2&productId=353

Or maybe the issue was something else:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2034127



1 comment:

  1. A Windows feature called superfetch preloads frequently used programmes into memory for quicker access. On some systems, nevertheless, it can result in performance concerns. To disable Superfetch in Windows, open Services, locate SysMain, right-click and choose Properties, change Startup type to Disabled, and then click Stop.

    ReplyDelete