VM in new host slower than VM in older host

Few months back, I led a project to upgrade an in-house application to a newer version running on new hardware.  New ESXi hosts with better configuration were added to the existing cluster and new Virtual Machines were created for the new version of the application.

One of the objectives of the upgrade is to increase the performance of the application based on the fact that the new VMs will be on new and more powerful hardware. 

So the team conducted performance testing but was getting some weird results.

Pin It

Missing Source Disk In VMware vCenter Converter

Last night I was trying to convert a physical server to a virtual machine using VMware vCenter Converter.  However, one of the disk from the source is not showing up in VMware vCenter Converter.

Pin It

VMware VirtualCenter Server failed to start after migrating to another datacenter

In my previous post Migrating VMware vCenter With Veeam Backup & Replication, I talked about the challenges I faced in migrating a vCenter VM across vCenter Datacenter managed by the vCenter itself.

Towards the end of the previous post, I mentioned that after successfully migrating the vCenter VM using Veeam Backup and Replication, I faced another issue.  The VMware VirtualCenter Server service failed to start.

The following error was found in the Windows application log.

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. If the event originated on another computer, the display information had to be saved with the event. The following information was included with the event: Failed to intialize VMware VirtualCenter. Shutting down... the message resource is present but the message is not found in the string/message table

Pin It

Migrating VMware vCenter With Veeam Backup & Replication

I have a vCenter managing two Datacenter (DC1 and DC2).  The vCenter is a virtual machine in DC1 Datacenter.  There is no shared storage between the two Datacenters.  I need to migrate the vCenter from DC1 to DC2 due to storage performance issue in DC1.  In DC1, there is only vSphere Standard Switch (VSS) so the vCenter is connected to VSS.  In DC2, vSphere Distributed Switch (VDS) is being use.

There are basically two major challenges in migrating the vCenter from DC1 to DC2.

1. Copying the vCenter’s VM files from DC1’s storage to DC2’s.  vMotion without shared storage feature does not work.  Using vCenter Converter to do a V2V does not work because it detected the destination host is managed by the vCenter and prevented it.  Shutting down the vCenter VM, copying out the VM files from the source datastore to an intermediate storage and then copying back to the destination datastore will take very long and causes extended downtime (remember the source storage is having performance issue).
2. With the vCenter powered off for copying, the VDS will not be available for selection in DC2.  The copied vCenter will not be able to connect to the VDS in DC2 because it is not available.  If the copied vCenter cannot connect to the VDS in DC2 then the copied vCenter cannot work but VDS needs vCenter.  It is a dead-lock situation.

Pin It

vCenter CPU and Memory Usage Not Working

One of my ESXi host was having “Monday Blue” (let’s call it blue host) this morning and decided to play a trick on me.  After logging on to vCenter, I saw many virtual machines were migrating off from the blue host to another ESXi host. 

I clicked on the Summary tab of the blue host and was shocked to see the CPU usage and Memory usage showing 0 MHz and 0.00MB respectively.  There were virtual machines running on the blue host but vCenter was not getting the usage information from the blue host.  The migration happened so fast that the other host was feeling the strain.

Pin It

vCenter Health Service–Health information is missing

I deployed VMware vCenter Operations Manager 5.7 (vCOPS) standard edition two weeks ago and saw that the Health badge having a score of 98.  Further drilling showed that vCenter was having an issue with an event status of vCenter Health Service–Health information is missing.

Pin It

Hold back your upgrade to vCenter Server 5.1 Update 1

If your vCenter Server login domain user account is associated with a large number of domain groups and multiple domains are configured as SSO identity sources, then you might not be able to logon to vCenter Server via the vSphere Web Client/vSphere Client after upgrade.

For more information check out this VMware Knowledge Base.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2050941

Pin It

VMware Tools Upgrade Failed

After upgrading my ESXi host, I tried to upgrade the VMware Tools of a virtual machine but it failed with the following error.

Pin It

VMware vCenter Converter P2V VSS Snapshot Error

A few weeks back, I was trying to convert a physical server into a virtual machine using VMware vCenter Converter.  However, conversion failed at 91% with the following error.

FAILED: The VSS snapshots cannot be stored because there is not enough space on the source volumes or because the source machine does not have any NTFS volumes. Error code:
2147754783 (0x8004231F).

I took a look at the application event log and saw this error.

Volume Shadow Copy Service error: Cannot find diff areas for creating shadow copies.  Please add at least one NTFS drive to the system with enough free space. The free space needed is at least 300 Mb for each volume to be shadow copied.

Pin It

Error Adding ESXi Host to vCenter Server Appliance

I deployed a VMware vCenter Server Appliance (VCSA) yesterday and got the following error when trying to add a ESXi 5 host.

Cannot contact the specified host (esxi server name).  The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

Call “Datacenter.QueryConnectionInfo” for object “DataCenter name” on vCenter Server “vCenter name” failed.

I am able to connect to the ESXi host directly using vSphere Client so the network configuration and management services seems to be working correctly.

Then I found this KB article saying that this issue may occur due to network latency.  The resolution is to increase the SSL handshake timeout.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1027672

I gave a try but it did not solve the issue.

Then I decided to add the host using IP address instead of its FQDN and it works.  This signalled that it must be a name resolution issue so I rechecked the VMware vCenter Server Appliance and found out that the DNS servers settings was missing.

I did enter the DNS servers’ IP addresses when going through the setup wizard so why did it disappear.  Well my guess is that I have entered more than two DNS server IP addresses (I entered three) but there is only room for two. 

Anyway, configuring the preferred DNS server and alternate DNS server IP addresses solved the problem.

Pin It

Custom ESXi 5.1 Image for Dell R720

As mentioned in my previous entry, I have decided to create a custom ESXi 5.1 image with the driver for Integrated Broadcom 5720 quad port network adapter included.  Here are the steps I took to create the custom image.

I will need to have VMware vSphere PowerCLI installed on the machine I will be using to create the image.

Download the ESXi 5.1 software depot (ESXi 5.1 Offline Bundle).  The file that I have downloaded is ESXi510-201210001.zip.

The offline bundle for the Broadcom 5720 network adapter driver is also needed.  The offline bundle which is tg3-3.124c.v50.1-offline_bundle-841079.zip is included in the downloaded tg3-3.124c.v50.1-841079.zip file.

Once I have everything ready, the first step is to import the software depots using the Add-EsxSoftwareDepot command.

I imported the ESXi 5.1 software depot (ESXi510-201210001.zip) follow by the Broadcom driver depot (tg3-3.124c.v50.1-841079.zip).

Pin It

Dell R720 NIC not detected by ESXi 5.1

Last week I installed VMware ESXi 5.1 on a new Dell PowerEdge R720 server.  After the installation, I was trying to the network settings and was “shocked” when the Integrated Broadcom 5720 quad port network adapter was not detected.  Well, it was not detected because the driver for the network adapter was not an inbox driver included in the ESXi 5.1 image.

To get the network adapter to work, I downloaded the driver from https://my.vmware.com/group/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/5_1#drivers_tools.  The driver version that I have downloaded is 3.124c.v50.1 and the file name is tg3-3.124c.v50.1-841079.zip.

After downloading the driver, I can either install the VIB file on the existing ESXi 5.1 or I can create a custom ESXi 5.1 image with the driver included. 

Pin It

Upgrade VMware ESXi 5.1 to 5.1.0a

ESXi 5.1 can be upgraded to ESXi 5.1.0a via Update Manager by applying the ESXi510-201210401-BG patch.

Attached the host patch baselines if it is not already attached.

Pin It

Upgrade VMware vCenter 5.1 to 5.1.0a

In less than a week after I have upgraded my vSphere 4.1  to vSphere 5.1, VMware released 5.1.0a to address some issues.  Read the release notes http://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-510a-release-notes.html to find out what are the resolved issues.

Fortunately upgrading from 5.1 to 5.1.0a is more straightforward and easier.  Just like all previous upgrade, vCenter is the first one to go.  Download the VMware vCenter Server 5.1.0a ISO from the VMware and mount it to my vCenter virtual machine.

Launch the VMware vCenter Installer.

Pin It

ESXi 5.1 Upgrade – No eXecute (NX) bit is not enabled on the host

I was trying to upgrade a ESXi 4.1 host to ESXi 5.1 and got the below warning message after a scan.

No eXecute (NX) bit is not enabled on the host.  New ESXi version requires a CPU with NX/XD bit supported and enabled.

Pin It

Initial Testing of Veeam Backup

I downloaded a copy of Veeam Backup & Replication 6.1 last week and did a quick initial test to backup two VMware virtual machines.

I am able to select the virtual disk that I want to backup instead of being forced to backup the whole virtual machine.  I am not able to do this with my current backup software.  This is a plus point for Veeam.

For the first virtual machine, I backup an 80GB virtual disk with about 27.9GB used space.  The backup took about 29 minutes to complete.  Only 15.2GB of disk storage was needed for the backup which is only 54.4% of the used space and 19% of the whole virtual disk.

For the second virtual machine, I backup a 1.2TB virtual disk with about 623GB used space.  The backup took about 9 hours 46 minutes to complete. Only 392.7GB of disk storage was needed for the backup which is only 63% of the used space and 33% of the whole virtual disk.

I am happy and impressed with the amount of saving in storage in this initial test.  The bottleneck for the backup is at the source and I am sure it can be further improve by some tweaking on the source.

Since Veeam Backup & Replication 6.5 is out this week, I will upgrade the trial and give it a try.

Pin It

vCenter Converter - vSphere 5.1 Upgrade Post Activity

Few days back, I was trying to convert a physical machine to virtual on my upgraded vSphere 5.1 environment just to find out that the Import Machine option was not there.

Pin It

Install Dell OpenManage Server Administrator on ESXi 5.1 - vSphere 5.1 Upgrade Post Activity

After the upgrading from ESXi 4.1 to ESXi 5.1, the Dell OpenManage Server Administrator (OMSA) previously installed on ESXi 4.1 was gone because it is not compatible.  So I need to re-install OMSA on my now ESXI 5.1 hosts.

The installation steps for OMSA on ESXi 5.1 are quite similar to the ESXi 4.1 as described in the entry http://deinfotech.blogspot.com/2011/08/installing-dell-openmanage-server.html.  However, you need to download the latest Dell OpenManage Offline Bundle and VIB for ESXi and Dell OpenManage Server Administrator Managed Node.  As of this writing, the version is 7.1.0, A00 and you can download them from the Dell website.

Dell OpenManage Offline Bundle and VIB for ESXi
http://www.dell.com/support/drivers/us/en/555/DriverDetails?driverId=Y0WHR

Dell OpenManage Server Administrator Managed Node
http://www.dell.com/support/drivers/us/en/04/DriverDetails?driverId=30T1C&fileId=2962652264

The first step is to install the Dell OpenManage Offline Bundle and VIB for ESXi on the ESXi 5.1 host.  In ESXi 4.1, you can use the vihostupdate.pl to do the installation but you can’t do it with ESXi 5.1.  if you try doing it, you will get the following error.

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl --server es
xi02 -i -b c:\install\OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00
Enter username: root
Enter password:
This operation is NOT supported on 5.1.0 platform.

Pin It

Capacity IQ - vSphere 5.1 Upgrade Post Activity

After I have successfully upgraded my vCenter 4.1 to vCenter 5.1, I noticed that the Plug-in name and Description for Capacity IQ were blank.

Pin It

Upgrade vSphere 4.1 to 5.1 – VMFS Upgrade

This will be the final post on upgrading vSphere 4.1 to 5.1.  In this post, I will be upgrade my datastore from VMFS-3 to VMFS-5.

Before rushing to upgrade from VMFS-3 to VMFS-5, I recommend reading the following tech paper to have an understand of what you will be getting from the upgrade.

http://www.vmware.com/files/pdf/techpaper/VMFS-5_Upgrade_Considerations.pdf

The upgrade can be done online and it is non-disruptive.  There is no need to shutdown virtual machines having virtual disks on the datastore to be upgraded.  However, it is always recommended to backup the virtual machines before the upgrade.

To upgrade the datastore from VMFS-3 to VMFS-5, all the ESXi hosts accessing the datastore must be ESXi 5.0 and above.  Select the datastore to be upgraded and click on the Upgrade to VMFS-5 link.

Pin It