VM in new host slower than VM in older host

Few months back, I led a project to upgrade an in-house application to a newer version running on new hardware.  New ESXi hosts with better configuration were added to the existing cluster and new Virtual Machines were created for the new version of the application.

One of the objectives of the upgrade is to increase the performance of the application based on the fact that the new VMs will be on new and more powerful hardware. 

So the team conducted performance testing but was getting some weird results.

Pin It

Create Dell DR4000 CIFS Disk Storage in Symantec Backup Exec 2012

I was trying to create a Disk Storage in Symantec Backup Exec 2012 pointing to a Dell DR4000 CIFS share.  I go through the steps in creating Disk Storage in Symantec Backup Exec 2012.

I chose Disk-based storage for the type of storage.

Pin It

Dell Latitude 10 Tablet

I was given the opportunity to get my hand on a demo unit of Dell Latitude 10 Tablet installed with Windows 8 Pro.  The tablet is powered by Intel Atom Processor Z2760 and weighs 660g.  There is no dockable keyboard like the Microsoft Surface.  There is a docking station for the tablet where I can connect up keyboard but I find the docking connector a bit flimsy.  The configuration that I get also comes with micro-SIM slot.  The speaker is located at lower right corner of the back and my right hand tends to cover it while holding it.  Overall, it is still a decent tablet which allows I can get some works done on the go however an option with a dockable keyboard and shifting the speaker up will be good.

Before returning the demo unit, I reset it back to the default factory settings and it is really easy with Windows 8.  Here are the steps to do the reset.

Slide from right to left to call out the charm bar.  Touch on Settings.

Pin It

Custom ESXi 5.1 Image for Dell R720

As mentioned in my previous entry, I have decided to create a custom ESXi 5.1 image with the driver for Integrated Broadcom 5720 quad port network adapter included.  Here are the steps I took to create the custom image.

I will need to have VMware vSphere PowerCLI installed on the machine I will be using to create the image.

Download the ESXi 5.1 software depot (ESXi 5.1 Offline Bundle).  The file that I have downloaded is ESXi510-201210001.zip.

The offline bundle for the Broadcom 5720 network adapter driver is also needed.  The offline bundle which is tg3-3.124c.v50.1-offline_bundle-841079.zip is included in the downloaded tg3-3.124c.v50.1-841079.zip file.

Once I have everything ready, the first step is to import the software depots using the Add-EsxSoftwareDepot command.

I imported the ESXi 5.1 software depot (ESXi510-201210001.zip) follow by the Broadcom driver depot (tg3-3.124c.v50.1-841079.zip).

Pin It

Dell R720 NIC not detected by ESXi 5.1

Last week I installed VMware ESXi 5.1 on a new Dell PowerEdge R720 server.  After the installation, I was trying to the network settings and was “shocked” when the Integrated Broadcom 5720 quad port network adapter was not detected.  Well, it was not detected because the driver for the network adapter was not an inbox driver included in the ESXi 5.1 image.

To get the network adapter to work, I downloaded the driver from https://my.vmware.com/group/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/5_1#drivers_tools.  The driver version that I have downloaded is 3.124c.v50.1 and the file name is tg3-3.124c.v50.1-841079.zip.

After downloading the driver, I can either install the VIB file on the existing ESXi 5.1 or I can create a custom ESXi 5.1 image with the driver included. 

Pin It

Install Dell OpenManage Server Administrator on ESXi 5.1 - vSphere 5.1 Upgrade Post Activity

After the upgrading from ESXi 4.1 to ESXi 5.1, the Dell OpenManage Server Administrator (OMSA) previously installed on ESXi 4.1 was gone because it is not compatible.  So I need to re-install OMSA on my now ESXI 5.1 hosts.

The installation steps for OMSA on ESXi 5.1 are quite similar to the ESXi 4.1 as described in the entry http://deinfotech.blogspot.com/2011/08/installing-dell-openmanage-server.html.  However, you need to download the latest Dell OpenManage Offline Bundle and VIB for ESXi and Dell OpenManage Server Administrator Managed Node.  As of this writing, the version is 7.1.0, A00 and you can download them from the Dell website.

Dell OpenManage Offline Bundle and VIB for ESXi
http://www.dell.com/support/drivers/us/en/555/DriverDetails?driverId=Y0WHR

Dell OpenManage Server Administrator Managed Node
http://www.dell.com/support/drivers/us/en/04/DriverDetails?driverId=30T1C&fileId=2962652264

The first step is to install the Dell OpenManage Offline Bundle and VIB for ESXi on the ESXi 5.1 host.  In ESXi 4.1, you can use the vihostupdate.pl to do the installation but you can’t do it with ESXi 5.1.  if you try doing it, you will get the following error.

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl --server es
xi02 -i -b c:\install\OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00
Enter username: root
Enter password:
This operation is NOT supported on 5.1.0 platform.

Pin It

PERC 4e/Di Memory/battery problems were detected

One of my Dell PowerEdge Server rebooted unexpectedly this morning and was having problem booting up.

The PERC 4e/Di controller after spinning the disks, threw out the following POST message.

Memory/battery problems were detected.
The adapter has recovered, but cached data was lost.
Press any key to continue.

I pressed any key to continue and the Windows Server tried to boot up.  But before it can get to the logon screen, it rebooted again.  Tried a few times but it just keep rebooting.

To make matter worse, the server is out of warranty.  Fortunately, I found a spare server which is same model.  I took out the PERC’s battery and RAID DIMM from the spare server.

Pin It

Dell IT Assistant and High CPU Usage

One of my virtual machine running on ESXi 4.1 was having constant 100% CPU usage and that triggered an alarm on the vCenter.  The virtual machine was configured with 2 vCPU (~1.86GHz per vCPU).  Just to confirm that there was no contention on the physical CPU, I checked on the CPU Ready counter of the virtual machine and the CPU Ready is almost 0%.

This leaded me to suspect that the application running on the virtual machine is causing the high CPU usage.  From the Windows Task Manager, the process DSM_ITA_Netmon32.exe was using 100% of the CPU resource.  The DSM_ITA_Netmon32.exe process is part of Dell IT Assistant. 

I logged on to the Dell IT Assistant and saw some orphaned servers listed in the Devices view.  I did some cleaning up by removing those orphaned servers from IT Assistant.  Then I restarted the DSM IT Assistant Network Monitor service to bring the CPU usage to normal level and did a manual status polling and inventory on all the include ranges.  The inventory on one of the range caused the CPU usage to go up to 100% and stayed there.  The inventory for that range did not complete and hanged at about 70%.

To find out which server(s) is/are causing the problem, I did a inventory refresh on all the servers.  The CPU usage went up to 100% again with 10 more servers remaining to be inventoried.  From the list of 10 servers, I noticed that 2 of them were being virtualized via P2V.  I removed both servers from the device list since they are not on physical server anymore.  I did an inventory on the range which was having problem again and it completed successful without causing the 100% CPU usage.

Half a day has passed since the problem has been resolved and the virtual machine is not having the high CPU usage issue.  As you can see from the performance chart below, the maximum usage 4007MHz happened during the 100% CPU usage.  Now that the problem has been resolved, the CPU usage is around 50%.

Pin It

Aftermath of motherboard replacement

In my previous entry Tale of a BIOS update, the motherboard of my Dell R810 server has to be replaced due to BIOS upgrade issue.  The motherboard replacement solved the BIOS issue but it is not the end of the story.

As part of the motherboard replacement, the on-board quad port Broadcom network card has been replaced as well.  Since the server is running ESXi, the vmnic numbering has been messed up  (e.g. vmnic1 becomes vmnic9).  Although, this does not cause any functional issue but it is really an eyesore and might cause confusion.  So I edited the /etc/vmware/esx.conf to manually re-number the vmnic.  Here is a link that provide some information on how to do it.  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1971044.

The next problem surfaced when I accessed the server’s iDRAC (Integrated Dell Remote Access Controller) using Firefox.  I got a certificate error as shown below.

Pin It

Tale of a BIOS update

Last week, I decided to perform an urgent BIOS update for my Dell R810 servers.  The version of the urgent BIOS update is 2.7.0 which was released April and the issues fixed are documented in http://downloads.dell.com/FOLDER00394755M/1/R810-020700BIOS.txt.  However, the latest version is 2.7.4 so obviously it makes sense to go for the latest version.

Since all my Dell R810 servers are running ESXi, I downloaded the non-packaged which needs to be executed in the DOS environment. The file is close to 2MB which is too big for 1.44MB floppy disk.  So I created a bootable USB flash drive using the Dell Diagnostic Utility and copy the BIOS file into the USB stick.

Pin It