Create Dell DR4000 CIFS Disk Storage in Symantec Backup Exec 2012

I was trying to create a Disk Storage in Symantec Backup Exec 2012 pointing to a Dell DR4000 CIFS share.  I go through the steps in creating Disk Storage in Symantec Backup Exec 2012.

I chose Disk-based storage for the type of storage.

Pin It

Backup Exec Remote Agent Crashed When Backing Up SharePoint 2010 Server Farm

I logged a case with Symantec Technical Support regarding Backup Exec Remote Agent crashing when I tried to backup SharePoint 2010 Server Farm in early April.

After a few attempts of verifying the configuration and troubleshooting, I was given a temporary workaround to bypass the SharePoint backup so that I can backup the content databases.  I have described how to do it in Bypassing Symantec Backup Exec 2012 SharePoint backup.

My case was escalated to Symantec Engineering Team with lots of logs collected.  In early May, Symantec had identified the cause of the issue was because my SharePoint 2010 Server Farm has more than 200 content databases.  The http://www.symantec.com/docs/TECH205286 Symantec article describes the issue.

Pin It

Backup Exec - Snapshot technology error (0xE000FE1F): The device cannot be found

In the entry Bypassing Symantec Backup Exec 2012 SharePoint backup, I mentioned that I am not able to backup my SharePoint 2010 farm and as a temporary workaround, I am backing up the content databases.  Symantec has been working on my case and I received a second debug binary from them yesterday to test it out.

After putting in the debug binary version of ndmpsrvr.dll into all my SharePoint and SQL servers, I started a test backup but it failed with the following error.

V-79-57344-65055 - Snapshot Technology: Initialization failure on: "SharePoint – sharepoint.deinfotech.com80\Content-DB 1 (SQLP01\SP_ROOTSITE)". Snapshot technology used: No.

Snapshot technology error (0xE000FE1F): The device cannot be found.

Check the Windows Event Viewer for details.

V-79-57344-65055 - Snapshot Technology: Initialization failure on: "SharePoint - sharepoint.deinfotech.com80\Content-DB 100 (SQLP01\Content_IT)". Snapshot technology used: No.

Snapshot technology error (0xE000FE1F): The device cannot be found.

Check the Windows Event Viewer for details.

Pin It

Free Symantec Backup Exec V-Ray Edition

Last week I have received two Symantec Backup Exec V-Ray Edition licences which also comes with one year of maintenance support.  Thanks to Matt Stephenson for informing me about the deal.  Also thanks to Iesa Behbehani for the help in getting the licences.

If you are a VMware vExpert, VMware Certified Professional or VMware Certified Instructor, you can get the free licences by submitting the request at http://bit.ly/BE_vExpert.

If you are a MVP, you can get the free licences by submitting the request at http://bit.ly/BE_MVP.

Pin It

Bypassing Symantec Backup Exec 2012 SharePoint backup

I am problem backing up my new SharePoint 2010 Farm using Symantec Backup Exec 2012 for the past few weeks and have been working Symantec support to resolve it.  The remote agent on the SQL Server crashes and the backup will fail.  The problem has yet to be resolved as Symantec Engineers are still analyzing the collected logs.

All my SharePoint and SQL servers are virtual machines running in VMware ESXi 5.1 and I thought of backing up the virtual machines for the time being but unfortunately ESXi 5.1 is still not supported by Backup Exec 2012 SP1 as described in http://www.symantec.com/connect/blogs/updated-quality-wins-every-time-vsphere-51-support.

So for the time being, I am backing up the SharePoint databases from the SQL Server.  However, when I tried to backup the databases in the SQL Server, Backup Exec is aware that SharePoint databases are hosted on the SQL Server.

Microsoft SharePoint Resources appears in the backup job properties.

Pin It

RAM Requirement for Backup Exec 2012 Deduplication

The minimum amount of RAM is 8 GB, which supports up to 5 TB of deduplicated data.

Note: This minimum requirement is in addition to the minimum requirements for Backup Exec.

For more than 5 TB of data, use the following calculation to determine the required amount of RAM:

1.5 GB x N, where N equals the number of TB of storage in the deduplicated volume. Up to 32 TB of data is supported.

Example: Use the following equation to calculate the required amount of RAM for 10 TB of data:

1.5 x 10 = 15 GB RAM

Source: Best practices for Backup Exec 2012 Deduplication Option

Pin It

Service Packs and Hotfixes for Backup Exec 2012

Summary of service packs and hotfixes released for Symantec Backup Exec 2012 provided by Symantec.

http://www.symantec.com/connect/articles/what-updates-service-packs-and-hotfixes-are-released-backup-exec-2012-version

Pin It

Hotfix for Backup Exec 2012 Backup Disk Storage Offline

The hotfix to fix the Backup Exec 2012 Backup Disk Storage going offline after database maintenance is out.  The hotfix also addresses many more other issues so do check it out if Backup Exec 2012 is giving you problem.

The hotfix is available at http://www.symantec.com/business/support/index?page=content&id=TECH199190.

Pin It

Oracle Database Backup Failed–Not in Backup Exec Server’s authentication list

A few days back, I configured a backup job for Oracle database in Symantec Backup Exec 2012.  I did a manual run after the configuration but the job failed with the following error.

The job failed with the following error: This server name is not in Backup Exec server's authentication list. Enter the server's fully qualified domain name and logon account into the Backup Exec server's authentication list. Server names in the authentication list are case-sensitive.

I was then searching high and low for the authentication list mentioned in the error message but just could not find it.  Tried googling but still can’t find where is the authentication list.  In the end, I did manage to find the authentication list with a little bit of exploring and testing.

The so called authentication list is actually in the Backup Exec Settings.  Click on the Backup Exec button, select Configuration and Settings and then Backup Exec Settings.

Pin It

Symantec Backup Exec 2012–Disk Storage Offline

After upgrading from Symantec Backup 2010 R3 to Symantec Backup 2012, I have been facing with the issue of Backup Disk Storage going offline daily.  The Backup Disk Storage is created on an iSCSI LUN with MPIO.  Although the Backup Disk Storage went offline in Symantec Backup Exec, it is still accessible by Windows.

I opened a case with Symantec support and after some troubleshooting and analysis, my issue is related to http://www.symantec.com/business/support/index?page=content&id=TECH189831.  The Disk Storage did go offline when Backup Exec Database Maintenance is completed.

Symantec support provided me a orphan fix which replaces the pvlsvr.exe and StorArMgmt.dll files.  After applying the orphan fix, the Disk Storage has been stable and did not go offline after the Backup Exec Database Maintenance.  I am advised not to apply any hotfixes to prevent overwriting of the pvlsvr.exe and StorArMgmt.dll files until the fix is publicly available.

Pin It

Backup Exec Desktop and Laptop Option is now a standalone product

With the introduction of Symantec Backup Exec 2012, Backup Exec Desktop and Laptop Option (DLO) is now a standalone product known as Symantec Desktop and Laptop Option 7.0.  Yes, replacing “Backup Exec” with “Symantec” in the name.

Now that it is a standalone product, I am able to install it without having to install Backup Exec Server first.  Furthermore, I am now able to consolidate all the DLO licenses that come with my five copies of Backup Exec Servers and used them on a single DLO server.  However, the major drawback with DLO is still the lack of deduplication functionality.

Here are some screen shots of the installation of Symantec Desktop and Laptop Option 7.0.

Pin It

File Server Migration (Data Migration) –Part 2 of 3

In my previous post, I talked about choosing and using DFS (Distributed File System) as our new file server design.  In this post, I would like to share how we did the data migration.  This might not be the best way to do data migration but it works for us.  In fact, the process is not complicated but just time consuming.

The data migration was a 3 steps process (restore, synchronise and final cut over) mainly using tape backup and robocopy.  The aim is to minimise disruption and make the migration as transparent as possible for the users.

Pin It

Virtual Machine Backup Caused Multiple Map Disk Region

Multiple “Map disk region” tasks kept showing up in the vSphere Client for a virtual machine after I started a backup job in the Backup Exec.

A new Map disk region task appeared almost every second and was kind of “scary” until I confirmed that the virtual machine is running fine.

Pin It

Backup Exec - Unable to open a disk of the virtual machine

In my earlier post “Backup Exec – VMware Backup Transport Mode”, I talked about how to configure Backup Exec to use “SAN – Use the SAN to move virtual disk data.” which requires presenting the SAN LUNs of the vSphere hosts to the backup server.

For testing purpose, I assigned just one LUN to the backup server and the “SAN – Use the SAN to move virtual disk data.” works well.  After confirming that it works, I assigned the rest of the LUNs to the backup server and create a backup job to backup virtual machines residing in the newly assigned LUNs but the job failed with the following error.

Pin It

Backup Exec–VMware Backup Transport Mode

My VMware backup using Backup Exec 2010 R3 has been using “NBD – Do not encrypt the virtual disk data for over-the-network transfer.” as the transport mode for the past month.  To improve job rate, I have decided to configure it to use “SAN – Use the SAN to move virtual disk data.”

However, for this transport mode to work, there are a couple of things to do.

Pin It

VM Backup–Failed to created a quiesced snapshot again!

I am again hit with the problem of backing up two virtual machines with the same error in my previous post Problem With Virtual Machine Backup–Part 2 of 2 but with a different cause.

The same error message shown up in vCenter.

Create virtual machine snapshot

Cannot create a quiesced snapshot because
the create snapshot operation exceeded the
time limit for holding off I/O in the frozen
virtual machine.

Pin It

Storage vMotion Failed

In my previous post “Problem With Virtual Machine Backup–Part 2 of 2”, I mentioned that I encountered an issue in trying to storage vMotion the configuration files to a new datastore.  This is the error message when I did the vMotion.

The method is disabled by “SYMC-FULL 29-10-2011 16:21”

I tried doing it with the virtual machine shutdown but the Migrate option was greyed out (disabled).

Checking the VPX_DISABLED_METHODS table in the Virtual Center database shows that backup is preventing the vMotion to happen.

I have verified that no backup job was running against the virtual machines during the attempt of vMotion.  So it seemed like an orphaned operation.  The solution is to re-inventory the affected virtual machines.

So I shutdown the affected virtual machines. Remove them from the inventory.

Add the virtual machines back to the inventory by navigating to the datastore and folder which contains the vmx file of the virtual machines.  Right click the vmx file and select “Add to Inventory”.

After the re-inventory, I am able to storage vMotion successfully.

Pin It

Problem With Virtual Machine Backup–Part 2 of 2

In part 1, I mentioned that I was still having problem backing up the second and third virtual machines. 

This was the error message from Backup Exec.

Job ended: Thursday, November 03, 2011 at 10:15:11 PM
Completed status: Failed
Final error: 0xe000959b - An attempt to take a snapshot of a virtual machine failed because it was unable to quiesce a file system.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-38299

This was the error message from vCenter.

Create virtual machine snapshot

Cannot create a quiesced snapshot because
the create snapshot operation exceeded the
time limit for holding off I/O in the frozen
virtual machine.

So I started troubleshooting the VSS since usually it is the culprit.  I checked to make sure SYNC driver and VMware snapshot provider were not installed.  Verified that VSS was working correctly using the VSSAdmin command and running the FREZEE.BAT.  I could not find anything wrong with the VSS.  Then I found the article Quiesced snapshots fail for Windows 2008 R2 guests and set disk.EnableUUID to false.  This will create crash consistent backup instead of application consistent backup.  However, this still does not work.

So I exported the ESXi host’s log to examine it and below were what I found.

In the  hostd log file, I saw Snapshot guest failed: The file is too big for the filesystem.

In the vmware.log file of the second and third virtual machines, I saw the following.

vmx| FILE: File_VMFSSupportsFileSize: Requested file size (1314259992576) larger than maximum supported filesystem file size (274877906944)
vmx| DiskLibCreateCustom: if your disk is on VMFS, you may consider increasing the block size.
….
vmx| SNAPSHOT: SnapshotBranch failed: The destination file system does not support large files (5).
vmx| [msg.checkpoint.save.fail2.std3] Error encountered while saving snapshot.
vmx| The destination file system does not support large files.

Bingo, that is the cause of the problem.  Both the virtual machines have virtual HDDs created in different datastores.  The problem was the datastore where the virtual machine’s vmx file (configuration file) resided was setup with a block size of 1MB.  By default the workingDir contains the vmx file and workingDir is the location where the snapshots are created.  With 1MB block size, the maximum virtual HDD size will be 256GB.  However, in the other datastores, there are virtual HDDs much bigger than 256GB and that caused the snapshot to fail.

According to the article Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported, I either change the workingDir to a datastore with enough block size or change the location of the virtual machine configuration files.  I chose the later and did a storage vMotion to migrate the configuration files to a new datastore with block size of 8MB.

Starting the storage vMotion while the virtual machine is still running.

Used the Advance interface to select what files to migrate and to which datastore.

By moving the configuration files to the new datatstore with the 8MB block size solved the backup issue.  The storage vMotion were not a smooth one and I should talk about it in my next post.

Pin It

Problem With Virtual Machine Backup–Part 1 of 2

I was trying to backup four virtual machines in using Symantec Backup Exec 2010 R3 with Agent for VMware Virtual Infrastructure but encountered some issues doing so.  The four virtual machines were hosted in vSphere ESXi 4.1 with update 1.

I setup the backup job for the four virtual machines and ran it.  When I checked on the backup the next day, it was still trying to take a snapshot after 16 hours.  I logon to the vCenter and it showed creating of snapshot was in progress for the first virtual machine.  The problem was that it is not moving and just stuck at 95%!

I had no choice but to cancelled the backup since it was going to be stuck at 95% forever.  After the backup had been cancelled, I started googling for issues related to virtual machine backup and there are tons of information out there.  Specifically for Backup Exec, it was mentioned that VMware Snapshot provider should not be installed and BE VSS Provider should be used instead.  Well, that was what I have.  It was also mentioned that for ESXi 4.0 and later, the FREEZE.BAT in the C:\Program Files\VMware\VMware Tools\backupScripts.d folder will be called before and after the snapshot process.  I ran the FREEZE.BAT on the virtual machine and it failed to stop the Volume Shadow Copy service.  The Volume Shadow Copy service was set to manual start so I guessed the backup must have started it but it went “haywire”.  I rebooted the virtual machine and ran FREEZE.BAT successfully.

However, the problem was still far from being resolved.  I ran the backup job again and it failed within a minute with the following error.

Job ended: Tuesday, November 01, 2011 at 9:12:18 AM
Completed status: Failed
Final error: 0xe0009574 - Unable to create a snapshot of the virtual machine. The virtual machine may be too busy to quiesce to take the snapshot.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-38260

The vCenter showed the following event.

“Create virtual machine snapshot”  “Another task is already in progress.”

I suspected that the previous snapshot was still in “progress” and will not end.  I followed the article  Collecting information about tasks in VMware ESX and ESXi to confirm it.

I logon to the ESXi host console, and enabled Local Tech Support.  Press Alt+F1 to switch to the console window.

Ran the command vim-cmd vimsvc/task_list and got an output similar to the following.

(ManagedObjectReference) [
   'vim.Task:haTask-162-vim.VirtualMachine.createSnapshot-3887',
…

Ran the command vim-cmd vmsvc/getallvms to confirm that the snapshot belonged to the virtual machine having the issue.

The solution is to restart the management agents.

I restarted the management agents without impacting the running virtual machines.  After restarting, I had to reconnect the ESXi host from the vCenter.

I ran the backup job again and the it successfully took a snapshot of the first virtual machine.  The backup for the first virtual machine completed successful and just when I was about to celebrate, it failed for the second and third virtual machines.  The fourth virtual machine completed successfully as well.

The issues for the second and third virtual machines were different for the first virtual machine so I will cover it in part 2.

Pin It

Errors Installing Symantec Backup Exec 2010 R2

I was trying to install to install Symantec Backup Exec 2010 R2 on a test server today and was really pissed off with the installation.  The test server is running Windows Server 2008 R2 x64.

I started by downloading the installation package from Symantec fileconnect.  Extract the ISO image to a folder and double click on Browser.exe and that was the start of my journey to this painful installation.  I received the following error message when I tried to run Browser.exe.

Hmmm… did I download a package for other operating system?

Okay, I decided to skip the Browser thingy and went straight to run setup.exe in the WinNT\Install\BE64.  It started well, but when it reached the Environment Check step, it threw out another error.

Found this on Symantec website  http://www.symantec.com/business/support/index?page=content&id=TECH124316.  Replaced the VxValidatorOps.dll but still no luck.

Anyway, if I click the OK button, I get the following screen and I can still choose to continue which I did after I gave up fixing the VxValidatorOps.dll error.

As I continue, the installation started and just as I thought everything is okay, error pop-up again.

Arghh.. . the ReportViewer.exe is not for x64!!!  I downloaded the ReportViewer.exe which support x64 from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=82833F27-081D-4B72-83EF-2836360A904D&displaylang=en and installed it.  I also replaced the ReportViewer.exe in the WinNT\Install\ReportViewer folder with the one I downloaded and re-run the setup.

The installation ran and again I was hit by another error.  The Microsoft Visual C++ 2005 SP1 Redistributable found in WinNT\Install\VCRedist is not for x64 although the file name is vcredist_x64.exe.  What the heck!  What is happening?

I downloaded the correct vcredist_x64.exe from http://www.microsoft.com/downloads/en/details.aspx?familyid=EB4EBE2D-33C0-4A47-9DD4-B9A6D7BD44DA&displaylang=en and  installed it.  I also replaced the copy in the WinNT\Install\VCRedist with the one I downloaded.  This is before I re-run the setup, I downloaded the SQL Express 2008 R2 from https://www.microsoft.com/betaexperience/pd/SQLEXPDBMT64/enus/ and installed it.  If you planned to use an existing SQL Server, there is no need to do so.  By installing SQL Express 2008 R2, .Net Framework 3.5 was also installed.  I have a feeling that the .Net Framework 3.5 provided by the package is also not for x64.

Okay, I re-run the setup and when prompted for the SQL Server, I choose to use the SQL Express 2008 R2 which I installed and everything went on smoothly.

Thanks to not having the correct components in the installation package (no excuse for this) that I have to go through all these hassle.

Pin It