Hello All,
Our Environment
Windows 2012 R2 Datacenter Hyperv V 4 node cluster
All VMs are stored on CSV's which are SAN disks from HP SAN
The backup tool is netbackup V 7.6.
One VM which has Windows 7 Service pack 1 OS is failing backups since 2nd August.
No changes on VM prior to backups staring to fail.
All other VM backups are success( Including other Windows 7 VMs)
Error in netbackup
24/08/2016 10:53:11 - Info bpfis(pid=3972) done. status: 4207: Could not fetch snapshot metadata or state files
snapshot error encountered(156)
I found that below mentioned VSS writers on VM keep failing when the backup is run
Com+ RegDB, WMI, Shadow copy optimization and Windows search writers keep failing
I have done the following troubleshooting to try and resolve the issue , but no resolution so far.
==============================================================================
2) Installed below mentioned hotfixes
https://support.microsoft.com/en-us/kb/2996928
Backup task fails with a time-out error in Windows
https://support.microsoft.com/en-us/kb/3121255
"0x00000024" Stop error in FsRtlNotifyFilterReportChange and VSS backup of PI Data server fails in Windows
https://support.microsoft.com/en-us/kb/2505454
Volsnap
3) Re registered all VSS DLL's
rem
net stop "System Event Notification Service"
net stop "Background Intelligent Transfer Service"
net stop "COM+ Event System"
net stop "Microsoft Software Shadow Copy Provider"
net stop "Volume Shadow Copy"
cd /d %windir%\system32
net stop vss
net stop swprv
regsvr32 /s ATL.DLL
regsvr32 /s comsvcs.DLL
regsvr32 /s credui.DLL
regsvr32 /s CRYPTNET.DLL
regsvr32 /s CRYPTUI.DLL
regsvr32 /s dhcpqec.DLL
regsvr32 /s dssenh.DLL
regsvr32 /s eapqec.DLL
regsvr32 /s esscli.DLL
regsvr32 /s FastProx.DLL
regsvr32 /s FirewallAPI.DLL
regsvr32 /s kmsvc.DLL
regsvr32 /s lsmproxy.DLL
regsvr32 /s MSCTF.DLL
regsvr32 /s msi.DLL
regsvr32 /s msxml3.DLL
regsvr32 /s ncprov.DLL
regsvr32 /s ole32.DLL
regsvr32 /s OLEACC.DLL
regsvr32 /s OLEAUT32.DLL
regsvr32 /s PROPSYS.DLL
regsvr32 /s QAgent.DLL
regsvr32 /s qagentrt.DLL
regsvr32 /s QUtil.DLL
regsvr32 /s raschap.DLL
regsvr32 /s RASQEC.DLL
regsvr32 /s rastls.DLL
regsvr32 /s repdrvfs.DLL
regsvr32 /s RPCRT4.DLL
regsvr32 /s rsaenh.DLL
regsvr32 /s SHELL32.DLL
regsvr32 /s shsvcs.DLL
regsvr32 /s /i swprv.DLL
regsvr32 /s tschannel.DLL
regsvr32 /s USERENV.DLL
regsvr32 /s vss_ps.DLL
regsvr32 /s wbemcons.DLL
regsvr32 /s wbemcore.DLL
regsvr32 /s wbemess.DLL
regsvr32 /s wbemsvc.DLL
regsvr32 /s WINHTTP.DLL
regsvr32 /s WINTRUST.DLL
regsvr32 /s wmiprvsd.DLL
regsvr32 /s wmisvc.DLL
regsvr32 /s wmiutils.DLL
regsvr32 /s wuaueng.DLL
sfc /SCANFILE=%windir%\system32\catsrv.DLL
sfc /SCANFILE=%windir%\system32\catsrvut.DLL
sfc /SCANFILE=%windir%\system32\CLBCatQ.DLL
net start "COM+ Event System
4) Deleted old shadow copies, restarted the VM multiple times
5) Resized and reverted shadow copy space allocation on C drive, currently set to unbounded.
6) Deleted old registry entries referencing old shadow copies using DevNodeClean-x32
7) updated windows disk signature for the disk in VM. -- MBRFIX
8) Ran VSS Trace on Hyper V host and VM, have not been able to deduce much from it.
9) installed MS https://support.microsoft.com/en-gb/kb/2627052 for Volsnap on VM.
10) Readded registry keys from working machine to key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\Providers.
11) Disabled ASR
12) Deleted registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{74600e39-7dc5-4567-a03b-f091d6c7b092}
Restarted this service 'Hyper-V Volume Shadow Copy Requestor on the VM
13) Deleted microsoft provider HKLM/System/CurrentControlSet/Services/VSS/Providers/GUIDS.
14 ) Added below registry key
HKLM/System/CurrentControlSet/Services/Volsnap
Right Side, New DWORD VALUE - "MinDiffAreaFileSize" - Modify and set the value in between "300-3000 MB" or any multiple
of 300 in between. Ideally we can set it to 3000 MB which works best if we have space.
Ref: http://support.microsoft.com/kb/826936
15) Restarted hyper V host and also restarted hyper V virtual machine service many times
========================================================================
Application event log of the VM has below errors
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PreFinalCommitSnapshots({816c0863-8f4e-4ed7-8f24-0689056b30dc}, 1) [hr = 0x800705b4, This operation returned because
the timeout period expired.
].
Operation:
Executing Asynchronous Operation
Context:
Current State: DoSnapshotSet
=================================================
Hyper V VMMS log entry
VSS writers inside virtual machine 'AFFECTED_VM' failed to perform BackupComplete to its shadow copy (VSS snapshot) set: A function call was made when the object was in an incorrect state for that function (0x80042301). (Virtual machine ID D17455D4-5EDB-49D8-85A0-1F9088376048)
==================================================
I have the latest VSS traces if anyone wants to take a look, please let me know of any suggestions or anything tha tI can try to make this work.
Thanks in anticipation.