VSS Errors

Topic

VSS (Volume Shadowcopy Services) is a core component of windows that permits backup operators to safely protect a windows system while it is running. If the VSS service cannot operate properly, it will produce specific errors visible to and collected by Datto, as well as log events in the Windows Event Viewer, and result in backup failure.

Environment

  • Microsoft Windows
  • Datto Windows Agent
  • Datto Endpoint Backup Agent

Description

VSS failures can sometimes be trivially solved and are transient, in other cases thay are related to improper deployment or configuration of the Windows OS or applications within, and sometimes indicate catastrophic damage to Windows has occurred often requiring rollback to prior known good states, disaster recovery, or complete server rebuild. Though a restart of services or Windows itself will often temporarily solve a VSS issue, understand that if the underlying condition or event that caused VSS to enter a failed state re-occurs, the VSS error will also reoccur.

If VSS is non-functional, Datto will be unable to take application aware backups of the server until the issue is resolved. Performing a backup does not cause VSS errors (though the act of attempting a snapshot will often have the result of encountering the error). No custom code is used by Datto to manage VSS, it is a native core component of Microsoft's OS. When VSS has failed it is because Windows itself cannot internally perform the requested operation. No code or settings changes to the agent can resolve these issues. They typically require repairing the OS itself or addressing infrastructure concerns.

Resolutions and Troubleshooting

Sometimes just simple steps like the following will help to resolve issues with writers:

  1. Ensure no backups or restores are running (no wbps.exe processes should be active in windows).

    • If performing a VM backup, ensure no VM snapshots are present and the VM is not currently migrating to alternate storage and that VM and in-guest backups are not both scheduled.
  2. Stop the impacted VSS Writer Service in windows, for example the VSS SQL Writer.

  3. Stop VSS core Service itself.

  4. Wait 30 sec.

  5. Start VSS Service.

  6. Start impacted VSS Writer Service.

  7. List the writers by running the command (in a command prompt on the local machine).

    vssadmin list writers

  8. Verify, if any errors related to writer.

  9. Re-attempt the backup operation.

  10. In case of errors check System and Application Events Log and see VSS and impacted VSS Writer error events and warning for more information and attempt to remediate.

  11. if errors persist, reboot the OS and retry steps 7-10.

Windows Hot Fixes

Datto assumes all currently available windows patches and relevant hotfixes are applied to servers. This will also include updates for Microsoft applications like SQL and Exchange and manufacturer drivers as well. Additionally, after windows updates run it is often critical to ensure a reboot occurs if any fixes require a reboot. As windows updates without rebooting are commonly done, this is the largest reason why a simple reboot tends to resolve most VSS errors, but this is often symptomatic of failing to properly manage windows updates and reboots.

Diagnostic Information for Microsoft Assistance

When it is determined that the VSS failure is outside the scope of Datto Technical Support, then the following information should be gathered for Microsoft support to examine:

  • Windows application event log
  • Windows system event log
  • VSS trace (see instructions below)

Event logs:

Examine the application and system event logs focusing on the error events created by the VolSnap and VSS sources at the time of failure. It is helpful to extract the germane events from the log to isolate the problem and have a more productive interaction with MS support.

How to perform a VSS trace:

vsstrace is a tool provided by Microsoft as part of the windows software developer toolkit (SDK). This is an advanced tool used for VSS data collection to be used with Microsoft senior support when investigating windows defects or damage. This tool and it's data are not used by Datto, but we recommend collection of this data for any customer who is advised to engage Microsoft support to determine the nature of their VSS errors.

The SDK is available for most versions of windows here and must be first installed before vsstrace can be used: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/ (external link)

Instructions for using vsstrace are here: https://learn.microsoft.com/en-us/windows/win32/vss/using-tracing-tools-with-vss (external link)

I/O performance diagnosis:

Use Microsoft I/O performance tools to gather data for analysis. Check disk defragmentation. Checking health of writers, the Datto Windows Agent uses the VSS interface to read SQL data from the disk. If there is a problem with the VSS SQL writer responsible for SQL, use the Microsoft utility vssadmin.exe to ensure that the SQL writer is available on the system if a problem is suspected.

Additional Resources