Best-Practices: Screenshot Verification

Datto appliances have several options for scheduling screenshot verification on an agent's snapshots. The Screenshot Verification settings can be configured on a per agent basis. The available selections are tied to the agent's backup schedule.

Backup schedule

The default backup schedule on Datto appliances follows an hourly interval, from 8 AM to 6 PM, Monday through Friday.

Datto recommends scheduling backups to run during the hours that the protected machine is in production. For example, if a server is operational between the hours of 8 AM and 8 PM, you should have 12 backups in that day, as we would want to take 1 backup every hour.

Using the Last point of the day screenshot option as an example, under this backup configuration, the verification would run right after the last recovery point of the day - 8 PM.

If your device has offsite replication service, Datto recommends using the default screenshot scheduling option of All Offsite Points to ensure that you have more than one bootable recovery point available in the Datto Cloud.

Screenshot schedule

The default schedule setting is All Offsite Points. If your device has offsite replication service, Datto recommends using the default screenshot scheduling option of All Offsite Points to ensure that you have more than one bootable recovery point available in the Datto Cloud.

NOTE  Do not use this option if your Datto appliance does not replicate offsite.

Based on the selection made, the screenshot scheduler process will add the corresponding snapshot to the screenshot queue. For example, the First point of the day option will tell the screenshot scheduler to queue screenshot verification on the first scheduled backup of the day.

The automated screenshot scheduler will only queue backups taken by the automated backup scheduler. If a scheduled backup for a screenshot verification is missed, that screenshot verification will not queue, and will be considered missed by the agent.

The Additional Wait Time setting determines how long the Datto appliance waits to take a screenshot after it detects a ready state in the test virtualization. This is useful for agents that take a long time to boot with the allotted resources.

Ready state refers to criteria detected on a running VM that indicates it has booted to an operating system. If the Datto appliance does not detect a ready state, it waits for 5 minutes, and then checks for a ready state again. When the ready state detection step ends, the Datto appliance waits for the period of time specified in the Additional Wait Time setting before taking the screenshot.

1. Test VM boots.
2. Ready state detection begins.
3. Checks to see if the ready state is detected, waits up to 5 minutes if not.
4. Wait for a set period of time (Additional wait time).
5. Captures a screenshot of the screen.

You should set the Additional Wait Time to the lowest possible value necessary to consistently capture a successful boot. If a machine is taking longer than normal to get to the boot, then checking the startup processes on the protected machine or windows updates is recommended before adjusting the wait time.

When a device is protecting many agents, setting custom schedules becomes essential for allowing screenshot verification to complete in an orderly fashion.

Total Verifications

A Datto appliance can perform between 22 and 240 verifications per day. No more than 25 screenshot verification sessions can be in the device queue at the same time.

At most, 10 agents can be verified per hour, because of the 5 - 6 minute minimum time necessary to complete the operation, for a theoretical total of 240 verifications a day. But due to a number of variables, including backups completing at slightly different times and scheduler processing, this number is often much lower.

If all agents need to have their Additional Wait Time set to 60 minutes, with the default 5-minute delay, each verification would take 65 minutes, for a total of 22 verifications per day.

Because of these considerations, Datto recommends spreading out the screenshot verification schedule. Try to evenly distribute screenshot sessions so that each agent is scheduled for a different hourly block. If you have more than 24 agents, begin by scheduling 2 verifications per block, then 3 per block and so forth. See the Full Verification Pass section of this article for considerations when devices need more than 24 hours to verify all of the agents.

Full Verification Pass

A full verification pass represents the total time needed to complete verification for all of a device's agents.

To calculate this value, configure all of your agents to the lowest additional wait time they need to capture a successful boot. Total all of the agents' additional wait time values, and add another 5 minutes per agent. The result is the total time your appliance will need for a full verification pass.

As an example, if you have 35 agents with 2 minutes of additional wait time each:

(35 x 2) + (35 x 5) = 70 + 175 = 145 Minutes

In this example, you would not need to set up a complex schedule, as the day's verification completes quickly.

However, if this result is greater than 24 hours, then you will encounter a situation where agents add the next day's backups to the screenshot queue before the queue has finished. This can result in “Screenshot has not occurred alerts.

To fix this, you will need to set up a custom schedule that reflects this total time.

For example, if the total time is 48 hours, set a custom schedule for all agents to screenshot every other day, and configure the Screenshot Has Not Occurred Notification to 60 hours. If configured correctly, all agents will be scheduled to take a single screenshot within the window of the full verification pass.

Windows updates

Pending windows updates can cause screenshots to fail. You should install any Windows updates that have been downloaded and are pending install on your system.