Preparing for a cloud virtualization test: Policies, procedures, and partner responsibilities

Topic

This article discusses preliminary steps to take before submitting a request for a cloud DR test.

For overviews of available cloud restore processes, see the following articles:

Environment

  • Datto SIRIS
  • Datto ALTO
  • Datto Cloud
  • Datto Backup for Microsoft Azure
  • Datto Endpoint Backup with Disaster Recovery

Description

Preparing the Protected Machines

Datto provides partners the ability to test their virtual environments in the Datto cloud in order to better prepare them for an outage or disaster. Datto recommends that all partners include a cloud DR Test, at least once annually, for all of their organizations to ensure that production systems and associated network configurations can be supported in the Datto cloud. Test virtualizations can be created in the Recovery Launchpad on an as-needed basis, however, when you need to test multiple systems, it is recommended that you schedule a DR Test with Datto support.

At a high level, a typical DR Test will include the following steps:

  1. Prepare for the Cloud DR Test.
  2. Submit the Offsite Virtualization Request Form
  3. Consult with your Datto Support technician
  4. Verify and Document Cloud DR results
  5. Clean up DR networks and virtualizations from the Datto Cloud

Prepare for the DR Test

When preparing for a Cloud DR) test, make sure that the protected machines are in the best possible state:

Verify the local screenshots for all agents to be tested in the Cloud.

  • Ensure that all Windows updates have been successfully installed on the target machines.
  • The machines cleanly return from reboots.
  • The machines can pass disk checks (chkdsk /r is preferred).
  • The machines are generally in good health at an application and operating system level.
  • Ensure that the agents are up to date offsite or that the points being tested are offsite.

Local virtualization capabilities can help verify this information. If the machines fulfill these conditions, make sure that those points are replicated to the Datto Cloud.

If the machines do not have a good local screenshot verified by local virtualization, Datto Technical Support reserves the right to exclude that Virtual Machine from the test environment until a good screenshot has been taken and that point has been replicated to the cloud.

Because Universal VM backup can back up any platform, Datto Technical Support's ability to troubleshoot systems restored via this method may be limited

Things to consider when planning your Cloud DR test

Fire Drills

Datto strongly advises against “Firedrill” DR tests, which are DR tests performed on an organization’s production systems and networks. (Standard DR testing – recommended by Datto – is not done in a production setting, and therefore isolates risk from an organization’s business.)

The risks and damage that can stem from Firedrill DR testing is significant and include, by way of example and not limitation:

  • lost data, which is not recaptured in the production environment
  • IP confusion among servers, which can result in lost access, functionality, or even the shutdown of entire networks
  • Third parties responding to the Firedrill as if it were a true DR event
  • Broken integrations with third party applications and/or inadvertent DR responses from such third parties.

By proceeding with a requested Firedrill, you acknowledge these risks and accept the responsibility of any damage stemming from the Firedrill. You understand that Datto advises against the Firedrill, and Datto shall not be liable for any damage or claim based on your decision to DR Test in a Firedrill setting.

Proof of Concept

If this test is a proof-of-concept for an organization, build in setup time before presenting to the organization. Decide what you and the organization will consider a successful disaster recovery test and agree upon the conditions to be satisfied.

Networking Configuration

Remember that after the virtual machines are booted in the cloud, additional setup time will be required to configure the IP settings for the machines that are running, as well as application launches and confirmation of network resource availability.

Datto does not support IP address schemes outside of those specified for private use by the RFC1918 specification. (external link).

Security

For the security of the test, Datto's default rules deny all inbound traffic to these systems.

The only inbound traffic exception would be for Port Forwarding, which must be set up via the Recovery Launchpad.

System Resources

For testing purposes, Datto generally allows for a maximum of 16 GB of RAM and 8 CPUs per virtualized machine. Additional resources are subject to review and approval by Datto Support.

NOTE   Initial testing should be done to determine if we need to go above the maximum defaults for the resources. Finding the minimum resources the production machine could run on would be more beneficial to know for an actual Disaster Recovery scenario, rather than the production value of those resources. Larger values will cause a change in how we offload and load balance the virtualizations. If this was determined in a previous test, please include that information in the additional comment section.

NOTE  For DR environments consisting of 5 or more VMs, it may be necessary for Datto Support to migrate those systems to one or more cloud servers that have resources available to support those virtualizations. This could take significant setup time if the systems being migrated are large.

When testing the virtualizations in the cloud, partners should bring up the protected machines that their organizations would find necessary for a full fail-over. Having a mixed fail-over or a temporary fail-over is not the best test case.

Test Data

Datto does not preserve data generated or changed within the DR Test environment and all test data will be deleted at the conclusion of the test. If you have specific requirements to backup your test VMs, please reach out to your Datto Support representative to see what options are available.

Production Machine Credentials

All production machines must have local administrator accounts created and reflected in the backup chain prior to scheduling a Cloud DR test. This allows you to access the production VM in the event that your test does not include the Domain Controller, or in the event of an Active Directory failure. Datto does not have the ability to view login credentials for virtualizations of a production machine, cannot bypass the login procedure for any virtual machine, and cannot create local administrator accounts.

Note that any passwords or login credentials should never be included in a support ticket. Datto only requires that your internal documentation regarding passwords and credentials be checked for completeness before submitting a DR test. If your protected machine is encrypted, the encryption password must be documented and available to your DR contact.

VLAN limitations

Whenever possible Datto recommends that you create your cloud networks in advance of the test using the new Recovery Launchpad, keeping in mind that:

  • If you require multiple VLANs in the cloud for your Cloud DR test, internet access is required in order for these networks to communicate with each other.
  • If you need to restrict access to the internet for a protected machine or specific software applications, use software firewall or iptables rules from within the VM to make these restrictions.
  • Offsite VLANs for cloud VMs are limited to /20 or 255.255.240.0 to avoid using large sections of available addresses for a small amount of VMs.

Organization and Site-to-Site VPN tests

Performing Layer 2 and Layer 3 VPN testing is highly-complex and carries with it a risk of disrupting your production network if performed on an unsegregated LAN.

Datto will only assist with Layer 2 or Layer 3 site-to-site VPN testing if a fully-sandboxed network, isolated from the production environment, is available for the duration of the test. Support for these scenarios will be provided in a best-effort capacity. Datto recommends that:

  • Requested VPN keys are generated in Recovery Launchpad and verified prior to the test (if applicable)
  • Required IPSec configuration is verified and verified prior to the test (if applicable)

In both test scenarios and live disaster recovery scenarios, Technical Support is only responsible for the VPN configuration to the offsite server. All organization-side routing is the partner's responsibility.

Submit the Offsite Virtualization Request Form

Datto requires at least 7 business days in advance to properly prepare to support an CloudDR Test request. Note that the complexity of your test may impact your time to properly set up and configure the test. It is recommended to schedule more than 7 business days for complex networking requirements or tests that are larger than 10 protected systems.

To request a Cloud DR test, fill out the following form (requires a Datto Partner Portal account):

Datto Technical Support provides the following connectivity to the Cloud VM:

  • VNC: In browser VNC can be accessed from the Recovery Launchpad once the VM is powered on…….
  • VPN Connectivity: To have users connect as if they were on the LAN, you can utilize a Site-to-Site IPsec VPN tunnel back to your location or request VPN keys to use OpenVPN client. For instructions see Datto Partner Portal: Connecting to Cloud Networks with Datto VPN Keys
  • Public IP / Port Forwards: If the virtualized machines require access on a public IP, provide the ports that need to be forwarded, as well as the protocol (TCP or UDP).

Datto service includes one Cloud DR test per quarter. Cloud virtualizations are intended for temporary use during a recovery event or related testing exercise and should be removed when no longer needed. Test virtualizations and associated networks that have been running for more than 30 days may be powered off and subsequently deleted.

Datto's Responsibilities

When performing an Cloud DR test, Datto Technical Support will create the virtual machines according to partner configurations and will verify the following:

  • All Virtual Machines are started and running.
  • All virtualized machines have been booted into Windows / Linux

And, as applicable:

  • Protected machines are started with the proper network interface(s)
  • Protected machines are able to communicate with one another
  • That the virtualized machines are able to ping both the gateway of the cloud network and common external resources. This may require you to login and configure addressing.
  • That any VPN keys requested are provided and are able to access the network's gateway.
  • That any public IPs the test requires are properly routed.
  • Any machines requiring DHCP have received an address within the DHCP pool (if applicable)
  • Verification that any port forwarding request from public IPs are forwarding to the expected service port
  • Any public IPs the test requires are properly routed
  • Use Datto cloud and DR resources for the sole purpose of disaster preparedness testing and/or temporary use if the production environment becomes unavailable.

Partner Responsibilities

A Datto Partner is expected to manage the following items:

  • That prerequisites have been satisfied as outlined here, including completing the Offsite Virtualization Request Form.
  • That primary server services are online and operational.
  • That protected machines and domains are properly configured and authenticating with each other.
  • That primary server roles are configured and are properly functioning within the cloud environment.
  • Virtual Local Networks and Sites are created by the Partner before the test date (if applicable)
  • That any public IPs provided have their services accessible on these public IPs.
  • That, if requested, a network topology map can be provided to Datto Technical Support for troubleshooting purposes.
  • That, if requested, server roles can be identified and provided to Datto Technical Support.
  • Ensure any applications that access the internet will not conflict with the production environment during the test.
  • Site to Site IPsec tunnels are properly configured no local routing devices
  • Ensuring that all testing objectives and criteria have been met, and documenting as required for partner or end organization’s records.
  • Powering off and deleting all virtualizations, and associated networks.

Completing the Cloud DR Test Form

*indicates required entries

Prerequisites

You will have the best experience if you first review the information provided in this article and have verified that the restore can successfully be mounted locally and virtualized without any issues.

Next indicate if you just want Datto Support to assist in setting up the Cloud DR test or if you would like additional support assistance during the test.

Setup Information

Name/Email/Telephone Number of Primary Testing Contact*

Please provide the name and contact information for the person who will be administering the Cloud DR test.

Device Serial Number*

Please provide the serial number of the Datto device associated with this test. If you intend to test more than one device and/or will require more than one cloud network, please submit a separate request for each device or network.

Desired Testing Date*

Please provide the desired date for the test.

Estimated Duration of Test*

Please provide an estimate of how long these test systems will need to be running in the Datto cloud. Datto reserves the right to power off and remove restores that remain running after the completion of a DR test or event.

Virtualize snapshots from just before this point in time*

Select the date of the snapshot that you would like to test with. A good screenshot verification for the selected point is required. If the protected system has a volume that is separate from the primary OS volume and is being backed up by the Datto appliance, a successful offsite file restore for the selected point must be run.

NOTE  Test virtualizations are prepared in advance using the most current and verified screenshots already replicated to the Datto cloud at the time the test is being set up by Support - similar to an actual DR event. Because of the time required to replicate to the Datto cloud and prepare the virtualizations, including any additional data migration and troubleshooting, it is not possible to utilize snapshots taken on the day of the scheduled test.

In your mind, what constitutes a successful DR test in regards to system bootability, network connectivity, and remote access?

It is helpful if you can provide the success criteria for the test along with any other information or comments that would help administer the test. For example, are you simply testing bootability without any network connectivity, bootability plus network connectivity between the cloud virtualizations, and/or testing with reachability via port forwarding, organization VPN or site-to-site IPSec tunnels.

Network/Access Configuration

Network Name*

This is a unique name used to identify this network and the associated virtualizations. Please provide a name for this network similar to when you create a network in the Recovery Launchpad.

NOTE  If you will require more than one cloud network, please submit a separate DR request form for each network.

Network Address*

The IP address for this network, either in IPv4 format (e.g. 198.51.100.0) or IPv4 CIDR notation (e.g. 198.51.100.0/24).

Subnet Mask*

The subnet mask for this network in IPv4 between 255.255.240.0 (/20) and 255.255.255.255 (/32) or CIDR notation. Datto only supports VLANs with subnet masks between /20 through /32.

Offsite VM(s) require internet access*

Please indicate if this network and its associated virtualizations will need public Internet access.

Gateway Address

The gateway IP address in IPv4 format (e.g. 198.51.100.1). Required for networks with Internet access enabled.

Enable DHCP

Please indicate if this network requires Datto to provide Dynamic Host Configuration Protocol functionality. This will default to No, unless Yes is selected.

DHCP Start / End

Required if you have indicated Datto-provided DHCP above. Provide the Start and ending address of a valid DHCP IP range in IPv4 notation (e.g. 198.51.100.0).

# of Client VPN Tokens (Optional)

Please indicate if you will require client VPN tokens to complete your DR test. You may select up to 25 VPN connections. Default = 0.

Client VPN O/S Type

What type of operating systems will your VPN users require: Select from Windows, Linux or Mac

Client VPN Name

Enter a base Name for the requested VPN Tokens to easily identify these users in the Recovery Launchpad, if needed (e.g. xyzorganization, or SiteABC). This prefix will be added to each of your VPN user keys.

Utilize Network Groups (Coming Soon)

If you need this network to connect to other cloud networks submitted on related DR test forms, select Yes. Until Cloud Network Groups are available, connecting cloud VLANs will require Internet access. Note that bridging any Datto DR Test to your production network has the potential to cause network conflict and may result in data loss.

Group with (Coming Soon)

Enter the Network Name(s) of the other networks that should be connected to this DR network in the Datto cloud. For complex networks, a network topology map is helpful, including determination of any bridging requirements.

Site-to-Site IPSec VPN Tunnels Required

Please indicate if you will be setting up any Site-to-site IPSec VPN tunnels. (Note that Support will require you to configure these connections once the network is established.)

# of Agents to Virtualize

Enter up to 100 agents for testing and complete the following information for each of them.

Agent Name

Agent name as displayed in Datto Partner Portal

Number of CPUs

This will default to one (1) CPU core, unless specified otherwise. Requests for 12 or more cores will require Support approval and are subject to availability at the time of your DR test. Please include the reason for your request for extra resources in the Additional comments field below. Cores can be increased later, if needed.

Amount of RAM

This will default to 2GB RAM, unless specified otherwise. Requests for 32GB or more will require Support approval and are subject to availability at the time of your DR test. Please include the reason for your request for extra resources in the Additional comments field below. RAM can be increased later, if needed.

Network

This specifies the connectivity that this virtualization will have. Select VLAN for the virtualization to connect to the network you created above. Selecting Simple Networking will boot the VM with full outbound access to the internet. Selecting None will boot the VM with no networking ability of any kind.

Controller

Controller: Enables you to select the virtualization storage controller, which the restore will use. Available controllers are VirtIO (recommended), IDE, SATA, or SCSI. If you do not use VirtIO, the storage controller value should match the configuration in use on the production machine.

NIC

Allows you to define the NIC which the virtualization should use. Selectable NICs are E1000 (1 GB NIC) or rtl8139 (10/100 NIC) or VirtIO (recommended.)

Boot Time Zone

Select the time zone that you need the virtualization to boot from. Default is Americas/New York.

Boot Priority

Select the order/priority for booting this agent. If this must be the first system(s) available, such as a domain controller (DC), select First. You can also indicate High, Medium, Low or None as needed.

Application Type

Select the type of workload that this server will be running from the dropdown or select Other and enter a description. This data will help Support better understand your environment.

I/O Intensive

Check this box if you feel that this server/workload requires high-performance storage resources. Support will discuss this with you prior to the test, but this information will help identify possible resource constraints that could result in poor performance for this virtual environment.

Port Forwarding Information (Optional)

If you require port forwarding to public IPs for this virtual machine, please enter that information here. You need to specify the protocol (TCP and/or UDP) and the desired Ports (e.g. 80, 443) for each unique IP address. Note that you are currently limited to 1 Public IP per virtual machine, but will soon be able to request up to 3. When requesting additional public IPs, Datto Support may need additional preparation time to accommodate this request.

Additional Comments

Use this space for any additional comments or information related to this virtualization/agent.

Verify and Documenting the Outcome of your DR Test

Once all of your environments are up and running, you can begin performing any required testing or validation.

Datto highly recommends that you fully document the results of your DR test for future reference or in the event of an actual disaster or outage. Make sure that your fully capture

  • VM resource requirements - What was originally requested in the form? What was actually required for the VM to run satisfactorily?Protected machine role. System priority rating.Final IP address and VLAN. Any special requirements for booting the protected machine?
  • Networking configuration - How were your VMs networked in the cloud? What VPN connections were required? Final IP scheme configuration. Connections to other networks / VLANs. Any special requirements required for accessing the VLAN with the protected machines. Any special requirements required for communication between other networks / VLANs
  • Other considerations - Did you encounter any problems with any part of the test? What was done to correct or overcome the issues? For each issue encountered during troubleshooting the following should be documented:
    • Issue number
    • Protected System(s) impacted
    • VLAN impacted
    • Issue description
    • Issue cause
    • Resolution / Workaround
    • Additional notes (if applicable)

NOTE  DR Tests are simulations and may not always accurately reproduce how your systems will recover in the event of an actual live disaster recovery. Differences in resource availability, personnel and other factors can affect outcomes and recovery times.

Cleaning up after your DR Test

Once all of your testing is completed, please remove all cloud networks and virtualizations. As stated above, test virtualizations and associated networks that have been running for more than 30 days may be powered off and subsequently deleted. You should also be sure that all local network and VPN connections are disabled or removed.