Veeam Backup for AWS: Backup Appliance Architecture and Deployment

We continue our series of posts on Veeam Backup for AWS (VBA) and we look at its architecture and deployment process. Veeam Backup for AWS appliance coordinates backup and recovery tasks, controls scheduling, notifications, manages retention tasks and controls deployment of additional components used during backup and restore operations. 

Veeam Backup for AWS appliance is a Linux-based EC2 instance running several services:

• Backup service - coordinates backup and recovery operations.
• Configuration database (PostgreSQL) - stores configuration of backup policies, worker instance, added IAM roles, sessions and so on
• Configuration restore service — allows to back up and restore the configuration of the backup appliance. While connected to a VBR server, the configuration backup will be configured and executed from VBR console.
• Web UI  - web interface to access VBA
• Veeam Updater service - install product and package updates
• Veeam FLR service — restore individual files and folders of protected EC2 instances
• Self Backup service — allows Veeam Backup for AWS to back up and restore the configuration database of the backup appliance.
• REST API service

Veeam Backup & Replication server (VBR) centrally manages VBA appliances. One or more instances can be managed from the same VBR allowing for multi-region deployment. The backup server (VBR) can be deployed either in AWS or on-premises. Backup server uses Veeam Plugin for AWS to connect to AWS services and manage the lifecycle of Veeam Backup for AWS components. 

To start VBA deployment from VBR  make sure you have access to an AWS account. During deployment process, Veeam Plugin for AWS connects to multiple AWS APIs and services:

• AWS IAM  - create IAM roles for VBA (impersonation role and backup/restore access role)
• Amazon EC2 - deploy the appliance from an EC2 Ubuntu image and create networking if necessary 
• AWS SSM - run scripts and configure the VBA appliance 
• Amazon S3 - deployment scripts and binaries
• Amazon KMS 
• Amazon STS 
• CloudWatch 
• Service Quota

You can choose to connect to an existing appliance or deploy a new one. Next add AWS account access key and secret key. Remember the account needs permissions to the above services. See below a list of all the APIs called during installation:

• cloudtrail.amazonaws.com
• config.amazonaws.com
• ec2.amazonaws.com
• kms.amazonaws.com
• monitoring.amazonaws.com
• notifications.amazonaws.com
• resource-explorer-2.amazonaws.com
• s3.amazonaws.com
• servicequotas.amazonaws.com
• ssm.amazonaws.com
• sso.amazonaws.com
• sts.amazonaws.com

Select the region where you want to deploy VBA, enter an instance name for the appliance and choose how to connect to it - dynamic public IP, static public IP or private IP. In the  case you choose private IP you need to make sure you have access to that IP range. The deployment process can also create networking infrastructure if it does not exist - VPC, subnets, security groups, internet gateway and so on. Finally enter the guest OS administrator credentials and the key pair to encrypt credentials with. 

The deployment process automatically goes through several stages: 

• get and deploy an EC2 Ubuntu image 
• create IAM roles
• prepare the instance (deploy and configure the required binaries in the appliance)
• create administrator credentials for the appliance 
• initialize services in the appliances
• check for updates for Ubuntu Linux, Microsoft Windows (ASP.NET Core Runtime) and PostgreSQL
• install updates 
• reboot appliance
• add permissions to IAM role

The installation can take some time, so be patient. During the installation IAM roles are automatically created.

"Impersonation role" is attached to the backup appliance and it is used to assume other IAM roles. It has the following permissions:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": "sts:AssumeRole",

            "Resource": "*"

        }

    ]

}

and the following trust relationship:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Principal": {

                "Service": "ec2.amazonaws.com"

            },

            "Action": "sts:AssumeRole"

        }

    ]

}

Another 2 roles are created: "VeeamInstanceBackupRestoreAccessRole" and "VcbSnapshotsRole"

VeeamInstanceBackupRestoreAccessRole - is the default role that will allow VBA to perform all backup and restore operations. It must be granted permissions to assume the IAM roles:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Principal": {

                "Service": [

                    "ec2.amazonaws.com",

                    "backup.amazonaws.com",

                    "batchoperations.s3.amazonaws.com"

                ],

                "AWS": "arn:aws:iam::1111111:role/VeeamImpersonationRole"

            },

            "Action": "sts:AssumeRole"

        }

    ]

}

The permissions list is pretty long and it can be found here. In IAM a series of policies are created and named with vbaws_policy_* and vcb_policy_* in order to add all required permissions. 

VcbSnapshotsRole - gives permissions to manage ec2 snapshots and it also needs assume permissions:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Principal": {

                "Service": "dlm.amazonaws.com"

            },

            "Action": "sts:AssumeRole"

        }

    ]

}

After VBA appliance is deployed, you may choose to create and configure a S3 bucket to use as a backup repository and start the wizard from the VBR console or open VBA console and start configuring the appliance. 

Veeam Replication - Automatically Fix Detected Invalid Snapshot Configuration

This one comes directly from the field. It is about  that moment when something goes wrong at a client's site and you need to fix. In this case, replicas went wrong. The link between the two sites not being reliable at all and some other networking issues at the DR site (yes, it's always network fault) determined replica VMs to end up in a corrupted state. If you want to keep the replica VMs that are already in the DR site then the fix is pretty simple: remove snapshots and remap vmdk's still pointing to delta files. Manually doing this is cumbersome, especially since some VMs can have multiple disks and when we are talking about tens or hundreds of  VMs. Luckily, some things can be scripted.

We will present some of the modules in the script, since the whole script is published here on GitHub. For ease of readability we will remove some of the try-catch blocks and logging messages from the original script and comment here only the logical part.

Script is given as input parameters vCenter Server where VM replicas are, backup server hostname, replica job status, replica job fail message and replica suffix. The replica suffix is important since it uses it to find VMs

$vbrServer = "vbr1"
$vcServer = "vc1"
$status = "Failed"
$reason = "Detected an invalid snapshot configuration."
$replicaSuffix = "_replica"

The script needs to be executed from a place where both vCenter Server and Veeam backup server are reachable and where PowerCLI module is imported as well as Veeam PowerShell snapin.

Add-PSSnapIn -Name VeeamPSSnapin
Connect-VBRServer -Server $vbrServer
Connect-VIServer -Server $vcServer

Next, the script will get all VMs for which the replication job has failed ($status) with the given reason ($status). For this, we use Get-VBRJob cmdlet, FindLastSession() and GetTaskSessions() methods. Once the VM in a replica job matches the chosen criteria, it i added to an array ($vmList)

$vmList = @()
# get failed replica VM names 
$jobs = Get-VBRJob  | Where {$_.JobType -eq "Replica"}
foreach($job in $jobs)
{
 $session = $job.FindLastSession()
 if(!$session){continue;}
 $tasks = $session.GetTaskSessions() 
 $tasks | foreach { 
        if (($_.Status -eq $status) -and ($_.Info.Reason -match $reason)) {
            $vmList += $_.Name
            }
        }
}

Once we have the list of VMs who's replica failed, it's time to get dirty. The VM name we are looking for is made of the original VM name (from the array) and the replica suffix.

$replicaName = $_ + $replicaSuffix

First, delete the snapshots. Fot this we use PowerCLI cmdlets: Get-VM, Get-Snapshots, Remove-Snapshot.

$replica = Get-VM -Name $replicaName -ea Stop
$replica | Get-Snapshot -ea Stop | Sort-Object -Property Created | Select -First 1 | Remove-Snapshot -RemoveChildren -Confirm:$false -ea Stop

Next, remap the disks if they are still pointing to the delta file. In order to do that, we get all the disks for the replica VM (Get-HardDisk) and we check if the disk name of the replica VM contains the specific delta file characters ("-0000"). This is how we determine if it's a delta disk or a normal disk. Delta disk name is parsed to generate the source disk name (removing the delta characters from vmdk name). Once this is done, it's just a matter of reattaching the source disk to the VM (Remove-HardDisk, New-HardDisk)

# get disks for a replica VM
$disk = $replica |  Get-HardDisk -ea Stop
# process each disk 
$disk | foreach {
    $diskPath = $_.Filename
    # check if disk is delta file
    if ($diskPath -Match "-0000") {
        $diskPath = $_.Filename
        # for each delta file parse the original vmdk name
        $sourceDisk = $diskPath.substring(0,$diskPath.length-12) + ".vmdk"
        # get the datastore where the delta is 
        $datastore = Get-Datastore -Id $_.ExtensionData.Backing.Datastore
        # check the original vmdk still exists on that datastore
        if (Get-HardDisk -Datastore $datastore.Name -DatastorePath $sourceDisk) {
            # remove delta
            Remove-HardDisk -HardDisk $_ -Confirm:$false -ea stop
            # attach original disk
            $newDisk = New-HardDisk -VM $replica -DiskPath $sourceDisk -ea Stop
        } else {
            Write-Host "WARN Could not find $($sourceDisk) on $($datastore.Name) "
            Add-Content -Path  $logFile -Value "WARN: Could not find $($sourceDisk) on $($datastore.Name) "
        }
    }
}

Last thing to do is to consolidate disks. We run Get-VM and check if consolidation is needed. If it's needed, we just run ConsolidateVMDisks_Task() method.

$vm = Get-VM -Name $replicaName -ea Stop
if ($vm.Extensiondata.Runtime.ConsolidationNeeded) {
    $vm.ExtensionData.ConsolidateVMDisks_Task()
}

Now the replica VMs are re-usable. There is some manual job to be done, though. You need to map the replica VM in the replication job and run the job.

Veeam Backup and Replication - Infinidat Integration

Starting with Update 3 (U3), Veeam Backup & Replication (VBR) offers a built-in integration framework for storage systems called Universal Storage Integration API. Storage vendors can use the API to develop plugins and integrate their storage systems with VBR. This is a huge step into extending the ecosystem of storage vendors that offer advanced functionality with VBR.

As of the writing of this article,  the following storage systems are already supported:

• IBM Spectrum Virtualize (since December)
• INFINIDAT InfiniBox (since March)
• Pure Storage FlashArray (since April) 

(Later edit: first storage vendor supported via the API was IBM, but it is integrated in U3 release)

The following article presents the integration with Infinidat InfiniBox.

I will not go into the details of installation and configuration of InfiniBox. The post covers only Veeam part

First, download the Infinidat plugin from Veeam site. Login to VBR server, close VBR console, make sure processes/jobs are not running, extract the zipped file and run the installer. If you connect remotely to VBR, then run the installer on the machines from where you connect, too. It's a next, next, next process (as seen in the following series of pictures):

Now, getting to the fun part. Open VBR console, go to Storage Infrastructure > Add Storage - there it is, Infinidat Infinibox is available:

Add storage hostname (or IP address):

Add storage credentials:

Select the protocol to use (in lab I am using only iSCSI), the volumes to scan and the proxies to use (I have left everything on automatic):

Review the summary, press Finish and wait for the successful installation:

It is time to test the newly configured storage. First, let's create a job that uses Backup from Storage Snapshots. Since using integrated storage snapshots is active by default, there is no need to configure anything special for this job. Just select the VMs from InfiniBox datastore and run the job. Looking at the logs we see that "storage snapshot" is being used for backup:

Another type of job is to use only snapshots, although this is not a proper backup solution (since both the source VM and the backup reside on the same storage). To make a backup job snapshot only, you need to select as destination repository the InfiniBox storage (not a VBR repository):

This time when the job runs, it will create a snapshot directly on the storage:

The snapshots will appear under Home - Backups:

As well as under Storage Infrastructure:

Now, it's time for you to test the recovery :-)