Showing posts with label Replication. Show all posts
Showing posts with label Replication. Show all posts

Saturday, July 27, 2019

Veeam Replication - Automatically Fix Detected Invalid Snapshot Configuration

This one comes directly from the field. It is about  that moment when something goes wrong at a client's site and you need to fix. In this case, replicas went wrong. The link between the two sites not being reliable at all and some other networking issues at the DR site (yes, it's always network fault) determined replica VMs to end up in a corrupted state. If you want to keep the replica VMs that are already in the DR site then the fix is pretty simple: remove snapshots and remap vmdk's still pointing to delta files. Manually doing this is cumbersome, especially since some VMs can have multiple disks and when we are talking about tens or hundreds of  VMs. Luckily, some things can be scripted.

We will present some of the modules in the script, since the whole script is published here on GitHub. For ease of readability we will remove some of the try-catch blocks and logging messages from the original script and comment here only the logical part.

Script is given as input parameters vCenter Server where VM replicas are, backup server hostname, replica job status, replica job fail message and replica suffix. The replica suffix is important since it uses it to find VMs

$vbrServer = "vbr1"
$vcServer = "vc1"
$status = "Failed"
$reason = "Detected an invalid snapshot configuration."
$replicaSuffix = "_replica"

The script needs to be executed from a place where both vCenter Server and Veeam backup server are reachable and where PowerCLI module is imported as well as Veeam PowerShell snapin.

Add-PSSnapIn -Name VeeamPSSnapin
Connect-VBRServer -Server $vbrServer
Connect-VIServer -Server $vcServer

Next, the script will get all VMs for which the replication job has failed ($status) with the given reason ($status). For this, we use Get-VBRJob cmdlet, FindLastSession() and GetTaskSessions() methods. Once the VM in a replica job matches the chosen criteria, it i added to an array ($vmList)

$vmList = @()
# get failed replica VM names 
$jobs = Get-VBRJob  | Where {$_.JobType -eq "Replica"}
foreach($job in $jobs)
{
 $session = $job.FindLastSession()
 if(!$session){continue;}
 $tasks = $session.GetTaskSessions() 
 $tasks | foreach { 
        if (($_.Status -eq $status) -and ($_.Info.Reason -match $reason)) {
            $vmList += $_.Name
            }
        }
}

Once we have the list of VMs who's replica failed, it's time to get dirty. The VM name we are looking for is made of the original VM name (from the array) and the replica suffix.

$replicaName = $_ + $replicaSuffix

First, delete the snapshots. Fot this we use PowerCLI cmdlets: Get-VM, Get-Snapshots, Remove-Snapshot.


$replica = Get-VM -Name $replicaName -ea Stop
$replica | Get-Snapshot -ea Stop | Sort-Object -Property Created | Select -First 1 | Remove-Snapshot -RemoveChildren -Confirm:$false -ea Stop

Next, remap the disks if they are still pointing to the delta file. In order to do that, we get all the disks for the replica VM (Get-HardDisk) and we check if the disk name of the replica VM contains the specific delta file characters ("-0000"). This is how we determine if it's a delta disk or a normal disk. Delta disk name is parsed to generate the source disk name (removing the delta characters from vmdk name). Once this is done, it's just a matter of reattaching the source disk to the VM (Remove-HardDisk, New-HardDisk)

# get disks for a replica VM
$disk = $replica |  Get-HardDisk -ea Stop
# process each disk 
$disk | foreach {
    $diskPath = $_.Filename
    # check if disk is delta file
    if ($diskPath -Match "-0000") {
        $diskPath = $_.Filename
        # for each delta file parse the original vmdk name
        $sourceDisk = $diskPath.substring(0,$diskPath.length-12) + ".vmdk"
        # get the datastore where the delta is 
        $datastore = Get-Datastore -Id $_.ExtensionData.Backing.Datastore
        # check the original vmdk still exists on that datastore
        if (Get-HardDisk -Datastore $datastore.Name -DatastorePath $sourceDisk) {
            # remove delta
            Remove-HardDisk -HardDisk $_ -Confirm:$false -ea stop
            # attach original disk
            $newDisk = New-HardDisk -VM $replica -DiskPath $sourceDisk -ea Stop
        } else {
            Write-Host "WARN Could not find $($sourceDisk) on $($datastore.Name) "
            Add-Content -Path  $logFile -Value "WARN: Could not find $($sourceDisk) on $($datastore.Name) "
        }
    }
}

Last thing to do is to consolidate disks. We run Get-VM and check if consolidation is needed. If it's needed, we just run ConsolidateVMDisks_Task() method.


$vm = Get-VM -Name $replicaName -ea Stop
if ($vm.Extensiondata.Runtime.ConsolidationNeeded) {
    $vm.ExtensionData.ConsolidateVMDisks_Task()
}

Now the replica VMs are re-usable. There is some manual job to be done, though. You need to map the replica VM in the replication job and run the job.

Wednesday, March 29, 2017

Veeam Backup and Replication: Offsite Backup Repository

One of the common scenarios for backup infrastructures is to send the local backups to a secondary site. In a Veeam Backup and Replication (VBR) environment, this can be easily done by deploying a backup repository in the secondary location and then configuring backup copy jobs for the backups that you want to be sent offsite. VBR  v9.5 supports 4 types of backup repositories:

  • Windows server with local or direct attached storage - local disk, USB drive, iSCSI/FC LUN
  • Linux server with local, direct attached storage or mounted NFS - local disk, USB drive, iSCSI/FC LUN, NFS mounts
  • CIFS (SMB) share
  • deduplicating storage appliance - only the following three are supported EMC Data Domain, ExaGrid and HPE StoreOnce and there is also Enterprise license required.
For the current implementation, the chosen solution is implemented in a VMware environment across two vCenter Servers. VBR Server and main repository are located in the primary site. In the secondary site a backup repository has been installed on top of a Windows VM. Data mover service is installed in both sites. 


Having a data mover service in secondary site also enables backups directly to secondary site. Now, let's see how to configure the offsite repository.

First we need to deploy the Windows VM. The process is "standard" procedure: reserve an IP address, deploy from template, select VM name, compute resource and storage, customize the guest OS (including joining to AD).

After the VM has been deployed, configure the VM hardware if necessary: repository space depending on the size of the backups and RAM (4GB for OS and up to 4GB for each concurrent backup job).

Once the Windows VM is up, go to VBR management console, backup infrastructure tab and start repository configuration wizard by right clicking on Backup Repository -> Add backup repository.
Add a name and a description for the new repository:

Select the type of repository (Windows, Linux, CIFS, Appliance):

On the repository server list page press "Add new"

This will open a new wizard that configures a new windows server repository. Add DNS name or IP address of the repository server (optionally a description):

Add the credentials to use for connecting to the VM. If you've saved them in the credential manager select them from the drop down list, otherwise click Add button and enter the username and password.

Review the components to be installed and press Apply. The wizard will install VBR components on the repository server and it displays the progress in the window:

A summary page is displayed with info about the target server:

Once server is configured the  new backup repository is displayed in the list of servers. Press Populate button to retrieve all the available storage locations. Select the appropriate storage, press Next.

Configure the repository parameters: backup folder path, maximum number of concurrent tasks, read and write data rates (if necessary). 

Advanced configuration features are realated mostly to storage appliances. Pressing Advanced button allows to select the following:
  • Align backup file data blocks - useful for better deduplication ratios on storage appliances that use constant block size deduplication
  • decompress backup data before storing - achieves better deduplication ratio on most storage appliances at the cost of performance
  • this repository is backed by rotated hard drives - if hard drives are rotated and removed from the server
  • user per-VM backup files - multiple I/O streams per VM will improve performance with storage appliances 
Next, select the mount server and whether to enable vPower NFS or not. Default TCP ports for mount server and vPower NFS could be changed if necessary (press Ports button).

Review the configuration page of the server and press Apply.

The summary page will display the tasks, the progress and their status - creating repository folder, installing components (mount server, vPower NFS), configuring components. 

The following services are installed on the repository server:
  • Veeam Data Mover Service - sends and receives data 
  • Veeam Mount Server - mounts backups and replicas for file-level access
  • Veeam Installer Service - installs, updates and configures VBR components
  • Veeam vPower NFS Service - enables running VMs directly from backup files by "publishing" VM vdmk's from backup files to vPower NFS datastore. The datastore is then mounted on the ESXi host.
Once the process finishes, the new repository appears in the repository list of the VBR console and it is ready to use. Use it as a destination for a copy job or as an offsite backup job.