Backup vSAN 7 File Share with Veeam Backup & Replication 10

This is a blog post about two of the new features that were released this year in vSAN 7 and in Veeam Backup & Replication v10:

• vSAN File Service
• Veeam NAS Backup 

As you've already guessed, we're going to create a file share on vSAN, put files on it and backup it up with Veeam Backup & Replication (VBR) using the native NAS support.

At a high level the environment looks as following:

A 2 node vSAN 7 cluster runs in a nested environment. Each node has 4 vCPU, 32 GB of RAM, 20 GB Cache SSD and 200 GB Data SSD. A witness appliance has been also deployed. VBR v10 is configured with the minimum hardware: 2 vCPU and 8GB. Both proxy and repository roles are installed on the same VM with backup server, as well as the embedded MSSQL Express DB. Network connectivity between vSAN and VBR is 1 Gbit.

The configuration of the vSAN cluster or the installation of VBR are not in scope of this blog as there are many resources out there. I've used William Lam's nested library for the vSAN nodes and downloaded the witness appliance directly from VMware site.

The prerequisites for the following steps are: a running vSAN cluster and a VBR installation. You will also need 2 IP's reserved for File Server virutal appliances (one for each node) and a working DNS server with records added for the 2 IP's

Part 1 - vSAN File Service

vSAN File Service provides file shares on top of vSAN storage. It supports NFS 3 an NFS 4 protocols. It comprises of vSAN Distributed File System (vDFS) and it is integrated with vSAN Storage Policy Based Management.

Enable vSAN File Services

At the cluster level go to Configure - vSAN - Services and press enable File Service

Select whether the File Server agent OVF appliance will be download automatically or it will be manually uploaded:

Give a namespace for file share, type in DNS server address and domain

Select a portgroup, type the netmask and default gateway IP

Type in the IP addresses for each of the nodes

Review the configuration and press finish. Installation process will start deploying the agents on each of the vSAN nodes.

Once it is finished you will 2 new components deployed on the vSAN cluster: the File Service Agents. These are virtual appliances running Photon OS 1 and Docker and they act as NFS file servers.

Create NFS file share

It's time to create the share and put some files to it. To create the share Configure - vSAN - File Services Share. Enter the name of the share, storage policy, soft and hard quota limits.

You can also assign labels to the file share. Quotas can be changed at a later time. Next define access control: what IP addresses have access to this share, what type of access and if to protect the share with root squash.

Lastly review the settings and create the share. Once created you will see it vSAN - File Services Share:

Now the file share is ready to be used. Because in part 2 we will use Veeam's NAS backup, we've added a small load of files on the share using a linux VM. First get the share path by selecting the file share and pressing copy on URL:

Next ssh to your favorite Linux VM and mount the file share:

To create random files with random size, we've used a simple script found on here and which was slightly adapted.

for n in {1..500}; do
    dd if=/dev/urandom of=file$( printf %03d "$n" ).bin bs=100 count=$(( RANDOM + 1024 ))
done

The workload tested was:
- 200 files ranging  between a few hundreds of KB and a few MB
- 50 files ranging from a few MB and few tens of M

The script can be adapted to create larger workloads with thousands of files. Since the whole lab is nested, performance testing was not in scope.

Part 2 - Veeam NAS Backup 

The architecture for NAS backup requires at a minimum a file share (in our case NFS from vSAN), file proxy, cache repository, backup repository and a Veeam Backup Server.

Additionally, secondary and archive repositories can be added in infrastructure. In our lab infrastructure all components are deployed on a single VM - the backup server.

File proxy - component that acts as the data mover transporting data from the source (file share) to the backup proxy. It is used in backup and restore activities.

Cache repository - is a location to store temporary metadata. This metadata is used to reduce the load on the file share during incremental backups.

Backup repository - main location of the backups

Add File Share 
In VBR Console go to Inventory - Add File Share

Select NFS share and specify the path to the share

Select the File proxy, cache repository and backup speed

Review the summary and finish the configuration.

Having the file share added, we will add it to a backup job. Right click it and add it to a new backup job (or use an existing one).

Select the file share to backup

Select destination repository, backup policy and archive repository if any.

Select a secondary target if required (backup copy for short term)

Enable the job to run automatically

Review the settings and run the job.

The initial execution backups up all 250 files on the share. For the incremental, we've removed and recreated the first 50 files on the share:

for n in {1..50}; do 
  rm -f file$( printf %03d "$n" ).bin
  dd if=/dev/urandom of=file$( printf %03d "$n" ).bin bs=100 count=$(( RANDOM + 1024 ))
done

As you can see the cache repository is pretty effective in determining which files to be backed up:

Check ESXi MTU settings with PowerCLI

Sometimes the simplest tasks can be time consuming in large environments. This time is about MTU settings on the ESXi host,

First let's see a bit about MTU (Maximum Transmission Unit). It is a setting that defines the largest protocol data unit that can be sent across a network (largest packet or frame). Default setting is 1500 bytes. Having a bigger MTU increases the performance for particular uses cases as large amounts of data transmission over Ethernet. So it's always been set to larger values (9000 bytes) for accessing iSCSI and NFS storage. For a vSphere environment it means it could (and should in some cases) be increased for almost all types of traffic: vMotion, vSAN, Provisioning, FT, iSCSI, NFS, VXLAN, FCoE.

Let's take the use case of accessing a NFS datastore, as seen in the picture below:

The biggest challenge with MTU is to have the environment properly configured end-to-end.This means when you want your ESXi to make use of large MTU for accessing a NFS datastore, you'll need to make sure that distributed virtual switches, physical network interfaces, vmkernel portgroups, physical switches at system level and per port level and filers are configured with the proper MTU. What happens in our example when some elements are configured with the default MTU (1500)? In the case the vmkernel portgroup is set at 1500, then no you will see no performance benefits at all. If one of the physical switches is configured with 1500 bytes, then you will get fragmentation of the packets (performance degradation).

Hoping this short theoretical intro to MTU was helpful, I will jump ahead to the topic: checking ESXi MTU with PowerCLI. We are not treating how to check physical switches and storage devices within present article.

At ESXi level we need to check 3 settings: distributed virtual switch, physical network interfaces (vmnic used for uplinks) and vmkernel portgroups. To accomplish this we make use of two different PowerCLI cmdlets: Get-EsxCli and Get-VMHostNetworkAdapter

The beauty of Get-EsxCli is that it exposes esxcli commands and you can access the host through vCenter Server connection (no root or direct login to the ESXi host is required). The not so nice part its you have to use esxcli syntax in PowerCLI as you will soon see.

Main checks

We will first look at the script checks. Please keep in mind $h is a variable initialized with Get-VMHost cmdlet.

• distributed virtual switch - will get the switch name, configured MTU and used uplinks; the loop ensures all dvswitches are checked

(Get-EsxCli -VMHost $h -V2).network.vswitch.dvs.vmware.list.Invoke() | foreach {
    Write-Host "DVSName  $($_.Name) MTU  $($_.MTU) UPLINKS  $($_.Uplinks)"
}

• vmnics - check configured MTU, admin and link status for each interface (there is no issue in having unused nics configured differently) 

(Get-EsxCli -VMHost $h -V2).network.nic.list.Invoke() |  foreach {
    Write-Host "NIC:"$_.Name "MTU:"$_.MTU "Admin:"$_.AdminStatus "Link:"$_.LinkStatus
}

• vmkernel portgroups - check configure MTU and IP address

$vmks = $h | Get-VMHostNetworkAdapter | Where { $_.GetType().Name -eq "HostVMKernelVirtualNicImpl" }
foreach ($v in $vmks) {
    Write-Host "VMK $($v.Name) MTU $($v.MTU) IP $($v.IP)"
    }

The script

Putting it all together, we'll add the three checks in a foreach loop. The loop iterates through all the clusters and within each cluster through all the hosts. The script creates one log file per cluster containing all the hosts in that cluster and their details:

foreach ($cls in Get-Cluster) {
    $fileName = $cls.Name + ".log"
    Write-Host "# CLUSTER $($cls)" -ForegroundColor Yellow
    foreach ($h in $cls |   Get-VMHost) {
        Write-Host "$($h)" -ForegroundColor Yellow
        Add-Content -Path $fileName -Value "$($h)"

        (Get-EsxCli -VMHost $h -V2).network.vswitch.dvs.vmware.list.Invoke() | foreach {
            Write-Host "DVSName  $($_.Name) MTU  $($_.MTU) UPLINKS  $($_.Uplinks)"
            Add-Content -Path $fileName -Value "DVSName $($_.Name) MTU $($_.MTU) UPLINKS $($_.Uplinks)"
        }
        (Get-EsxCli -VMHost $h -V2).network.nic.list.Invoke() |  foreach {
            Write-Host "NIC:"$_.Name "MTU:"$_.MTU "Admin:"$_.AdminStatus "Link:"$_.LinkStatus
            Add-Content -Path $fileName -Value "NIC: $($_.Name) MTU: $($_.MTU) Admin: $($_.AdminStatus) Link: $($_.LinkStatus)"
        }
        $vmks = $h | Get-VMHostNetworkAdapter | Where { $_.GetType().Name -eq "HostVMKernelVirtualNicImpl" }
        foreach ($v in $vmks) {
            Write-Host "VMK $($v.Name) MTU $($v.MTU) IP $($v.IP)"
            Add-Content -Path $fileName -Value "VMK $($v.Name) MTU $($v.MTU) IP $($v.IP)"
         }
    }
}

Opening one of the log files, you see similar output to below:

esx-01.rio.lab
DVSName dvs-Data1 MTU 9000 UPLINKS vmnic3 vmnic2 vmnic1 vmnic0
NIC: vmnic0 MTU: 9000 Admin: Up Link: Up
NIC: vmnic1 MTU: 9000 Admin: Up Link: Up
NIC: vmnic2 MTU: 9000 Admin: Up Link: Up
NIC: vmnic3 MTU: 9000 Admin: Up Link: Up
VMK vmk0 MTU 9000 IP 192.168.10.11
VMK vmk1 MTU 9000 IP 192.168.20.11
VMK vmk2 MTU 9000 IP 192.168.30.11

In this case everything looks good at ESXi level. Easy part is over, so start digging in the physical switches CLI and any other equipment along the path to ensure end to end MTU consistency.

Command line upgrade of ESXi 5.1 to 5.1 U1

The reason I`ve chosen this method is not having VUM installed and also wanting to remember a bit of esxcli. The process is pretty simple:

• download the zip bundle (update-from-esxi5.1-5.1_update01.zip) and put it on a datastore accessible by all hosts (I am using a NFS share)
• check to see if the host has the datastore mounted, if not - add the datastore

~ # esxcli storage nfs list

~ # esxcli storage nfs add -H 192.168.1.1 -s /mnt/vol1-nfs -v shared1-nfs-sata

• check the host version

~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-799733
   Update: 0

• check the update file and see if it requires the host in maintenance mode (in this case, it does not - however since the update will request a system reboot, I think it is better to place the host in maintenance mode and have it cleaned up of VMs)

~ # esxcli software sources vib get --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip
...
   Maintenance Mode Required: False

• install the update

~ # esxcli software vib update --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true

• reboot the system (after all VMs have been migrated to other hosts) and check the new host version

~ # reboot
...
~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-1065491
   Update: 1

NFS traffic rate limiting on Juniper switches

"And now for something completely different"... 

The configuration for the ESXi infrastructure I worked with is a bit tricky. VMs are hosted on NFS filer, but the same VMs are mounting NFS exports from the same filer. Much like in the picture below:

The physical bandwidth between the access switch where ESXi are connected and the core switches where the Filer is connected is limited. So, I had a lot of ESXi servers, with a lot more VMs competing over the same physical links. Lucky me, I had control over the access switches - Juniper. The next step was elementary: ensure enough bandwidth for ESXi and give some to the VMs - rate limiting using firewall filtering from JunOS.

And this is how it was done. First, a policer was created for NFS traffic coming from VMs (guest OS) which limits the allocated bandwidth - in my case 200 Mbits with a burst size of 10 MB. When the limit is reached packets are discarded. This way two things are achieved: a decent 200 Mbit bandwidth  is ensured for the VMs and small files (up to 10 MB) are transferred very fast to the Filer (no limits). When the VMs demand a lot of resources, the policer steps in and ensures that the critical ESXi vmk traffic gets its share.

[edit firewall]
set policer policer-NFS-1 if-exceeding bandwidth-limit 200m burst-size-limit 10m
set policer policer-NFS-1 then discard

Then, the firewall filter is created. The filter matches all traffic that goes to the IP address of the Filer and applies the policer to it:

[edit firewall family inet]
set filter limit-vlan100-NFS term term-1 from destination-address 192.168.100.192
set filter limit-vlan100-NFS term term-1 then policer policer-NFS-1
set filter limit-vlan100-NFS term term-default then accept

Last, add firewall filter is applied to the interface - in this case it is VLAN 100:

[edit]
set interfaces vlan unit 100 family inet filter input limit-vlan100-NFS

The downside is that firewall filtering adds a bit of a load on the CPUs of the switches. Care should be taken when implementing such solutions (as always).

Rsync cyrus volumes from EMC to NetAPP

A couple of weeks ago I was faced with the following problem: clean up the EMC filer that hosts one of our client`s cyrus e-mail mailboxes.Putting it all on paper I got to the following:

- there were more than 3 million files and folders

- the only available protocol is NFS

- the bandwidth is 1 Gbit

- source filer is EMC, destination is NetApp

The chosen option was a copy-paste operation from one filer to another with service interruption to avoid synchronization problems. One question was raised: how long it will take to do it and what tool to use? 

After some parking lot meetings and discussions, the shortlist for tools included cp, rsync and cpio. First thing first, speed tests. I have generated a workload of 50000 files having the following composition: 40% @ 1 KB, 20% @ 2 KB, 20% @ 3 KB and 20% @ 10 KB using for and dd:

### first 20000 files 1K, then increase seq and bs 

for i in $(seq 1 1 20000); do dd if=/dev/zero of=/tmp/FILER/vol99/$i count=1 bs=1024 2>/dev/null; done

Next, the tests, using time command:

### cp

time  cp -a vol99/ /tmp/DEST/vol99

### rsync

time rsync -a vol99/ /tmp/DEST/vol99/ 

### cpio 

time find . -print -depth | cpio -padmuv /tmp/DEST/vol99/

cp -a does the same thing as cp -dpR - preserves links, permissions and is recursive. rsync -a does the same as rsync -rlptgoD and I was mainly interested to ensure recursive transfer, preservation of permissions, modification times, groups and owners. cpio -padmu ensures creation of directories where needed, preservation of mtime.

The fastest was cp and it was considered the baseline. rsync was 7% slower, while cpio was 46% slower (because of the big find piped to it). Since the obvious choice was cp, it was decided to use rsync (mostly because of the possibility to continue interrupted transfers and to do check sums - which was never used). The speed tests also allowed to approximate transfer times of the real data at around 100 minutes. This way I could go ahead with the formalities on the client side.

When rsync was tested against real data (around 400K files and directories), it generated a bit of a surprise. It started to build up its file list, it took more than 15 minutes to do it and it started a very slow transfer process - after 1 hour it did not finish one volume. And we had 5.  The main cause was the partition structure used by cyrus imap for mail store and the big number of files. To get an idea: vol1 has 20 imap partitions: cyrus1 to cyrus20. Each cyrusN partition has the the mailbox structure from A to Z. The first rsync test was:

nohup rsync -a /SRC/vol1/ /DEST/vol1/ &

and this got stuck into creating the file list and transferring very slow. The idea was to have fast transfer, so for the second test rsync was run in parallel for each cyrus partition from 1 to 20:

nohup rsync -a /SRC/vol1/cyrus1/ /DEST/vol1/cyrus1/ &

nohup rsync -a /SRC/vol1/cyrus2/ /DEST/vol1/cyrus2/ &

...

nohup rsync -a /SRC/vol1/cyrus20/ /DEST/vol1/cyrus20/ &

This time it worked just as expected. Actually, during the operation we managed to increase the CPU on a NetApp 6280 by 10%-15% using parallel rsync.

One more thing to be said: rsync syntax is a bit tricky. If the slash at the end of source directory is omitted, then it will create the directory itself in the destination and you will end up with a shifted folder structure: /DEST/vol99/cyrus20/cyrus20/.